How can I use loguru to take over all the log output of a program?

Background and Pain Points - Log Requirements: Unified Output Format

loguru I don't need to introduce how excellent it is. Our own business code can be easily import loguru to print the log, a unified output of doubts

However, many log modules integrated by third-party libraries are in the standard library logging , and its format are various

I hope it can be achieved: use loguru to take over the ---d57ad98893bf66cea975350188652951--- of all libraries, and use the format output of loguru logging uniformly

reason? Because it is convenient for log collection and analysis and storage! For example, docking loki , sls , elk and so on.

Best practice - my ideal log output format:

I want every log output to look like this

 {
    "text": "2022-04-25 18:12:40.179 | ERROR    | __main__:<module>:18 - An error has been caught in function '<module>', process 'MainProcess' (60788), thread 'MainThread' (139849590506112):\nTraceback (most recent call last):\n\n> File \"/home/bot/Desktop/ideaboom/test_logger/loguru_contextualize.py\", line 18, in <module>\n    div(1,0)\n    └ <function div at 0x7f3143edf490>\n\n  File \"/home/bot/Desktop/ideaboom/test_logger/loguru_contextualize.py\", line 8, in div\n    return a/b\n           │ └ 0\n           └ 1\n\nZeroDivisionError: division by zero\n",
    "record": {
        "elapsed": { "repr": "0:00:00.007519", "seconds": 0.007519 },
        "exception": {
            "type": "ZeroDivisionError",
            "value": "division by zero",
            "traceback": true
        },
        "extra": {
            "context_id": "fd77b1d159224d939c1cc0d7a566d09b",
            "message_id": "8ad2c351ef3240998fc10a5015c591c1"
        },
        "file": {
            "name": "loguru_contextualize.py",
            "path": "/home/bot/Desktop/ideaboom/test_logger/loguru_contextualize.py"
        },
        "function": "<module>",
        "level": { "icon": "❌", "name": "DEUBG", "no": 40 },
        "line": 18,
        "message": "An error has been caught in function '<module>', process 'MainProcess' (60788), thread 'MainThread' (139849590506112):",
        "module": "loguru_contextualize",
        "name": "__main__",
        "process": { "id": 60788, "name": "MainProcess" },
        "thread": { "id": 139849590506112, "name": "MainThread" },
        "time": {
            "repr": "2022-04-25 18:12:40.179070+08:00",
            "timestamp": 1650881560.17907
        }
    }
}

The corresponding code is this:

 from loguru import logger
from mark import BASE_DIR
import logging

logger.add(BASE_DIR/'run.log', serialize='json')

I want the log output of loguru to a file, and then the log collector collects the log file. Instead of stdout, I want stdout to be seen by humans, not by log collectors.

That is to say, the log outputs two copies:

• stdout 、 stderror ， tail 、 docker-compose logs 、 kubectl logs查看， Show it to humans (the format does not need to be unified, anyway, humans can understand it if it is not unified, and this is simple text, not josn or something);
• Another output is to the run.log file, logtail , promtail , etc. The log collection program collects the log records in the run.log. (Have a unified log format, and one line is a json)

Advantages of outputting logs as json:

• It is convenient to collect, and there is no collection error caused by format problems! Don't worry about stack wrapping and need to use regular matching expressions at the beginning of the line, etc.
• It is convenient for analysis and docking es , mongodb very convenient

What about the downside?

• Volume expansion, json is n times larger than direct text

Code implementation - loguru

Create a loggers.py file with the following content:

 from loguru import logger
from mark import BASE_DIR
import logging

logger.add(BASE_DIR/'run.log', serialize='json')


class InterceptHandler(logging.Handler):
    def emit(self, record):
        # Retrieve context where the logging call occurred, this happens to be in the 6th frame upward
        logger_opt = logger.opt(depth=6, exception=record.exc_info)
        logger_opt.log(record.levelno, record.getMessage())


# logging.getLogger("uvicorn").setLevel(10)
# logging.getLogger("uvicorn").handlers = []
# logging.getLogger("uvicorn").addHandler(InterceptHandler())

# logging.getLogger("uvicorn.error").setLevel(10)
# logging.getLogger("uvicorn.error").handlers = []
# logging.getLogger("uvicorn.error").addHandler(InterceptHandler())

# logging.getLogger("fawn.error").setLevel(10)
# logging.getLogger("fawn.error").handlers = []
# logging.getLogger("fawn.error").addHandler(InterceptHandler())

logger_name_list = [name for name in logging.root.manager.loggerDict]
# logger_name_list = [name for name in logging.root.manager.loggerDict if '.' not in name]

for logger_name in logger_name_list:
    logging.getLogger(logger_name).setLevel(10)
    logging.getLogger(logger_name).handlers = []
    if '.' not in logger_name:
        logging.getLogger(logger_name).addHandler(InterceptHandler())


print(logger_name_list)
print(logging.root.manager.loggerDict)

In reality, for example uvicorn , peewee have their own output formats, as follows:

 ─➤  python api.py
INFO:     Started server process [2382519]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

After using the above code, it becomes:

 ─➤  uvicorn api:app                                                                                                   1 ↵
2022-04-30 23:09:11.283 | INFO     | uvicorn.server:serve:75 - Started server process [2395776]
2022-04-30 23:09:11.283 | INFO     | uvicorn.lifespan.on:startup:45 - Waiting for application startup.
2022-04-30 23:09:11.283 | INFO     | uvicorn.lifespan.on:startup:59 - Application startup complete.
2022-04-30 23:09:11.284 | INFO     | uvicorn.server:_log_started_message:206 - Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

explain the code

Do you know what the above code means?

todo

Reference article:
filter object
how can i logging peewee with loguru
How To Override Uvicorn Logger in FastAPI using Loguru