Loguru: The ultimate solution for Python logging

Pay attention to WeChat public account: K brother crawler, QQ exchange group: 808574309, continue to share advanced crawler, JS/Android reverse engineering and other technical dry goods!

Importance of logs

The role of the log is very important. The log can record user operations and program exceptions, and can also provide a basis for data analysis. The purpose of the log is to record errors during the running of the program, facilitate maintenance and debugging, and quickly locate errors Place to reduce maintenance costs. Every programmer should know that logs are not recorded for the sake of logging, and logs are not arbitrarily recorded. It is necessary to realize the process of restoring the execution of the entire program only through the log file, and achieve the goal of transparently seeing the execution of the program and where each thread and each process is executed. The log is like the black box of an airplane, and it should be able to recover the entire scene and even the details of the abnormality!

Common logging methods

print()

The most common one is to use the output function print() as a log record method to directly print various prompt messages. It is common in personal practice projects. Usually it is too lazy to configure the log separately, and the project is too small and does not require log information, does not need to be online, no It needs to run continuously. It is not recommended to print log information directly for a complete project. In reality, almost no one does this.

Self-written template

We can see in many small projects that the author wrote a log template. Usually, you can use print() or sys.stdout encapsulate a little to achieve simple log output. Here sys.stdout is the standard output stream in Python, and the function print() sys.stdout print(obj) to print the object in Python, we actually call sys.stdout.write(obj+'\n') , print() the content to the console, and then appends a newline character \n .

The self-written log template is suitable for relatively small projects. You can write the template according to your own preferences. It does not require too much complicated configuration, which is convenient and fast. However, this method of recording logs is not very standardized. You may feel that the reading experience is good, but When others come into contact with your project, they often need to spend a certain amount of time to learn the logic, format, output method, etc. of the log. This method is also not recommended for larger projects.

An example of a simple self-written log template:

Log template log.py:

import sys
import traceback
import datetime


def getnowtime():
    return datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")


def _log(content, level, *args):
    sys.stdout.write("%s - %s - %s\n" % (getnowtime(), level, content))
    for arg in args:
        sys.stdout.write("%s\n" % arg)


def debug(content, *args):
    _log(content, 'DEBUG', *args)


def info(content, *args):
    _log(content, 'INFO', *args)


def warn(content, *args):
    _log(content, 'WARN', *args)


def error(content, *args):
    _log(content, 'ERROR', *args)


def exception(content):
    sys.stdout.write("%s - %s\n" % (getnowtime(), content))
    traceback.print_exc(file=sys.stdout)

Call the log module:

import log


log.info("This is log info!")
log.warn("This is log warn!")
log.error("This is log error!")
log.debug("This is log debug!")

people_info = {"name": "Bob", "age": 20}

try:
    gender = people_info["gender"]
except Exception as error:
    log.exception(error)

Log output:

2021-10-19 09:50:58 - INFO - This is log info!
2021-10-19 09:50:58 - WARN - This is log warn!
2021-10-19 09:50:58 - ERROR - This is log error!
2021-10-19 09:50:58 - DEBUG - This is log debug!
2021-10-19 09:50:58 - 'gender'
Traceback (most recent call last):
  File "D:/python3Project/test.py", line 18, in <module>
    gender = people_info["gender"]
KeyError: 'gender'

Logging

In a complete project, most people will introduce a dedicated logging library, and Python's own standard library logging is specifically for logging. The functions and classes defined by the logging module are implemented for the development of applications and libraries. A flexible event log system. The key benefit of the logging API provided by standard library modules is that all Python modules can use this logging function. Therefore, your application log can integrate your own log information with information from third-party modules.

Although the logging module is powerful, its configuration is relatively cumbersome. In large projects, it is usually necessary to initialize the log separately, configure the log format, etc. In daily use, K brother usually does the following encapsulation for logging, so that the log can be daily Save, keep 15 days of logs, you can configure whether to output to the console and files, as shown below:

# 实现按天分割保留日志


import os
import sys
import logging
from logging import handlers


PARENT_DIR = os.path.split(os.path.realpath(__file__))[0]  # 父目录
LOGGING_DIR = os.path.join(PARENT_DIR, "log")              # 日志目录
LOGGING_NAME = "test"                                      # 日志文件名

LOGGING_TO_FILE = True                                     # 日志输出文件
LOGGING_TO_CONSOLE = True                                  # 日志输出到控制台

LOGGING_WHEN = 'D'                                         # 日志文件切分维度
LOGGING_INTERVAL = 1                                       # 间隔少个 when 后，自动重建文件
LOGGING_BACKUP_COUNT = 15                                  # 日志保留个数，0 保留所有日志
LOGGING_LEVEL = logging.DEBUG                              # 日志等级
LOGGING_suffix = "%Y.%m.%d.log"                            # 旧日志文件名

# 日志输出格式
LOGGING_FORMATTER = "%(levelname)s - %(asctime)s - process:%(process)d - %(filename)s - %(name)s - line:%(lineno)d - %(module)s - %(message)s"


def logging_init():
    if not os.path.exists(LOGGING_DIR):
        os.makedirs(LOGGING_DIR)
    logger = logging.getLogger()
    logger.setLevel(LOGGING_LEVEL)
    formatter = logging.Formatter(LOGGING_FORMATTER)

    if LOGGING_TO_FILE:
        file_handler = handlers.TimedRotatingFileHandler(filename=os.path.join(LOGGING_DIR, LOGGING_NAME), when=LOGGING_WHEN, interval=LOGGING_INTERVAL, backupCount=LOGGING_BACKUP_COUNT)
        file_handler.suffix = LOGGING_suffix
        file_handler.setFormatter(formatter)
        logger.addHandler(file_handler)

    if LOGGING_TO_CONSOLE:
        stream_handler = logging.StreamHandler(sys.stderr)
        stream_handler.setFormatter(formatter)
        logger.addHandler(stream_handler)


def logging_test():
    logging.info("This is log info!")
    logging.warning("This is log warn!")
    logging.error("This is log error!")
    logging.debug("This is log debug!")
    people_info = {"name": "Bob", "age": 20}

    try:
        gender = people_info["gender"]
    except Exception as error:
        logging.exception(error)


if __name__ == "__main__":
    logging_init()
    logging_test()

Output log:

INFO - 2021-10-19 11:28:10,103 - process:15144 - test.py - root - line:52 - test - This is log info!
WARNING - 2021-10-19 11:28:10,105 - process:15144 - test.py - root - line:53 - test - This is log warn!
ERROR - 2021-10-19 11:28:10,105 - process:15144 - test.py - root - line:54 - test - This is log error!
DEBUG - 2021-10-19 11:28:10,105 - process:15144 - test.py - root - line:55 - test - This is log debug!
ERROR - 2021-10-19 11:28:10,105 - process:15144 - test.py - root - line:61 - test - 'gender'
Traceback (most recent call last):
  File "D:/python3Project/test.py", line 59, in logging_test
    gender = people_info["gender"]
KeyError: 'gender'

It looks like this in the console:

Of course, if you don’t need very complicated functions and want to be more concise, and you only need to output a log in the console, you can also just make a simple configuration:

import logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logging.getLogger()

A more elegant solution: Loguru

For the logging module, even if it is simple to use, you need to define the format yourself. Here is a more elegant, efficient and concise third-party module: loguru. The official introduction is: Loguru is a library which aims to bring enjoyable logging in Python. Loguru aims to bring pleasant logging to Python. Here is an official GIF to quickly demonstrate its function:

Install

Loguru only supports Python 3.5 and above, use pip to install it:

pip install loguru

Out of the box

The main concept of Loguru is that there is only one: logger

from loguru import logger

logger.info("This is log info!")
logger.warning("This is log warn!")
logger.error("This is log error!")
logger.debug("This is log debug!")

Console output:

It can be seen that there is no need to manually set it. Loguru will configure some basic information in advance, and automatically output information such as time, log level, module name, line number, etc., and automatically set different colors according to different levels, which is convenient for observation and real work. Arrived out of the box!

add() / remove()

What if I want to customize the log level, customize the log format, and save the log to a file? Unlike the logging module, no Handler or Formatter is needed, only a add() function is needed. For example, we want to save the log to a file:

from loguru import logger

logger.add('test.log')
logger.debug('this is a debug')

We don't need to declare a FileHandler like the logging module, just a line of add() statement. After running, we will find that the debug information just output from the console also appears in the directory test.log.

Contrary to the add() statement, the remove() statement can delete the configuration we added:

from loguru import logger

log_file = logger.add('test.log')
logger.debug('This is log debug!')
logger.remove(log_file)
logger.debug('This is another log debug!')

At this time, the console will output two debug messages:

2021-10-19 13:53:36.610 | DEBUG    | __main__:<module>:86 - This is log debug!
2021-10-19 13:53:36.611 | DEBUG    | __main__:<module>:88 - This is another log debug!

There is only one debug message in the test.log log file. The reason is that we used the remove() statement before the second debug statement.

Full parameters

Loguru has very powerful support for the configuration of output to files, such as supporting output to multiple files, outputting in different levels, creating new files when too large, and automatically deleting when too long, and so on. Let's take a look at the detailed parameters of the add()

Basic syntax:

add(sink, *, level='DEBUG', format='<green>{time:YYYY-MM-DD HH:mm:ss.SSS}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>', filter=None, colorize=None, serialize=False, backtrace=True, diagnose=True, enqueue=False, catch=True, **kwargs)

Interpretation of basic parameters:

• sink : It can be a file object, such as sys.stderr or open('file.log', 'w') , or it can be a str or a pathlib.Path , that is, the file path, or it can be a method, you can define your own output implementation, or it can be a Handler of a logging module, For example, FileHandler, StreamHandler, etc., can also be coroutine function , that is, a function that returns a coroutine object, etc.
• level : log output and save level.
• format : log format template.
• filter : An optional command used to determine whether each recorded message should be sent to the sink.
• colorize : Whether the color mark contained in the formatted message should be converted to ansi code for terminal coloring, or stripped in other ways. If not, the selection is made automatically according to whether the sink is a tty (acronym for teletypewriter).
• serialize : Whether the recorded message should be converted into a JSON string before sending to the sink.
• backtrace : Whether the formatted exception trace should be expanded upwards beyond the capture point to show the full stack trace of the generated error.
• diagnose : Whether abnormal tracking should display variable values to simplify debugging. It is recommended to set False in the production environment to avoid leaking sensitive data.
• enqueue : Whether the message to be recorded should pass through the multi-process secure queue before reaching the sink. This is useful when recording to a file through multiple processes. The advantage of this is that the logging call is non-blocking.
• catch : Whether to automatically catch errors that occur when the sink processes log messages, if it is True , an sys.stderr , but the exception will not be propagated to the sink, thus preventing the application from crashing.
• **kwargs : Additional parameters valid only for configuring coroutines or file receivers (see below).

If and only if sink is a coroutine function, the following parameters apply:

• loop : The event loop in which asynchronous logging tasks will be scheduled and executed. If None , will use asyncio.get_event_loop() cycle returned.

If and only if sink is a file path, the following parameters apply:

• rotation : A condition indicating when the currently recorded file should be closed and a new file should be started.
• retention : The instruction to filter old files, the old files will be deleted during the end of the cycle or program.
• compression : The compressed or archive format that the log file should be converted to when it is closed.
• delay : Whether to create the file immediately after configuring the sink, or to delay the creation until the first recorded message. The default is False .
• mode : The open mode of the built-in open() a (open the file in append mode).
• buffering : The buffering strategy of the built-in open() 1 (line buffer file).
• encoding : Built-in file encoding of open() None , the default is locale.getpreferredencoding() .
• **kwargs : Other parameters passed to the built-in open() function.

With so many parameters, you can see add() function. Only one function can achieve many functions of the logging module. Next, we will introduce a few more commonly used methods.

rotation log file separation

add() function can realize the creation of a new log file at a fixed time, such as setting a new log file at 0 o'clock every day:

logger.add('runtime_{time}.log', rotation='00:00')

Set more than 500 MB to create a new log file:

logger.add('runtime_{time}.log', rotation="500 MB")

Set to create a new log file every other week:

logger.add('runtime_{time}.log', rotation='1 week')

retention log retention time

add() function can set the maximum retention time of the log. For example, set the maximum retention time of the log file to 15 days:

logger.add('runtime_{time}.log', retention='15 days')

Set up to keep 10 log files:

logger.add('runtime_{time}.log', retention=10)

It can also be a datetime.timedelta object, such as setting the log file to be retained for up to 5 hours:

import datetime
from loguru import logger

logger.add('runtime_{time}.log', retention=datetime.timedelta(hours=5))

compression log compression format

add() the compression parameter of the 0618a389cafad4 function, you can configure the compression format of the log file, which can save storage space, such as setting to save in the zip file format:

logger.add('runtime_{time}.log', compression='zip')

Its format support: gz , bz2 , xz , lzma , tar , tar.gz , tar.bz2 , tar.xz

String formatting

Loguru also provides a very friendly string formatting function when outputting log, which is equivalent to str.format() :

logger.info('If you are using Python {}, prefer {feature} of course!', 3.6, feature='f-strings')

Output:

2021-10-19 14:59:06.412 | INFO     | __main__:<module>:3 - If you are using Python 3.6, prefer f-strings of course!

Anomaly traceability

In Loguru, you can directly use the decorator it provides to directly capture exceptions, and the logs obtained are extremely detailed:

from loguru import logger


@logger.catch
def my_function(x, y, z):
    # An error? It's caught anyway!
    return 1 / (x + y + z)


my_function(0, 0, 0)

Log output:

2021-10-19 15:04:51.675 | ERROR    | __main__:<module>:10 - An error has been caught in function '<module>', process 'MainProcess' (30456), thread 'MainThread' (26268):
Traceback (most recent call last):

> File "D:/python3Project\test.py", line 10, in <module>
    my_function(0, 0, 0)
    └ <function my_function at 0x014CDFA8>

  File "D:/python3Project\test.py", line 7, in my_function
    return 1 / (x + y + z)
                │   │   └ 0
                │   └ 0
                └ 0

ZeroDivisionError: division by zero

The output in the console is like this:

Compared with Logging, Loguru is far superior to Logging in terms of configuration, log output style, or abnormal tracking. Using Loguru can undoubtedly improve developer efficiency. This article only introduces some commonly used methods, if you want to learn more, please refer to Loguru official document or follow Loguru GitHub .