Github star 5.2K+! bloomberg open source Python memory analyzer "Memray"

Recently, bloomberg open sourced a Python memory analyzer - "Memray". At present, the project has obtained 5.2K stars on Github, and the popularity is good.

Memray is a Python memory profiler that tracks memory allocations in Python code, native extension modules, and the Python interpreter itself. Several different types of reports can be generated to help analyze captured memory usage data. Typically used as a CLI tool, but also as a library to perform more fine-grained analysis tasks.

main feature:

• Traces each function call so that it can accurately represent the call stack, unlike the sampling profiler.
• Handling native calls in C/C++ libraries, the entire call stack is present in the result.
• high speed! Profiling minimizes the speed of the application. Tracing native code is slightly slower and can be enabled or disabled as needed.
• Various reports can be generated on the collected memory usage data, such as flame graphs.
• Use Python threads.
• Works with native threads such as C++ threads in C extensions.

Memray can help with the following issues:

• Analyze allocations in your application to help discover the cause of high memory usage.
• Find memory leaks.
• Find code hotspots that cause large allocations.

It is worth noting that Memray only works on Linux and cannot be installed on other platforms.

Install

Memray requires Python 3.7+ and can be easily installed using the most common Python packaging tools. It is recommended to install the latest stable version of PyPI using pip:

python3 -m pip install memray

Note that Memray includes a C extension, so both release and source code are distributed as binary code. If your system (Linux x86/x64) does not have a binary wheel available, you need to make sure that the installed system satisfies all dependencies.

usage

There are many ways to use Memray. The easiest way is to use it as a command line tool to run scripts, applications or libraries.

usage: memray [-h] [-v] {run,flamegraph,table,live,tree,parse,summary,stats} ...

Memory profiler for Python applications

Run "memray Run" to generate a memory profile report, then use a reporter command such as "memray flamegraph" or "memray table" to convert the results to HTML.

native mode

Memray supports tracing native C/C++ functions and Python functions. This is especially useful when profiling applications with C extensions, such as numpy or pandas, because it gives a complete picture of how much memory is allocated by the extension, as well as how much memory is allocated by Python itself.

To activate native tracing, you need to provide the --native parameter when using the run subcommand:

memray run --native my_script.py

This will automatically add native information to the results file, and any reporter programs (like flamegraph or table reporters) will automatically use it.

The reporter displays native frames in a different color than Python frames, and also distinguishes them by looking at the file location within the frame (Python frames are usually generated from files with a .py extension, while native frames are generated by a .c extension , .cpp or .h files).

Live mode

Memray's Live mode runs a script or module in a terminal-based interface, allowing you to interactively inspect its memory usage at runtime. This is useful for debugging scripts or modules that take a long time to run or render multiple complex memory patterns.