Python binding: call C or C++ from Python

Summary: Are you a Python developer who has a C or C++ library that you want to use from Python? If so, then the Python binding allows you to call functions and pass data from Python to C or C++, allowing you to take advantage of the advantages of both languages.

This article is shared from HUAWEI CLOUD Community " Python Binding: Calling C or C++ from Python | [Grow! Python!】 ", original author: Yuchuan.

Are you a Python developer with a C or C++ library that you want to use from Python? If so, then the Python binding allows you to call functions and pass data from Python to C or C++, allowing you to take advantage of the advantages of these two languages. In this tutorial, you will see an overview of some of the tools that can be used to create Python bindings.

In this tutorial, you will learn:

Why call C or C++ from Python
How to pass data between C and Python
Which tools and methods can help you create Python bindings

This tutorial is for intermediate Python developers. It assumes that the reader has a basic knowledge of Python and an understanding of functions and data types in C or C++. You can get all the sample code you will see in this tutorial by clicking the link below:

Overview of Python Bindings

Before delving into how to call C from Python, it's best to spend some time understanding why. There are several situations where it is a good idea to create Python bindings to call C libraries:

You already have a large, tested and stable library written in C++, and you want to use it in Python. This may be a communication library or a library that talks to specific hardware. It does not matter what it does.
you want by converting a key part of C to speed Python code specific portions of . C not only has a faster execution speed, but also allows you to get rid of the restrictions of the GIL, provided you are careful.
You want to use the Python testing tool to test its system on a large scale.

All of the above are important reasons to learn to create Python bindings to interact with the C library.

Note: In this tutorial, you will create Python bindings to C and C++. Most common concepts apply to both languages, so unless there are specific differences between the two languages, C will be used. Generally, each tool supports C or C++, but not both.

let's start!

Marshalling data type

wait! Before you start writing Python bindings, let's take a look at how Python and C store data and what types of problems this can cause. First, let us define the group . This concept is defined by Wikipedia as follows:

The process of converting the memory representation of an object into a data format suitable for storage or transmission.

For your purposes, marshalling is what Python binding does when preparing data to move it from Python to C or vice versa. Python bindings require marshalling because Python and C store data in different ways. C stores data in the most compact form in memory. If you use uint8_t, it will only use 8-bit memory in total.

On the other hand, in Python, everything is a object . This means that each integer uses several bytes in memory. How much depends on the version of Python you are running, operating system, and other factors. This means that your Python binding will need to convert the C integer to the Python integer passed across the boundary.

Other data types have similar relationships between the two languages. Let's take a look at it in turn:

integer stores the count number. Python stores integers in arbitrary precision, which means you can store very, very large numbers. C specifies the exact size of the integer. You need to pay attention to the data size when moving between languages to prevent Python integer values from overflowing C integer variables.
floating point number is a number with decimal places. Python can store floating point numbers much larger (and much smaller) than C. This means that you must also pay attention to these values to ensure that they stay within range.
complex number is a number with an imaginary part. Although Python has built-in complex numbers and C has complex numbers, there is no built-in method for grouping between them. To marshal plural numbers, you need to build structs or classes in C code to manage them.
string is a sequence of characters. As such a common data type, when you create Python bindings, strings will prove to be quite tricky. Like other data types, Python and C store strings in completely different formats. (Unlike other data types, this is also a different area of C and C++, which adds to the fun!) Each solution you will study has a slightly different way of dealing with strings.
Boolean variable can only have two values. Since they are supported in C, grouping them will prove to be fairly simple.

In addition to data type conversion, there are other issues to consider when building Python bindings. Let us continue to explore them.

Understand mutable and immutable values

In addition to all of these data types, you must also learn how Python objects variable or immutable . When it comes to passing the value or passing , C has a similar concept of function parameters. In C, all parameters are passed by value. If you want to allow the function to change the variable in the caller, you need to pass a pointer to that variable.

immutable limit simply passing the immutable object to C using a pointer. Unless you go to the extreme of ugliness and non-portability, Python won't give you a pointer to object, so this won't work. If you want to modify Python objects in C, then you need to take additional steps to achieve this. These steps will depend on the tool you use, as shown below.

Therefore, you can add immutability to the list of items to consider when creating Python bindings. In the grand journey of creating this checklist, your last stop is how to deal with the different ways Python and C handle memory management.

Manage memory

C and Python different ways of managing memory . In C, the developer must manage all memory allocations and ensure that they are released once and only once. Python uses the garbage collector to handle this problem for you.

Although each of these methods has its advantages, it does add extra trouble for creating Python bindings. You need to know memory allocation for each object where , and be sure that it was released on the same side of the language barrier.

For example, when you set x = 3. The memory used for this is allocated on the Python side and needs to be garbage collected. Fortunately, it is difficult to do anything else with Python objects. Look at the reverse in C and directly allocate a block of memory:

int* iPtr = (int*)malloc(sizeof(int));
When doing this, you need to make sure to release this pointer in C. This may mean manually adding code to the Python bindings to do this.

This completes your list of general topics. Let's start setting up your system so that you can write some code!

Set up your environment

In this tutorial, you will use the pre-existing C and C++ libraries from the Real Python GitHub repository to showcase the testing of each tool. The goal is that you will be able to apply these ideas to any C library. To follow all the examples here, you need to have the following:

Knowledge of installed C++ libraries and command line call paths
Python development tools:
For Linux, this is the python3-dev or python3-devel package, depending on your distribution.
For Windows, there are multiple options.
Python 3.6 or higher
A virtual environment (recommended, but not required)
The invoke tool

The last one may be new to you, so let's take a closer look at it.

Use the invoke tool

Invoke is the tool you will use in this tutorial to build and test Python bindings. It has a similar purpose, but uses Python instead of Makefiles. You need to invoke the following command to install in the virtual environment using pip:

$ python3 -m pip install invoke
To run it, type invoke followed by the task to be performed:

$ invoke build-cmult
==================================================
= Building C Library
* Complete

To see which tasks are available, use the following --list option:

$ invoke --list
Available tasks:

  all              Build and run all tests
  build-cffi       Build the CFFI Python bindings
  build-cmult      Build the shared library for the sample C code
  build-cppmult    Build the shared library for the sample C++ code
  build-cython     Build the cython extension module
  build-pybind11   Build the pybind11 wrapper library
  clean            Remove any built objects
  test-cffi        Run the script to test CFFI
  test-ctypes      Run the script to test ctypes
  test-cython      Run the script to test Cython
  test-pybind11    Run the script to test PyBind11

Note that when you view the tasks.py file that defines the task invoke, you will see that the name of the second task listed is build_cffi. However, the output from the --list shows it as build-cffi. Minus sign (-) cannot be used as part of a Python name, so the file uses an underscore (_) instead.

For each tool you will check, a build- and a test-task are defined. For example, to run the code CFFI, you can type invoke build-cffi test-cffi. The one exception is ctypes, because there is no build phase ctypes. In addition, for convenience, two special tasks have been added:

invoke all runs the build and test tasks of all tools.
invoke clean deletes any generated files.

Now that you have an idea of how to run the code, before looking at the tool overview, let's take a look at the C code you will be packaging.

C or C++ source code

In each example section below, you will create the Python binding in C or C++. These sections are designed to give you an experience of what each method looks like, not an in-depth tutorial about the tool, so the functions you will encapsulate are small. The function for which you will create a Python binding takes anint and afloat as input parameters and returns a float that is the product of two numbers:

// cmult.c
float cmult(int int_param, float float_param) {
    float return_value = int_param * float_param;
    printf("    In cmult : int: %d float %.1f returning  %.1f\n", int_param,
            float_param, return_value);
    return return_value;
}

C and C++ functions are almost the same, with slightly different names and strings between them. You can get a copy of all the code by clicking the link below:

Now that you have cloned the repo and installed the tools, you can build and test these tools. So let's dive into each part below!

ctypes

You will start with ctypes, which is a tool in the standard library for creating Python bindings. It provides a low-level toolset for loading shared libraries and marshalling data between Python and C.

How is it installed

One of the great advantages of ctypes is that it is part of the Python standard library. It was added in Python 2.5, so you probably already have it. You can import just like using sys or time modules.

Call functions

All the code to load the C library and call the function will be in the Python program. This is great because there are no extra steps in your process. You just need to run your program and everything will be processed. To create Python bindings in ctypes, you need to perform the following steps:

loads your library
wraps some input parameters of
tells ctypes the return type of your function.

You will look at each of them in turn.

Library loading

ctypes provides you with multiple ways to load shared libraries, some of which are platform-specific. For your example, you will directly create the object ctypes.CDLL by passing in the full path of the required shared library:

# ctypes_test.py
import ctypes
import pathlib

if __name__ == "__main__":
    # Load the shared library into ctypes
    libname = pathlib.Path().absolute() / "libcmult.so"
    c_lib = ctypes.CDLL(libname)

This works when the shared library is in the same directory as the Python script, but be careful when trying to load a library from a package other than the Python binding. In the ctypes platform-specific and situation-specific documentation, there is a lot of detailed information about loading libraries and finding paths.

Note: may have many platform-specific problems during library loading. It is best to make incremental changes after the example is working.

Now that you have loaded the library into Python, you can try to call it!

Call your function

Remember, the function prototype of your C function is as follows:

// cmult.h
float cmult(int int_param, float float_param);

You need to pass in an integer and a floating point number, and you can expect a floating point number to return. Integers and floating point numbers are natively supported in Python and C, so you want this to work for reasonable values.

After loading the library into the Python binding, the function will become the attribute of c_lib, which is the object of CDLL you created earlier. You can try to call it like this:

x, y = 6, 2.3
answer = c_lib.cmult(x, y)

Damn! This does not work. This line is commented out in the example repo because it failed. If you try to run with this call, Python will report an error:

$ invoke test-ctypes
Traceback (most recent call last):
  File "ctypes_test.py", line 16, in <module>
    answer = c_lib.cmult(x, y)
ctypes.ArgumentError: argument 2: <class 'TypeError'>: Don't know how to convert parameter 2

It looks like you need to specify any ctypes parameters that are not integers. ctypes unless you tell it explicitly, you don't know anything about the function. Any parameters that are not marked in other ways are assumed to be integers. ctypes doesn't know how to convert the value stored in 2.3 to a y integer, so it fails.

To solve this problem, you need c_float to create one from the number. You can do this in the line where the function is called:

# ctypes_test.py
answer = c_lib.cmult(x, ctypes.c_float(y))
print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

Now, when you run this code, it will return the product of the two numbers you passed in:

$ invoke test-ctypes
    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 48.0

Wait a minute... 6 times 2.3 is not 48.0!

It turns out that just like input parameters, ctypes assumes your function returns an int. In fact, your function returns a float, which is incorrectly marshalled. Just like input parameters, you need to tell ctypes to use different types. The syntax here is slightly different:

# ctypes_test.py
c_lib.cmult.restype = ctypes.c_float
answer = c_lib.cmult(x, ctypes.c_float(y))
print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

This should be enough. Let's run through the entire test-ctypes target and see what you have. Remember, the first part of the output is before restype fixes the function as a floating point number:

$ invoke test-ctypes
==================================================
= Building C Library
* Complete
==================================================
= Testing ctypes Module
    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 48.0

    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

This is better! Although the first uncorrected version returns the wrong value, your fixed version is consistent with the C function. Both C and Python get the same result! Now it works, see why you may or may not want to use ctypes.

pro and con

The biggest advantage of ctypes over other tools you will check here is that its built into the standard library . It does not require additional steps, because all the work is done as part of the Python program.

In addition, the concepts used are low-level, which makes exercises like the one you just did easy to manage. However, due to the lack of automation, more complex tasks become cumbersome. In the next section, you will see a tool that adds some automation to the process.

CFFI

CFFI is Python's C foreign function interface . Generating Python bindings requires a more automated approach. CFFI has multiple ways to build and use Python bindings. There are two different options to choose from, giving you four possible modes:

ABI vs API: The API mode uses a C compiler to generate a complete Python module, while the ABI mode loads the shared library and interacts with it directly. It is easy to make mistakes to get the correct structure and parameters without running the compiler. This document strongly recommends the use of API mode.
inline vs. : 160ef9db04c29a The difference between these two modes is the trade-off between speed and convenience:
- Every time the script is run, the inline mode compiles the Python bindings. This is convenient because you don't need additional build steps. However, it does slow down your program.
- Out-of-line mode requires an extra step to generate Python bindings once, and then use them each time the program is run. This is much faster, but it may not matter to your application.

For this example, you will use the API outreach mode, which generates the fastest code and usually looks similar to the other Python bindings you will create later in this tutorial.

How is it installed

Since CFFI is not part of the standard library, you need to install it on your machine. It is recommended that you create a virtual environment for this. Fortunately, CFFI has pip installed:

$ python3 -m pip install cffi
This will install the package into your virtual environment. If you have already installed requirements.txt from then you should pay attention to this. You can view requirements.txt by visiting the repo in the following link:

Get the sample code: Click here to get the sample code that you will use to understand the Python binding in this tutorial.

Now that you have CFFI installed, it's time to give it a try!

Call functions

Unlike ctypes, CFFI you are creating a complete Python module. You will be able to import and use the module like any other module in the standard library. You need to do some extra work to build Python modules. To use CFFIPython binding, you need to perform the following steps:

write some Python code describing the binding of
runs this code to generate a loadable module.
modify the calling code to import and use the newly created module.

This may seem like a lot of work, but you will complete each of these steps and understand how it works.

Write binding

CFFI provides C header file to complete most of the work when generating Python bindings. In the documentation of CFFI, the code to do this is placed in a separate Python file. For this example, you will put the code directly into the build tool invoke, which uses a Python file as input. To use CFFI, you must first create a cffi.FFI object, which provides the three methods you need:

# tasks.py
import cffi
...
""" Build the CFFI Python bindings """
print_banner("Building CFFI Module")
ffi = cffi.FFI()

Once you have FFI, you will use .cdef() to automatically process the contents of the header file. This will create a wrapper function for you to marshal data from Python:

# tasks.py
this_dir = pathlib.Path().absolute()
h_file_name = this_dir / "cmult.h"
with open(h_file_name) as h_file:
    ffi.cdef(h_file.read())

Reading and processing the header file is the first step. After that, you need to use .set_source() to describe the source file that CFFI will generate:

# tasks.py
ffi.set_source(
    "cffi_example",
    # Since you're calling a fully-built library directly, no custom source
    # is necessary. You need to include the .h files, though, because behind
    # the scenes cffi generates a .c file that contains a Python-friendly
    # wrapper around each of the functions.
    '#include "cmult.h"',
    # The important thing is to include the pre-built lib in the list of
    # libraries you're linking against:
    libraries=["cmult"],
    library_dirs=[this_dir.as_posix()],
    extra_link_args=["-Wl,-rpath,."],
)

The following is a breakdown of the parameters you passed in:

"cffi_example" is the base name of the source file that will be created on your file system. CFFI will generate a .c file, compile it into a .o file, and link it to a .<system-description>.so or .<system-description>.dll file.
'#include "cmult.h"' is a custom C source code, it will be included in the generated source code before compilation. Here, you only need to include the .h file for which you want to generate bindings, but this can be used for some interesting customizations.
libraries=["cmult"] tells the linker the names of your pre-existing C libraries. This is a list, so you can specify as many libraries as you need.
library_dirs=[this_dir.as_posix(),] is a directory list that tells the linker where to find the above library list.
extra_link_args=['-Wl, ,.'] 160ef9db04c82a is a set of options for generating shared objects. It will look for other libraries it needs to load in the current path (.).

Build Python binding

Calling .set_source() will not build Python bindings. It only sets metadata to describe the content that will be generated. To build the Python binding, you need to call .compile():

# tasks.py
ffi.compile()

This is done by generating .c files, .o files and shared libraries. In the invoke you just go through the task you can run on the command line to build the Python binding:

$ invoke build-cffi
==================================================
= Building C Library
* Complete
==================================================
= Building CFFI Module
* Complete

You have your CFFIPython bindings, so it's time to run this code!

Call your function

After all you have done to configure and run the CFFI compiler, using the generated Python bindings will look just like using any other Python module:

# cffi_test.py
import cffi_example

if __name__ == "__main__":
    # Sample data for your call
    x, y = 6, 2.3

    answer = cffi_example.lib.cmult(x, y)
    print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

You import the new module, and then you can call cmult() directly. To test it, use the following test-cffi task:

$ invoke test-cffi
==================================================
= Testing CFFI Module
    In cmult : int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

This will run your cffi_test.py program, which will test your use of CFFI. This concludes the part about writing and using CFFIPython binding.

pro and con

Compared with the CFFI example you just saw, ctypes seems to require less work. Although this is correct for this use case, CFFI can scale better to larger projects than ctypes due to the automation of most of the functional packaging.

CFFI also produced a completely different user experience. ctypes allows you to load pre-existing C libraries directly into your Python program. CFFI, on the other hand, creates a new Python module that can be loaded like other Python modules.

More importantly, using the external API method used above, the time loss of creating a Python binding is done once when you build it, and it does not happen every time you run the code. For small programs, this may not be a big problem, but it can also be better extended to larger projects through CFFI.

Just like ctypes, usingCFFI only allows you to directly interact with the C library. The C++ library requires a lot of work to use. In the next section, you will see a Python binding tool focused on C++.

PyBind11

PyBind11 uses a completely different method to create Python bindings. In addition to shifting the focus from C to C ++, it also using C ++ and the building blocks specified , it can be utilized in the C ++ programming tool element. Like CFFI, the generated Python binding PyBind11 is a complete Python module that can be directly imported and used.

PyBind11 is based on the Boost::Python library and has a similar interface. However, it limits its use to C++11 and newer versions, which makes it possible to simplify and speed up processing compared to Boost, which supports everything.

How is it installed
The "First Steps" section of the document will guide you through PyBind11 on how to download and build PyBind11. Although this may not seem to be a strict requirement, completing these steps will ensure that you have the correct C++ and Python tools set up.

Note : Most examples of PyBind11 use cmake, which is a good tool for building C and C++ projects. However, for this demo, you will continue to use the invoke tool, which follows the instructions in the manual build section of the documentation.

You need to install this tool into your virtual environment:

$ python3 -m pip install pybind11
PyBind11 is a full-head library, similar to most of Boost. This allows pip to install the actual C++ source code of the library directly into your virtual environment.

Call functions

Before you dive into it, please note that you are using a different C++ source file , cppmult.cpp, not the C file you used for the previous example. The functions of the two languages are basically the same.

Write binding

Similar to CFFI, you need to create some code to tell the tool how to build your Python bindings. Unlike CFFI, this code will use C++ instead of Python. Fortunately, very little code is required:

// pybind11_wrapper.cpp
#include <pybind11/pybind11.h>
#include <cppmult.hpp>

PYBIND11_MODULE(pybind11_example, m) {
    m.doc() = "pybind11 example plugin"; // Optional module docstring
    m.def("cpp_function", &cppmult, "A function that multiplies two numbers");
}

Let's look at it one at a time, because PyBind11 packs a lot of information into a few lines.

The first two lines include the files of the pybind11.hC++ library and the header file cppmult.hpp. After that, you have the PYBIND11_MODULE macro. This will expand to the C++ code block detailed in the PyBind11 source code:

This macro creates an entry point, which will be called when the Python interpreter imports an extension module. The module name is given as the first parameter and should not be enclosed in quotation marks. The second macro parameter defines a py::module type variable that can be used to initialize the module. (source)

This means to you that, in this example, you are creating a module called pybind11_example, and the rest of the code uses m as the name of the py::module object. On the next line, in the C++ function you defined, you create a docstring for the module. Although this is optional, it is a good choice to make your module more Pythonic.

Finally, you have the m.def() call. This will define a function exported by your new Python binding, which means it will be visible in Python. In this example, you will pass three parameters:

cpp_function is the exported name of the function you will use in Python. As shown in this example, it does not need to match the name of the C++ function.
&cppmult Get the address of the function to be exported.
"A function..." is an optional documentation string for the function.

Now that you have the Python binding code, let's take a look at how to build it into a Python module.

Build Python binding

The tool PyBind11 used to build Python bindings is the C++ compiler itself. You may need to modify the default values of the compiler and operating system.

First, you must build the C++ library for which you want to create bindings. For such a small example, you can build the cppmult library directly into the Python binding library. However, for most practical examples, you will have a pre-existing library to be packaged, so you will build that library separately by cppmult. Build is a standard call to the compiler to build a shared library:

# tasks.py
invoke.run(
    "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC cppmult.cpp "
    "-o libcppmult.so "
)

Run this invoke build-cppmult to generate libcppmult.so:

$ invoke build-cppmult
==================================================
= Building C++ Library
* Complete

On the other hand, the construction of the Python binding requires some special details:

1# tasks.py
 2invoke.run(
 3    "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC "
 4    "`python3 -m pybind11 --includes` "
 5    "-I /usr/include/python3.7 -I .  "
 6    "{0} "
 7    "-o {1}`python3.7-config --extension-suffix` "
 8    "-L. -lcppmult -Wl,-rpath,.".format(cpp_name, extension_name)
 9)

Let's go through it line by line. line 3 contains fairly standard C++ compiler flags indicating several details, including you want to catch all warnings and treat them as errors, you need shared libraries, and you are using C++11.

line 4 is the first step of the magic. It calls the pybind11 module to include PyBind11. You can run this command directly on the console to see what it does:

$ python3 -m pybind11 --includes
-I/home/jima/.virtualenvs/realpython/include/python3.7m
-I/home/jima/.virtualenvs/realpython/include/site/python3.7

Your output should be similar but show a different path.

In the fifth line called by compiling, you can see that you have also added the path includes for Python dev. Although it is recommended that you do not link the Python library itself, the source code requires some code Python.h to perform its magic. Fortunately, the code it uses is fairly stable in the Python version.

Line 5 is also used for -I. Add the current directory to the include path list. This allows #include <cppmult.hpp> to parse lines in the wrapper code.

line 6 specifies the name of the source file, which is pybind11_wrapper.cpp. Then, on line , and you will see more build magic is happening. This line specifies the name of the output file. Python has some special ideas in module naming, including Python version, machine architecture and other details. Python also provides a tool to help solve this problem python3.7-config:

$ python3.7-config --extension-suffix
.cpython-37m-x86_64-linux-gnu.so

If you are using a different version of Python, you may need to modify this command. If you use a different version of Python or on a different operating system, your results may change.

The last line of the build command, , line 8 , points the linker to the libcppmult library you built earlier. The rpath part tells the linker to add information to the shared library to help the operating system libcppmult find it at runtime. Finally, you will notice that the format of this string is cpp_name and extension_name. You will use this function again when Cython builds the Python binding module in the next section.

Run this command to build the binding:

$ invoke build-pybind11
==================================================
= Building C++ Library
* Complete
==================================================
= Building PyBind11 Module
* Complete

That's it! You are already using PyBind11. It's time to test it out!

Call your function

Similar to the CFFI example above, once you have completed the heavy lifting of creating Python bindings, calling your function looks like normal Python code:

# pybind11_test.py
import pybind11_example

if __name__ == "__main__":
    # Sample data for your call
    x, y = 6, 2.3

    answer = pybind11_example.cpp_function(x, y)
    print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

Since your pybind11_example is used as the name of the module in the PYBIND11_MODULE macro, this is the name you imported. In the call to m.def() you tell PyBind11 to export the cppmult function as cpp_function, this is the method you use to call it from Python.

You can also test it invoke:


$ invoke test-pybind11
==================================================
= Testing PyBind11 Module
    In cppmul: int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

This is what PyBind11 looks like. Next, you will learn when and why PyBind11 is the right tool for the job.

pro and con

PyBind11 focuses on C++ instead of C, which makes it different from ctypes and CFFI. It has several features that make it very attractive to C++ libraries:

It supports classes.
It handles polymorphic subclassing.
It allows you to add dynamic attributes to objects from Python and many other tools, which is difficult to do with the C-based tools you have checked.

That being said, you need to do a lot of setup and configuration to get PyBind11 up and running. It can be a bit finicky to install and build correctly, but once it's done, it seems pretty reliable. In addition, PyBind11 requires you to use at least C++11 or higher. For most projects, this is unlikely to be a big limitation, but it may be a consideration for you.

Finally, the additional code that needs to be written to create Python bindings is written in C++, not Python. This may or may not be your problem, but it is different than the other tools you see here. In the next section, you will continue to discuss Cython, which uses a completely different approach to solve this problem.

Cython

This method Cython needs to create a Python binding and use like Python language to define the binding, and then the generated C or C++ code can be compiled into a module. There are several ways to use Cython. The most common one is to use setupfrom distutils. For this example, you will stick to the invoke tool, which allows you to use the exact command run.

How is it installed

Cython is a Python module that can be installed into your virtual environment from PyPI:

$ python3 -m pip install cython
Similarly, if you have installed the requirements.txt file into the virtual environment, the file already exists. You can obtain a copy of requirements.txt by clicking the following link:

Get the sample code: Click here to get the sample code you will use to understand the Python binding in this tutorial.

This should prepare you to work with Cython!

Call functions

To bind Cython with Build Python, you will follow similar steps to those used for CFFI and PyBind11. You will write the bindings, build them, and then run the Python code to call them. Cython can support C and C++ at the same time. For this example, you will use the cppmult library you used in the PyBind11 example above.

Write binding

Cython, the most common form of declaring a module, is to use a .pyx file:

1# cython_example.pyx
 2""" Example cython interface definition """
 3
 4cdef extern from "cppmult.hpp":
 5    float cppmult(int int_param, float float_param)
 6
 7def pymult( int_param, float_param ):
 8    return cppmult( int_param, float_param )

There are two parts here:

lines 3 and 4 v. Cython You are using cppmult() from cppmult.hpp.
lines 6 and 7 create a wrapper function pymult() to call cppmult().

The language used here is a special combination of C, C++ and Python. However, it looks fairly familiar to Python developers, because its goal is to make the process easier.

The first part withcdef extern... tells Cython that the following function declarations can also be found in the cppmult.hpp file. This is useful for ensuring that Python bindings are built according to the same declarations as C++ code. The second part looks like a normal Python function-because it is! This section creates a Python function cppmult that can access C++ functions.

Now that you have defined the Python bindings, it is time to build them!

Build Python binding

The build process Cython is similar to the build process you use PyBind11. You first run Cython on the .pyx file to generate the .cpp file. Once this is done, you can compile PyBind11 using the same functions used for:

 1# tasks.py
 2def compile_python_module(cpp_name, extension_name):
 3    invoke.run(
 4        "g++ -O3 -Wall -Werror -shared -std=c++11 -fPIC "
 5        "`python3 -m pybind11 --includes` "
 6        "-I /usr/include/python3.7 -I .  "
 7        "{0} "
 8        "-o {1}`python3.7-config --extension-suffix` "
 9        "-L. -lcppmult -Wl,-rpath,.".format(cpp_name, extension_name)
10    )
11
12def build_cython(c):
13    """ Build the cython extension module """
14    print_banner("Building Cython Module")
15    # Run cython on the pyx file to create a .cpp file
16    invoke.run("cython --cplus -3 cython_example.pyx -o cython_wrapper.cpp")
17
18    # Compile and link the cython wrapper library
19    compile_python_module("cython_wrapper.cpp", "cython_example")
20    print("* Complete")

You first run cython your .pyx file. You can use several options on this command:

--cplus tells the compiler to generate C++ files instead of C files.
-3 switches Cython to generate Python 3 syntax instead of Python 2.
-o cython_wrapper.cpp specifies the name of the file to be generated.

After generating the C++ file, you can use the C++ compiler to generate Python bindings, just like you did for PyBind11. Please note that the call to include using the pybind11 tool to generate additional paths is still in the function. There will be no harm here, because your source does not need these.

Running this task in invoke will produce the following output:

$ invoke build-cython
==================================================
= Building C++ Library
* Complete
==================================================
= Building Cython Module
* Complete

You can see that it built the cppmult library, and then built the cython module to wrap it. Now you have CythonPython bindings. (Try to say it is fast...) It's time to test it!

Call your function

The Python code that calls the new Python binding is very similar to the code used to test other modules:

1# cython_test.py
 2import cython_example
 3
 4# Sample data for your call
 5x, y = 6, 2.3
 6
 7answer = cython_example.pymult(x, y)
 8print(f"    In Python: int: {x} float {y:.1f} return val {answer:.1f}")

Line 2 imports the new Python binding module, and pymult() is called on line 7. Remember that the .pyx file provides a Python wrapper cppmult() and renamed it to pymult. Using invoke to run your test will produce the following results:

$ invoke test-cython
==================================================
= Testing Cython Module
    In cppmul: int: 6 float 2.3 returning  13.8
    In Python: int: 6 float 2.3 return val 13.8

You get the same result as before!

pro and con

Cython is a relatively complex tool that can provide you with deeper control when creating Python bindings for C or C++. Although you haven't covered it in depth here, it provides a Python-style way to write code that manually controls the GIL, which can significantly speed up the processing of certain types of problems.

However, this Python-style language is not entirely Python, so there is a slight learning curve when you want to quickly determine which parts of C and Python fit and where.

in conclusion

Congratulations! You now have an overview of the different options for creating Python bindings. You have learned about marshalling data and the issues to consider when creating bindings. You have learned how to call C or C++ functions from Python using the following tools:

ctypes
CFFI
PyBind11
Cython

You now know that although ctypes allows you to load DLLs or shared libraries directly, the other three tools require additional steps but still create complete Python modules. As a bonus, you also used the invoke tool to run command line tasks from Python.

Click to follow, and get to know the fresh technology of

Python binding: call C or C++ from Python

Overview of Python Bindings

Marshalling data type

Understand mutable and immutable values

Manage memory

Set up your environment

Use the invoke tool

C or C++ source code

ctypes

How is it installed

Call functions

Library loading

Call your function

pro and con

CFFI

How is it installed

Call functions

Write binding

Build Python binding

Call your function

pro and con

PyBind11

Call functions

Write binding

Build Python binding

Call your function

pro and con

Cython

How is it installed

Call functions

Write binding

Build Python binding

Call your function

pro and con

Other solutions

PyBindGen

Boost.Python

SIP

Cppyy

Shiboken

SWIG

in conclusion

华为云开发者联盟

引用和评论

华为云开发者联盟入选 2023 中国技术品牌影响力企业榜，深耕开发者生态

Jerry和您聊聊Chrome开发者工具

将 Python 和 Rust 融合在一起，为 pyQuil® 4.0 带来和谐

环境搭建 | Windows中MinGW-w64及GCC的下载、安装与配置

Ubuntu20.04 从源代码编译安装 python3.10

Python装饰器：让你的代码优雅又高效的秘密武器

从javascript到python(一):基本环境搭建