Mac m1 arm install confluent-kafka error solution

                    'c.          [e-mail] 
                 ,xNMM.          ----------------- 
               .OMMMMo           OS: macOS 11.6 20G165 arm64 
               OMMM0,            Host: MacBookPro17,1 
     .;loddo:' loolloddol;.      Kernel: 20.6.0 
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 3 days, 22 hours, 18 mins 
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 44 (brew) 
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: zsh 5.8 
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 1440x900, 2560x1440 
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua 
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Quartz Compositor 
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    WM Theme: Blue (Light) 
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal: pycharm 
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   CPU: Apple M1 
    kMMMMMMMMMMMMMMMMMMMMMMd     GPU: Apple M1 
     ;KMMMMMMMWXXWMMMMMMMk.      Memory: 1616MiB / 8192MiB 
       .cooc,.    .,coo:.
                                                         
                                                         

There is no problem on Linux, but not on Mac. Installing confluent-kafka will report the error that the relevant header file cannot be found, because confluent-kafka uses librdkafka written in C language as the basis, and the poor header file management of Mac makes everything become bad.

I got an error when I installed confluent-kafka using the following command

pip install confluent-kafka

The error content is as follows:

ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-install-pura_lpm/confluent-kafka_889dcb8c9d304f88b93b91571870ef8e/setup.py'"'"'; __file__='"'"'/private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-install-pura_lpm/confluent-kafka_889dcb8c9d304f88b93b91571870ef8e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-record-7jy58dai/install-record.txt --single-version-externally-managed --compile --install-headers /Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/confluent-kafka
         cwd: /private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-install-pura_lpm/confluent-kafka_889dcb8c9d304f88b93b91571870ef8e/
...
 In file included from /private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-install-pura_lpm/confluent-kafka_889dcb8c9d304f88b93b91571870ef8e/src/confluent_kafka/src/Admin.c:17:
    /private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-install-pura_lpm/confluent-kafka_889dcb8c9d304f88b93b91571870ef8e/src/confluent_kafka/src/confluent_kafka.h:23:10: fatal error: 'librdkafka/rdkafka.h' file not found
    #include <librdkafka/rdkafka.h>
             ^~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-install-pura_lpm/confluent-kafka_889dcb8c9d304f88b93b91571870ef8e/setup.py'"'"'; __file__='"'"'/private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-install-pura_lpm/confluent-kafka_889dcb8c9d304f88b93b91571870ef8e/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/3r/fkd7zds92j16b43nby2yv8s80000gn/T/pip-record-7jy58dai/install-record.txt --single-version-externally-managed --compile --install-headers /Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/confluent-kafka Check the logs for full command output.

Workaround, install librdkafka

brew install librdkafka

Light is not enough, we still can't find the header file at this time, we also need to let pip find where librdkafka is (Linux please ignore)

Looks like Python is not making use of the homebrew installation location (which should be linked to /usr/local/), try fixing the homebrew package with brew link --overwrite librdkafka and if that doesn't work you can force it in the right direction by:

C_INCLUDE_PATH=/usr/local/Cellar/librdkafka/0.11.0/include LIBRARY_PATH=/usr/local/Cellar/librdkafka/0.11.0/lib pip install confluent_kafka

So you need to set two environment variables C_INCLUDE_PATH and LIBRARY_PATH

🚫 The librdkafka version I installed is 1.7.0 , remember to replace the version number in the command below with your own

C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.7.0/include LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.7.0/lib pip install confluent_kafka

TIPS: The above is a line of commands, do not disassemble and execute, otherwise it will be invalid

With these two environment variables, pip can correctly find where the header files are!

Reference article:
linux gcc header file search path
Install librdkafka dependency on Mac OSX