多处理导致 Python 崩溃并给出一个错误，当 fork() 被调用时，可能已经在另一个线程中进行了

Question

我对 Python 比较陌生，正在尝试为我的 for 循环实现一个多处理模块。

我有一组图像 url 存储在 img_urls 中，我需要下载并应用一些 Google 视觉。

 if __name__ == '__main__':

    img_urls = [ALL_MY_Image_URLS]
    runAll(img_urls)
    print("--- %s seconds ---" % (time.time() - start_time))

这是我的 runAll() 方法

def runAll(img_urls):
    num_cores = multiprocessing.cpu_count()

    print("Image URLS  {}",len(img_urls))
    if len(img_urls) > 2:
        numberOfImages = 0
    else:
        numberOfImages = 1

    start_timeProcess = time.time()

    pool = multiprocessing.Pool()
    pool.map(annotate,img_urls)
    end_timeProcess = time.time()
    print('\n Time to complete ', end_timeProcess-start_timeProcess)

    print(full_matching_pages)

def annotate(img_path):
    file =  requests.get(img_path).content
    print("file is",file)
    """Returns web annotations given the path to an image."""
    print('Process Working under ',os.getpid())
    image = types.Image(content=file)
    web_detection = vision_client.web_detection(image=image).web_detection
    report(web_detection)

当我运行它并且 python 崩溃时，我收到了这个警告

objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

原文由 SriTeja Chilakamarri 发布，翻译遵循 CC BY-SA 4.0 许可协议

Answer 1

出现此错误是因为在 macOS High Sierra 和更高版本的 macOS 中增加了限制多线程的安全性。我知道这个答案有点晚了，但我使用以下方法解决了这个问题：

设置环境变量 .bash_profile （或 .zshrc 对于最新的 macOS）以允许在新的 macOS High Sierra 安全规则下运行多线程应用程序或脚本。

打开一个终端：

 $ nano .bash_profile

将以下行添加到文件末尾：

 OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

保存，退出，关闭终端并重新打开终端。检查环境变量是否已设置：

 $ env

您将看到类似于以下内容的输出：

 TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/pn/vasdlj3ojO#OOas4dasdffJq/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.E7qLFJDSo/Render
TERM_PROGRAM_VERSION=404
TERM_SESSION_ID=NONE
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

您现在应该能够使用多线程运行您的 Python 脚本。

原文由 raw-bin hood 发布，翻译遵循 CC BY-SA 4.0 许可协议

Answer 2

其他答案告诉您设置 OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES ，但不要这样做！你只是在警示灯上贴了胶带。对于某些遗留软件，您可能需要根据具体情况进行设置，但肯定不会在您的 .bash_profile 中进行设置！

这已在 https://bugs.python.org/issue33725 (python3.8+) 中修复，但最好使用

with multiprocessing.get_context("spawn").Pool() as pool:
    pool.map(annotate,img_urls)

原文由 Thomas Grainger 发布，翻译遵循 CC BY-SA 4.0 许可协议