请问:为什么Python自带的Multi-processing模块需要一定时间才能启动?

我用Python自带的Multi-processing模块做了一个并发的Async的系统。只有2个任务,一快一慢,并发执行,快的等慢的。

import sys, os, time, datetime
from timeit import timeit
import threading
import multiprocessing
from multiprocessing import Pool

def multiprocessor_hack(instance, name, args=(), kwargs=None):
    if kwargs is None:
        kwargs = {}
    return getattr(instance, name)(*args, **kwargs)

class Controller:
    request = "Whatever"
    def task1(self, request):
        print ("Now starting task1 at {}.\n".format(datetime.datetime.now()))
        time.sleep(3)
        print ("Now finished task1 at {}.\n".format(datetime.datetime.now()))
        return "111111"
    def task2(self, request):
        print ("Now starting task2 at {}.\n".format(datetime.datetime.now()))
        time.sleep(10)
        print ("Now finished task2 at {}.\n".format(datetime.datetime.now()))
        return "222222"

    def moderate(self, request):
        pool = multiprocessing.Pool(processes = 2)
        results = [pool.apply_async(multiprocessor_hack, args = (self, task1', (request,))), pool.apply_async(multiprocessor_hack, args = (self, 'task2', (request,)))]
        pool.close()
        map (multiprocessing.pool.ApplyResult.wait, results)
        response = [r.get() for r in results]
        return response


if "__main__" == __name__:
    ctrl = Controller()
    print ("\nThe pool starts at {}.\n".format(datetime.datetime.now()))
    response = ctrl.moderate("Whatever")
    print ("\nResponse emerges at {}.".format(datetime.datetime.now()))
    print ("\nThe pool ends at {}.".format(datetime.datetime.now()))

运行结果是:

The pool starts at 2018-03-23 15:03:51.187000.
Now starting task1 at 2018-03-23 15:03:51.522000.  # 延迟?
Now starting task2 at 2018-03-23 15:03:51.522000.  # 延迟?
Now finished the task1 at 2018-03-23 15:03:54.524000.
Now finished the task2 at 2018-03-23 15:04:01.524000.
Response emerges at 2018-03-23 15:04:01.526000.
The pool ends at 2018-03-23 15:04:01.528000.

这里,pool在下午三点 15:03:51.187 开始运行,但是2个任务直到 15:03:51.522 才开始执行!… 请问这里为什么会延迟0.4秒呢?有没有什么办法减小这个延迟?

谢谢了先!

阅读 4.2k
3 个回答

开多进程要fork,开销算是非常大的,相当于你重新打开了一个python。可以不开多进程就不开多进程,可以用线程代替进程,CPython还有GIL这种东西,有时候开了多进程或多线程CPU利用率反倒会降低(调度和规划没做好的话)。

猜测可能是在编译class,编译时又要做导入,所以慢。你可以试试把py编译一下看看是否有提升

我搜了一下,Multi-processing模块需要一定时间在main process和work process之间建立通信,并且由系统分配资源,所以需要一定时间来启动。

撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
推荐问题