python测试脚本运行到3万多个子线程时出现`thread.error: start new thread`

在公司服务器上运行了一个mosquitto server端,并在服务器上运行如下python脚本创建10万个moquitto_sub客户端(公司以前人写的)连接到服务器:

#!/usr/bin/env python
#coding=utf-8
import os
import threading
from time import ctime,sleep
import subprocess

process = []

def mqtt_sub_test(func):
#    print "mosquitto thread %d" %(func)
#    os.system("mosquitto_sub -h 120.237.96.34 -t rock_test_mosquitto -k 10")
    user = "rock_mosquitto_id-3w-%d" %(func)
    print user

    p = subprocess.Popen(["./mosquitto_sub", "-h", "127.0.0.1", "-t", "llltest", "-k", "10"])
        process.append(p)
    print "process.append %d" %(func)

threads = []

for i in range(1,100000):
    t = threading.Thread(target=mqtt_sub_test,args=(i,))
    threads.append(t)

if __name__ == '__main__':
    for t in threads:
        t.setDaemon(True)
        t.start()
    sleep(0.01)

    raw_input('q')

    for sub in process:
        print sub.pid
        a =sub.kill()
    if (a!= None):
        print a

    print "all over %s" %ctime()

但是当创建子线程数大概达到32300个的时候,程序报错:错误信息如下

 Traceback (most recent call last):
  File "mqtt_python_3w.py", line 45, in <module>
      t.start()
        File "/usr/lib/python2.6/threading.py", line 474, in start
            _start_new_thread(self.__bootstrap, ())
            thread.error: can't start new thread
Python subprocess.Popen “OSError: [Errno 12] Cannot allocate memory”

此时运行系统命令诸如ls等,都会出现Cannot allocate memory的错误。
查看下系统内存,16G的系统内存使用了4G,内存是足够的,运行ulimit -a输出信息如下

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

现在想解决的问题如下:

  • 可以修改系统那些参数来扩展最大线程数?

  • 或者说如何写测试程序

诸位前辈的任何回答将不甚感激,谢谢了

阅读 12.2k
2 个回答

能看看subprocess怎么写的嘛?

撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
推荐问题