程序占据过多内存而killed掉,推测应该由于permutations生成了过大数据量,求优化建议。

新手上路,请多包涵

想问问各位大神:

1. 程序如何优化比较好?感觉自己的代码执行效率很低,但不清楚怎么做会更好/
2. 此段程序在数据量少的时候是没有问题的,有没有可能并行?并行的话,需要怎么改呢?

代码如下:

from __future__ import division
from numpy import matrix
from itertools import combinations,permutations
import copy
import pp
import time

A = [(1, 0, 0, 0, 0), (0, 1, 0, 0, 0), (0, 0, 1, 0, 0), (0, 0, 0, 1, 0), (0, 0, 0, 0, 1), (1, 1, 0, 0, 0), (1, 0, 1, 0, 0), (1, 0, 0, 1, 0), (1, 0, 0, 0, 1), (1, -1, 0, 0, 0), (1, 0, -1, 0, 0), (1, 0, 0, -1, 0), (1, 0, 0, 0, -1), (0, 1, 1, 0, 0), (0, 1, 0, 1, 0), (0, 1, 0, 0, 1), (0, 1, -1, 0, 0), (0, 1, 0, -1, 0), (0, 1, 0, 0, -1), (0, 0, 1, 1, 0), (0, 0, 1,0, 1), (0, 0, 1, -1, 0), (0, 0, 1, 0, -1), (0, 0, 0, 1, 1), (0, 0, 0, 1, -1), (1, 1, 1, 0, 0), (1, 1, 0, 1, 0), (1, 1, 0, 0, 1), (1, -1, 1, 0, 0), (1, -1, 0, 1, 0), (1, -1, 0, 0, 1), (1, 1, -1, 0, 0), (1, 1, 0, -1, 0), (1, 1, 0, 0, -1), (1, -1, -1, 0, 0), (1, -1, 0, -1, 0), (1, -1, 0, 0, -1), (1, 0, 1, 1, 0), (1, 0, 1, 0, 1), (1, 0, -1, 1, 0), (1, 0, -1, 0, 1), (1, 0, 1, -1, 0), (1, 0, 1, 0, -1), (1, 0, -1, -1, 0), (1, 0, -1, 0, -1), (1, 0, 0, 1, 1), (1, 0, 0, -1, 1), (1, 0, 0, 1, -1), (1, 0, 0, -1, -1), (0, 1, 1, 1, 0), (0, 1, -1, 1, 0), (0, 1,1, -1, 0), (0, 1, -1, -1, 0), (0, 1, 1, 0, 1), (0, 1, -1, 0, 1), (0, 1, 1, 0, -1), (0, 1, -1, 0, -1), (0, 1, 0, 1, 1), (0, 1, 0, -1, 1), (0, 1, 0, 1, -1), (0, 1, 0, -1, -1), (0, 0, 1, 1, 1), (0, 0, 1, -1, 1), (0, 0, 1, 1, -1), (0, 0, 1, -1, -1), (1, 1, 1, 1, 0), (1, 1, 1, -1, 0), (1, 1, -1, 1, 0), (1, 1, -1, -1, 0), (1, -1, 1, 1, 0), (1, -1, 1, -1, 0), (1, -1, -1, 1, 0), (1, -1, -1, -1, 0), (1, 1, 1, 0, 1), (1, 1, 1, 0, -1), (1, 1, -1, 0, 1), (1, 1, -1, 0, -1), (1, -1, 1, 0, 1), (1, -1, 1, 0, -1), (1, -1, -1, 0, 1), (1, -1, -1, 0, -1), (1, 1, 0, 1, 1), (1, 1, 0, 1, -1), (1, 1, 0, -1, 1), (1, 1, 0, -1, -1), (1, -1, 0, 1, 1), (1, -1, 0, 1, -1), (1, -1, 0, -1, 1), (1, -1, 0, -1, -1), (1, 0, 1, 1, 1), (1, 0, 1, 1, -1), (1, 0, 1, -1, 1), (1, 0, 1, -1, -1), (1, 0, -1, 1, 1), (1, 0, -1, 1, -1), (1, 0, -1, -1, 1), (1, 0, -1, -1, -1), (0, 1,1, 1, 1), (0, 1, 1, 1, -1), (0, 1, 1, -1, 1), (0, 1, 1, -1, -1), (0, 1, -1, 1, 1), (0, 1, -1, 1, -1), (0, 1, -1, -1, 1), (0, 1, -1, -1, -1), (1, 1, 1, 1, 1), (1, 1, 1, 1, -1), (1, 1, 1, -1, 1), (1, 1, -1, 1, 1), (1, -1, 1, 1, 1), (1, -1, -1, 1, 1), (1, -1, 1, -1, 1), (1, -1, 1, 1, -1), (1, 1, -1, -1, 1), (1, 1, -1, 1, -1), (1, 1, 1, -1, -1), (1, -1, -1, -1, 1), (1, -1, 1, -1, -1), (1, -1, -1, 1, -1), (1, 1, -1, -1, -1), (1, -1, -1, -1, -1)] 

Energy=[4.56e-09,8.28e-09,1.75e-08,2.31e-08,2.59e-08,3.34e-08,3.98e-08,4.64e-08,5.96e-08,6.89e-08,8.18e-08,9.53e-08,1.12e-07,1.18e-07,1.29e-07,1.34e-07,1.41e-07,1.51e-07,1.56e-07,1.67e-07,1.72e-07,1.77e-07,1.82e-07,1.89e-07,1.93e-07,1.97e-07,2.00e-07,2.09e-07,2.22e-07,2.44e-07,2.66e-07,2.79e-07,2.87e-07,2.98e-07,3.05e-07,3.09e-07,3.15e-07,3.23e-07,3.29e-07,3.32e-07,3.47e-07,3.50e-07,3.68e-07,3.71e-07,3.80e-07,3.87e-07,4.12e-07,4.16e-07,4.45e-07,4.53e-07,4.57e-07,4.62e-07,4.78e-07,4.80e-07,4.83e-07,4.99e-07,5.03e-07,5.21e-07,5.24e-07,5.41e-07,5.47e-07,5.64e-07,6.14e-07,6.31e-07,6.35e-07,6.38e-07,6.53e-07,6.56e-07,6.70e-07,6.75e-07,6.78e-07,7.14e-07,7.19e-07,7.24e-07,7.87e-07,7.90e-07,7.97e-07,8.10e-07,8.20e-07,8.26e-07,8.29e-07,8.32e-07,8.41e-07,8.69e-07,8.73e-07,9.36e-07,9.39e-07,9.77e-07,9.99e-07,1.00e-06,1.01e-06,1.02e-06,1.03e-06,1.08e-06,1.13e-06,1.14e-06,1.18e-06,1.18e-06,1.21e-06,1.22e-06,1.31e-06,1.31e-06,1.46e-06,1.48e-06,1.77e-06,1.92e-06,2.14e-06,2.14e-06,2.38e-06,2.86e-06,3.12e-06,3.35e-06,3.35e-06,3.81e-06,3.89e-06,4.09e-06,4.57e-06,6.90e-06,8.43e-06,8.70e-06,8.71e-06]     #PeakPosition

 ener=list(permutations(Energy[0:121],5))
num=121
enum=121
Dim=len(ener)##obtaincombination
a1,a2,a3,a4,a5=[0,0,0,0,0]
def calculation_tang(A,Ener,Energy,num,enum):
     id_ener = []
     cmp_score = []
     Res = [[0 for j in xrange(121)] for k in xrange(121)]
     Dim = len(Ener) ##obtain combination
     for i in xrange(Dim)
         X = 0
         Y = 0
         Z = 0
         L = 0
         M = 0
         ss= 0
         res_id = 0
         resnew = []
         score = 0

         fft = matrix(A) * matrix(Ener[i]).T
     
         for k in xrange(num):
             for j in xrange(num):
                 Res[j][k] = abs((fft[j]-Energy[k])/Energy[k])
 
         for l in xrange(num):
             resnew.append(min(Res[l][:]))
             res_id = Res[l][:].index(min(Res[l][:]))
             
             for m in xrange(l+1,num):
                 Res[m][res_id]=100
 
         for x in xrange(len(resnew)):
     
              if resnew[x] <= 5e-3: X += 1
         
              elif 5e-3 < resnew[x]<= 1e-2: Y += 1
             
              elif 1e-2 < resnew[x] <= 5e-2: Z += 1
                 
              elif 5e-2 < resnew[x] <=1e-1: L += 1
                     
              else: M +=1
       
         ss = X + Y +  Z + M + L
         score = (X /ss *0.5 + Y /ss *0.25 + Z /ss *0.13 + L /ss *0.08 + M /ss *0.04)*100
     
         if X + Y >= (num/3.0) :
             cmp_score.append(score)
             id_ener.append(i)
         return id_ener,cmp_score


start=time.time()
job_server=pp.Server()
jobs=[job_server.submit(calculation_tang,(A,ener,Energy,num,enum,),(),())]
r_job=calculation_tang(A,ener,Energy,num,enum)

index_energy=r_job[0]
score_choice=r_job[1]
f=open('value5_180122_try_permutation.txt','w')
f.write("AllresultssatisfyX+Y>141/3:n")
for xx in xrange(len(index_energy)):
    a,b,c,d,e = Ener[index_energy[xx]]
    f.write("%3.4e %3.4e %3.4e %3.4e %f\n" % (a,b,c,d,score_choice[xx]))
    a1,a2,a3,a4,a5 =  Ener[index_energy[score_choice.index(max(score_choice))]]
f.write("the highest score is %3.4e %3.4e %3.4e %3.4e %3.4e %f \n" % (a1,a2,a3,a4,a5,max(score_choice)))
elapsed = (time.time()-start)
f.write("Time used: %5.4f  \n" % elapsed)
f.close()
print('Finished.')

被killed后,查询原因如下:
Out of memory: Kill process 11025 (python) score 988 or sacrifice child
Killed process 11025, UID 503, (python) total-vm:396447360kB, anon-rss:391521192kB, file-rss:8kB

非常感谢各位的帮助!

阅读 3.2k
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
推荐问题