用途:爬虫(框架:scrapy)
cap = dict(DesiredCapabilities.PHANTOMJS)
cap["phantomjs.page.settings.resourceTimeout"] = 1000
cap["phantomjs.page.settings.loadImages"] = False
cap["phantomjs.page.settings.webSecurityEnabled"] = False
driver = webdriver.PhantomJS(executable_path='/usr/local/bin/phantomjs',desired_capabilities=cap)
driver.set_page_load_timeout(20)
driver.set_script_timeout(20)
try:
driver.get(request.url)
except Exception as e:
print e
finally:
driver.quit()
发现phantomjs经常quit不掉,以致cpu越来越高,pgrep phantomjs,phantomjs一大堆。
个人认为,driver.get(request.url),一旦遇到js的坑,就会死在这儿,从而phantomjs也就无法quit。
请问大家有无好办法?