pyspider0.3.8 crawl_config 参数设置不生效

我试了一下官方部署的DEMO是没有问题的,我在Win和Linux下都是新安装部署的都不行,旧版的就可以。。。为什么?
部署环境:(Win8.1+Python2.7 CentOS6.8+Python2.7)
配置如下:

class Handler(BaseHandler):
    crawl_config = {
        "proxy":"172.16.2.2:3128",
        "headers":{
            "Proxy-Connection": "keep-alive",
            "Pragma": "no-cache",
            "Cache-Control": "no-cache",
            "User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36",
            "Accept": "*/*",
            "DNT": "1",
            "Accept-Encoding": "gzip, deflate, sdch",
            "Accept-Language": "zh-CN,zh;q=0.8,en-US;q=0.6,en;q=0.4",

        }
    } 
    ...

执行的时候如下,fetch是空的:

{
  "fetch": {},
  "process": {
    "callback": "index_page"
  },
  "project": "news_top",
  "schedule": {
    "age": 864000
  },
  "taskid": "73804bb407415e0129e69659e1678264",
  "url": "http://www.cwrank.com/"
}
阅读 4.5k
2 个回答
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进