pyspider调试无错，但自动运行没有结果

Question

1.写一个pyspider的脚本，调试运行无错，也可以往数据库插入，但是自动运行第一次成功后，以后再也无法运行成功，提示信息都是success，但就是没有插入数据
代码如下：

!/usr/bin/env python

-- encoding: utf-8 --

Created on 2018-04-27 11:58:40

Project: tripadvistor_bj

from pyspider.libs.base_handler import *
import pymongo

class Handler(BaseHandler):

crawl_config = {
    'headers': {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.98 Safari/537.36 LBBROWSER'
    }
}

client = pymongo.MongoClient('localhost')
db = client['trip']

@every(minutes=24 * 60)
def on_start(self):
    self.crawl('https://www.tripadvisor.cn/Attractions-g294212-Activities-Beijing.html', callback=self.index_page ,validate_cert=False, auto_recrawl = True)

@config(age=10 * 24 * 60 * 60)
def index_page(self, response):
    for each in response.doc('.listing_info .listing_title a').items():
        self.crawl(each.attr.href, callback=self.detail_page, validate_cert=False)
        
    next = response.doc('.pagination .nav.next').attr.href
    self.crawl(next,callback = self.index_page, validate_cert=False)
    
@config(priority=2)
def detail_page(self, response):
    
    name = response.doc('#HEADING').text() + response.doc('#HEADING > span').text()
    addr = response.doc('span.street-address').text()
    type = response.doc('span.header_detail.attraction_details > div > a:nth-child(1)').text()
    phone = response.doc('div.blEntry.phone > span:nth-child(2)').text()
    desc = response.doc('div.prw_rup.prw_common_location_description > div > div.text').text()
    
    return {
        "url": response.url,
        "title": response.doc('title').text(),
        "name": name,
        "addr": addr,
        "type": type,
        "tel": phone,
        "desc": desc
    }

def on_result(self,result):
    if result:
        self.save_to_mongo(result)
            
def save_to_mongo(self,result):
    if self.db['beijing'].insert(result):
        print('save to mongo',result)

3.手工调试执行结果：
[I 180428 10:55:10 tornado_fetcher:419] [200] tripadvistor_bj:2370a7545cee745073ca951c016aa06f https://www.tripadvisor.cn/At... 0.83s
[I 180428 10:55:12 tornado_fetcher:419] [200] tripadvistor_bj:a89af7ac6933d3284facd55ec0210234 https://www.tripadvisor.cn/At... 0.72s
查看数据库，增加了一条数据
自动运行结果：
[I 180428 10:56:03 scheduler:906] task done tripadvistor_bj:on_start data:,on_start
[I 180428 10:56:04 scheduler:858] restart task tripadvistor_bj:on_finished data:,on_finished
[I 180428 10:56:04 scheduler:965] select tripadvistor_bj:on_finished data:,on_finished
[I 180428 10:56:04 tornado_fetcher:188] [200] tripadvistor_bj:on_finished data:,on_finished 0s
[I 180428 10:56:04 processor:202] process tripadvistor_bj:on_finished data:,on_finished -> [200] len:11 -> result:None fol:0 msg:0 err:None
[I 180428 10:56:05 scheduler:906] task done tripadvistor_bj:on_finished data:,on_finished
查看数据库，没有增加数据。
请大神指点一下如何查找问题，这其中tornado_fetcher:后面的代码表示什么意思呢？是否和这个有关？

Answer 1

pyspider 会自动去重，抓过的连接不会重新抓取，引用大神的：https://segmentfault.com/q/10...
pyspider说明手册
http://docs.pyspider.org/en/l...

我改成这样就好了：

class Handler(BaseHandler):

crawl_config = {
    'itag': 'v223'
}

Answer 2

请教下 这边的itag是怎么工作的，v223是代码中哪里的信息？

Answer 3

我也遇到类似的问题，手动一个个调试的时候，能够获取数据，等到 runing 的时候，就报错
[I 180819 21:07:33 tornado_fetcher:188] [200] meizitu_com:on_finished data:,on_finished 0s
[I 180819 21:07:33 processor:202] process meizitu_com:on_finished data:,on_finished -> [200] len:11 -> result:None fol:0 msg:0 err:None
[I 180819 21:07:33 scheduler:906] task done meizitu_com:on_finished data:,on_finished