scrapy url去重

Question

请问scrapy是url自动去重的吗？比如下面这段代码，为什么运行时start_urls里面的重复url会重复爬取了？

class TestSpider(scrapy.Spider):
    name = "test"
    allowed_domains = ["baidu.com"]
    start_urls = ['http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9',
                  'http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9',
                  'http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9',]

    def parse(self, response):
        for sel in response.xpath('//div[@class="grid-list grid-list-spot"]/ul/li'):
            item = TestspiderItem()
            item['title'] = sel.xpath('div[@class="list"]/a/text()')[0].extract()
            item['link'] = sel.xpath('div[@class="list"]/a/@href')[0].extract()
            yield item

Answer 1

建一个Url管理器，就不会重复抓取了

Answer 2

知道了，改成这样就可以了。

def start_requests(self):

yield scrapy.Request('http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9', self.parse)
yield scrapy.Request('http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9', self.parse)
yield scrapy.Request('http://baike.baidu.com/fenlei/%E5%A8%B1%E4%B9%90%E4%BA%BA%E7%89%A9', self.parse)

Answer 3

1、默认start_urls里的链接不会去重。
如果要实现需要重写start_requests。

def start_requests(self):
    for url in self.start_urls:
        yield Request(url, **dont_filter**=True)

2、是否去重由dont_filter控制
scrapy.Request（）添加参数dont_filter=False才会去重。

Answer 4

一般使用redisbloomfilter去重吧