使用scrapy的时候，用了start_requests()循环生成要爬取的网址，还需要写start_urls吗？

Question

使用scrapy的时候，用了start_requests()循环生成要爬取的网址，还需要写start_urls吗？

比如：

class demoSpider(RedisSpider):
    name = "demospider"

    redis_key = 'demospider:start_urls'
    start_urls = ['http://www.example.com']

    def start_requests(self):
        pages=[]
        for i in range(1,10):
            url='http://www.example.com/?page=%s'%i
            page=scrapy.Request(url)
            pages.append(page)
        return pages

Answer 1

不需要 而且写了start_urls也没有用

http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.start_requests

The default implementation uses make_requests_from_url() to generate Requests for each url in start_urls

你重写start_requests也就不会从start_urls generate Requests了

看看源码

scrapy/spiders/__init__.py:68行

def start_requests(self):
    for url in self.start_urls:
        yield self.make_requests_from_url(url)