Scrapy只有一句话

Question

1. items.py

     class DouyuspiderItem(scrapy.Item):
            name = scrapy.Field()# 存储照片的名字
            imagesUrls = scrapy.Field()# 照片的url路径
            imagesPath = scrapy.Field()# 照片保存在本地的路径

1. spiders/douyu.py

    import scrapy
    import json
    from douyuSpider.items import DouyuspiderItem
    
    class DouyuSpider(scrapy.Spider):
        name = "douyu"
        allowd_domains = ["http://capi.douyucdn.cn"]
    
        offset = 0
        url = "http://capi.douyucdn.cn/api/v1/getVerticalRoom?limit=20&offset="
        start_urls = [url + str(offset)]
    
      def parse(self, response):
          # 返回从json里获取 data段数据集合
          data = json.loads(response.text)["data"]
    
          for each in data:
              item = DouyuspiderItem()
              item["name"] = each["nickname"]
              item["imagesUrls"] = each["vertical_src"]
    
              yield item
    
          self.offset += 20
          yield scrapy.Request(self.url + str(self.offset), callback = self.parse)

1. 设置setting.py

    ITEM_PIPELINES = {'douyuSpider.pipelines.ImagesPipeline': 1}
    
    # Images 的存放位置，之后会在pipelines.py里调用
    IMAGES_STORE = "/Users/Power/lesson_python/douyuSpider/Images"
    
    # user-agent
    USER_AGENT = 'DYZB/2.290 (iPhone; iOS 9.3.4; Scale/2.00)'

1. pipelines.py

import scrapy
import os
from scrapy.pipelines.images import ImagesPipeline
from scrapy.utils.project import get_project_settings

class ImagesPipeline(ImagesPipeline):
    IMAGES_STORE = get_project_settings().get("IMAGES_STORE")

    def get_media_requests(self, item, info):
        image_url = item["imagesUrls"]
        yield scrapy.Request(image_url)

    def item_completed(self, results, item, info):
        # 固定写法，获取图片路径，同时判断这个路径是否正确，如果正确，就放到 image_path里，ImagesPipeline源码剖析可见
        image_path = [x["path"] for ok, x in results if ok]

        os.rename(self.IMAGES_STORE + "/" + image_path[0], self.IMAGES_STORE + "/" + item["name"] + ".jpg")
        item["imagesPath"] = self.IMAGES_STORE + "/" + item["name"]

        return item

#get_media_requests的作用就是为每一个图片链接生成一个Request对象，这个方法的输出将作为item_completed的输入中的results，results是一个元组，每个元组包括(success, imageinfoorfailure)。如果success=true，imageinfoor_failure是一个字典，包括url/path/checksum三个key。

在项目根目录下新建main.py文件,用于调试

from scrapy import cmdline
cmdline.execute('scrapy crawl douyu'.split())

执行程序

py2 main.py

---

出现的错误是这样~~

2018-01-16 14:51:09 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: douyu)
2018-01-16 14:51:09 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.3.1, w3lib 1.18.0, Twisted 17.9.0, Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), cryptography 2.1.4, Platform Windows-10-10.0.14393
2018-01-16 14:51:09 [scrapy.crawler] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'douyu.spiders', 'SPIDER_MODULES': ['douyu.spiders'], 'ROBOTSTXT_OBEY': True, 'USER_AGENT': 'DYZB/2.290 (iPhone; iOS 9.3.4; Scale/2.00)', 'BOT_NAME': 'douyu'}
2018-01-16 14:51:09 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2018-01-16 14:51:10 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-01-16 14:51:10 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-01-16 14:51:10 [scrapy.middleware] INFO: Enabled item pipelines:
['douyu.pipelines.ImagesPipeline']
2018-01-16 14:51:10 [scrapy.core.engine] INFO: Spider opened
2018-01-16 14:51:10 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-01-16 14:51:10 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-01-16 14:51:10 [scrapy.core.engine] INFO: Closing spider (finished)
2018-01-16 14:51:10 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 1, 16, 6, 51, 10, 521000),
 'log_count/DEBUG': 1,
 'log_count/INFO': 7,
 'start_time': datetime.datetime(2018, 1, 16, 6, 51, 10, 516000)}

Answer 1

额，allowd_domains单词写错了，而且里面的元素应该是域名而不是url，即不带http