urllib里的Request方法, 换个URL就不能正常工作

Question

以下代码运行会报错.

from urllib import request

req = request.Request('https://www.douban.com')
with request.urlopen(req) as f:
    print('Status:', f.status, f.reason)
    for k, v in f.getheaders():
        print('%s: %s' % (k, v))
    print('Data:', f.read().decode('utf-8'))

但是只要将url替换为https://api.douban.com/v2/boo... 就能正常工作.
这是什么原因呢?
第一天学习爬虫, 百思不得其解.

Answer 1

>>> from urllib import request
>>> rsp=request.urlopen('https://www.douban.com')
>>> rsp.headers.items()
[('Date', 'Fri, 23 Sep 2016 04:04:07 GMT'), ('Content-Type', 'text/html; charset=utf-8'), ('Content-Length', '95658'), ('Connection', 'close'), ('Vary', 'Accept-Encoding'), ('X-Xss-Protection', '1; mode=block'), ('X-Douban-Mobileapp', '0'), ('Expires', 'Sun, 1 Jan 2006 01:00:00 GMT'), ('Pragma', 'no-cache'), ('Cache-Control', 'must-revalidate, no-cache, private'), ('Set-Cookie', 'll="118303"; path=/; domain=.douban.com; expires=Sat, 23-Sep-2017 04:04:07 GMT'), ('Set-Cookie', 'bid=bBpUcfoRNzY; Expires=Sat, 23-Sep-17 04:04:07 GMT; Domain=.douban.com; Path=/'), ('X-DOUBAN-NEWBID', 'bBpUcfoRNzY'), ('X-DAE-Node', 'sindar9c'), ('X-DAE-App', 'sns'), ('Server', 'dae'), ('Strict-Transport-Security', 'max-age=15552000;')]
>>> rspapi=request.urlopen('https://api.douban.com/v2/book/2129650')
>>> rspapi.headers.items()
[('Date', 'Fri, 23 Sep 2016 04:08:07 GMT'), ('Content-Type', 'application/json; charset=utf-8'), ('Content-Length', '2055'), ('Connection', 'close'), ('Vary', 'Accept-Encoding'), ('Expires', 'Sun, 1 Jan 2006 01:00:00 GMT'), ('Pragma', 'no-cache'), ('Cache-Control', 'must-revalidate, no-cache, private'), ('Set-Cookie', 'bid=iRABGolqvts; Expires=Sat, 23-Sep-17 04:08:07 GMT; Domain=.douban.com; Path=/'), ('X-DOUBAN-NEWBID', 'iRABGolqvts'), ('X-DAE-Node', 'dis16'), ('X-DAE-App', 'book'), ('Server', 'dae')]
>>> 

我试了2种方法，都没问题，你最好把错误信息贴出来。

Answer 2

设置User-Agent试试看

request.Request(url, headers={
            "User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36"})