为什么用gb2312解码内涵段子吧得到的中文是乱码？

Question

用以下代码获取了网页内容之后，解码再编码print出来的中文部分是乱码。可能是什么原因

import chardet # 一个检查编码的库
url = "http://www.neihan8.com/article/list_5_1.html"
user_agent = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0'

headers = {'User-Agent': user_agent}
req = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(req)
html = response.read()
checkCode = chardet.detect(html) # 检测网页的编码格式
print('checkCode', checkCode)  
#上面那句输出的结果checkCode {'encoding': 'GB2312', 'confidence': 0.99, 'language': 'Chinese'}
gbk_html = html.decode(checkCode['encoding']).encode('utf-8')
print(gbk_html)

这上面的代码运行结果里第一个title标签如下所示

<title>\xd7\xee\xd3\xd0\xc4\xda\xba\xad\xb5\xc4\xd0\xa6\xbb\xb0_\xc3\xbf
\xc8\xd5\xd2\xbb\xd0\xa6\xca\xd5\xbc\xaf\xd7\xee\xd0\xc2\xb5\xc4\xc4\xda
\xba\xad\xb8\xe3\xd0\xa6\xb6\xce\xd7\xd3\xd0\xa6\xbb\xb0_\xbb\xe7\xb6\xce\
xd7\xd3\xd0\xa6\xbb\xb0_\xc4\xda\xba\xad\xb0\xc9</title>\r\n

回答里的代码运行的结果第一个title标签是这样的

import requests
url = "http://www.neihan8.com/article/list_5_1.html"
print requests.get(url).content.decode('gb2312').encode('utf-8')

<title>\xe6\x9c\x80\xe6\x9c\x89\xe5\x86\x85\xe6\xb6\xb5\xe7\x9a\x84\xe7\
xac\x91\xe8\xaf\x9d_\xe6\xaf\x8f\xe6\x97\xa5\xe4\xb8\x80\xe7\xac\x91\xe6
\x94\xb6\xe9\x9b\x86\xe6\x9c\x80\xe6\x96\xb0\xe7\x9a\x84\xe5\x86\x85\xe6
\xb6\xb5\xe6\x90\x9e\xe7\xac\x91\xe6\xae\xb5\xe5\xad\x90\xe7\xac\x91\xe8
\xaf\x9d_\xe8\x8d\xa4\xe6\xae\xb5\xe5\xad\x90\xe7\xac\x91\xe8\xaf\x9d_\x
e5\x86\x85\xe6\xb6\xb5\xe5\x90\xa7</title>\r\n

是不是我这边的ide的输入输出配置有问题？

Answer 1

首先推荐你使用requests，简单好用。其次，你要的这个功能，这样子就能解决：

import requests
url = "http://www.neihan8.com/article/list_5_1.html"
print requests.get(url).content.decode('gb2312').encode('utf-8')

Answer 2

你在爬去一个网页的内容首先要看网页的编码方式的，一般在网页的head中。然后在爬取的时候在选择相应的编码方式

Answer 3

import requests

url = "http://www.neihan8.com/article/list_5_1.html"
user_agent = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0'
headers = {'User-Agent': user_agent}
req = requests.get(url, headers=headers)

print(req.encoding)
req.encoding = 'gbk'
print(req.encoding)
print(req.text)

Answer 4

多谢大家的回答。
不过问题确实有好几个。我把后来改正过来代码贴一下。

# python3
url = "http://www.neihan8.com/article/list_5_" + str(page) + ".html"
user_agent = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0'

headers = {'User-Agent': user_agent}
req = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(req)
html = response.read()
print('python3 response.read()', type(html))
checkCode = chardet.detect(html)
print('checkCode', checkCode)
_html = html.decode(checkCode['encoding'])
print('python3 response.read().decode(gb2312)',type(_html))

# 输出：
python3 response.read() <class 'bytes'>
checkCode {'encoding': 'GB2312', 'confidence': 0.99, 'language': 'Chinese'}
python3 response.read().decode(gb2312) <class 'str'>
python3 type(requests.get(url).content) <class 'bytes'>
python3 type(requests.get(url).content.decode('gb2312')) <class 'str'>

下面是python2的代码：

url = "http://www.neihan8.com/article/list_5_" + str(page) + ".html"
user_agent = 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0'

headers = {'User-Agent': user_agent}
req = urllib2.Request(url, headers=headers)
response = urllib2.urlopen(req)
html = response.read()
print('python2 response.read()', type(html))
checkCode = chardet.detect(html)
print('checkCode', checkCode)
#gbk_html = html.encode(checkCode['encoding']).decode('utf-8')
_html = html.decode('gb2312')#.decode('gb2312')
print('python2 response.read().decode(\'gb2312\')', type(_html))

# 输出：
('python2 response.read()', <type 'str'>)
('checkCode', {'confidence': 0.99, 'language': 'Chinese', 'encoding': 'GB2312'})
("python2 response.read().decode('gb2312')", <type 'unicode'>)
('python2 type(requests.get(url).content)', <type 'str'>)
("python2 type(requests.get(url).content.decode('gb2312'))", <type 'unicode'>)

虽然2和3返回的源编码不是同一类型，但只要decode成unicode格式就能print出来了，总结起来还是对输入输出的编码理解有问题。