求解Beautifulsoup不能爬取网页url链接地址。

Question

我想抓取这个网页（http://www.zhujiage.com.cn/article/List_2.html）和这个网页（http://www.qinbing.cn/jidanjiage/）的url链接地址。

from urllib2 import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://www.zhujiage.com.cn/article/List_2.html")
bsobj = BeautifulSoup(html,'lxml')
for link in bsobj.find_all(name="a"):
       if 'href' in  link.attrs:
           print link.attrs['href']

上面这段代码只改变网址，前一个网址不能抓取网页URL，后面的网址可以抓取。

Answer 1

这个是网站编码的问题, 第一网站编码是gbk, 第二个是utf-8, BeaufifulSoup默认是将网页以uft-8编码, 只需要加个参数from_encoding ="gbk" 就可以解决了

from urllib2 import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://www.zhujiage.com.cn/article/List_2.html")
bsobj = BeautifulSoup(html,'lxml',from_encoding="gbk")
for link in bsobj.find_all(name="a"):
       if 'href' in  link.attrs:
           print link.attrs['href']

也可以

from urllib2 import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://www.zhujiage.com.cn/article/List_2.html").read().decode('gbk')
bsobj = BeautifulSoup(html,'lxml')
for link in bsobj.find_all(name="a"):
       if 'href' in  link.attrs:
           print link.attrs['href']