urllib和request的爬虫的两个小问题

Question

在网上查到使用urllib下载文件显示百分比的方法如下

def report(count, blockSize, totalSize):
  percent = int(count*blockSize*100/totalSize)
  sys.stdout.write("\r%d%%" % percent + ' complete')
  sys.stdout.flush()
sys.stdout.write('\rFetching ' + name + '...\n')
urllib.urlretrieve(getFile, saveFile, reporthook=report)
sys.stdout.write("\rDownload complete, saved as %s" % (fileName) + '\n\n')
sys.stdout.flush()

而用request下载文件只查到

s=requests.session()
down = s.get(download_url)
local =os.path.join(path,filename)
with open(local, "wb") as f:
    f.write(down.content)
    

问题有两个：
1.如何用request的下载功能实现下载文件显示百分比？
2.urllib能否实现request的session功能，目前使用request下载正常，使用urllib哪怕正常登录了获取到文件地址了下载也是失败

Answer 1

request里好像没有progress callback，urllib2／3里我也没看到。

用request的话你可以通过stream把文件下载，同时显示百分比：

import requests
import sys
url = "..."
filename = "..."

def download_progress(session, url, filename, bar_width = 50):
    with open(filename, "wb") as f:
        print("Downloading %s from %s" % (filename, url))
        response = s.get(url, stream=True)
        content_length = response.headers.get('Content-Length')
        if content_length:
            content_length = int(content_length)
            print("Loading %d bytes" % content_length)
        else:
            content_length = -1
        loaded = 0
        for data in response.iter_content(chunk_size=4096):
            loaded += len(data)
            f.write(data)
            msg = "\r"
            if content_length > 0:
                bar = int(bar_width * loaded / content_length)
                msg += "[%s%s]" % ('=' * bar, ' ' * (bar_width - bar))
            msg += " loaded %8d bytes" % loaded
            print(msg, file=sys.stdout, end="")

s=requests.session()
download_progress(s, url, filename)

urllib2有opener功能，根request的session差不多。