import base64
class ProxyMiddleware(object):
def process_request(self, request, spider):
request.meta['proxy'] = "http://IP:posrt"
#proxy_user_pass = "user:psw"
#encoded_user_pass = base64.encodestring(proxy_user_pass)
#request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
如何知道此次爬取,是用代理的ip爬取?
可以爬一个检测ip地址的网站来验证一下,比如这个http://ip.filefab.com/