python 爬虫反爬后 换ip 也是空 ,求人分析网络包请求过程 原因

本地电脑 返回正常数据
curl -v 'https://www.qichacha.com/gong...' -H 'cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529479347; hasShow=1; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529201010,1529202769,1529381570,1529481047; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529481036815%2C%22updated%22%3A%201529481052862%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529481053' -H 'origin: https://www.qichacha.com' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' -H 'accept: /' -H 'referer: https://www.qichacha.com/' -H 'authority: www.qichacha.com' -H 'x-requested-with: XMLHttpRequest' --data 'key=baidu&type=0' --compressed

  • Trying 42.81.4.215...
  • TCP_NODELAY set
  • Connected to www.qichacha.com (42.81.4.215) port 443 (#0)
  • TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate: *.qichacha.com
  • Server certificate: GeoTrust SSL CA - G3
  • Server certificate: GeoTrust Global CA
POST /gongsi_getList HTTP/1.1
Host: www.qichacha.com
cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529479347; hasShow=1; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529201010,1529202769,1529381570,1529481047; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529481036815%2C%22updated%22%3A%201529481052862%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529481053
origin: https://www.qichacha.com
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
content-type: application/x-www-form-urlencoded; charset=UTF-8
accept: /
referer: https://www.qichacha.com/
authority: www.qichacha.com
x-requested-with: XMLHttpRequest
Content-Length: 16
  • upload completely sent off: 16 out of 16 bytes

< HTTP/1.1 200 OK
< Server: Tengine
< Content-Type: text/html; charset=UTF-8
< Content-Length: 462
< Connection: keep-alive
< Date: Wed, 20 Jun 2018 07:53:46 GMT
< Vary: Accept-Encoding
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Content-Encoding: gzip
< Via: cache19.l2em21-1[138,200-0,M], cache19.l2em21-1[138,0], cache8.cn247[164,200-0,M], cache5.cn247[168,0]
< X-Cache: MISS TCP_MISS dirn:-2:-2 mlen:-1
< X-Swift-SaveTime: Wed, 20 Jun 2018 07:53:46 GMT
< X-Swift-CacheTime: 0
< Timing-Allow-Origin: *
< EagleId: 2a51048515294812259037675e
<

  • Connection #0 to host www.qichacha.com left intact

被反爬后的电脑 返回数据是空:

shenjianlin@cron04:~$ curl 'https://www.qichacha.com/gong...' -H 'cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529479347; hasShow=1; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529201010,1529202769,1529381570,1529481047; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529481036815%2C%22updated%22%3A%201529481052862%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529481053' -H 'origin: https://www.qichacha.com' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' -H 'accept: /' -H 'referer: https://www.qichacha.com/' -H 'authority: www.qichacha.com' -H 'x-requested-with: XMLHttpRequest' --data 'key=baidu&type=0' --compressed
shenjianlin@cron04:~$ curl -v 'https://www.qichacha.com/gong...' -H 'cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529479347; hasShow=1; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529201010,1529202769,1529381570,1529481047; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529481036815%2C%22updated%22%3A%201529481052862%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529481053' -H 'origin: https://www.qichacha.com' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36' -H 'content-type: application/x-www-form-urlencoded; charset=UTF-8' -H 'accept: /' -H 'referer: https://www.qichacha.com/' -H 'authority: www.qichacha.com' -H 'x-requested-with: XMLHttpRequest' --data 'key=baidu&type=0' --compressed

  • About to connect() to www.qichacha.com port 443 (#0)
  • Trying 42.81.4.217...
  • Connected to www.qichacha.com (42.81.4.217) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
  • Server certificate:
  • subject: CN=*.qichacha.com,OU=IT,O=苏州企查查网络科技有限公司,L=苏州市,ST=江苏省,C=CN
  • start date: Jun 16 00:00:00 2017 GMT
  • expire date: Jun 15 23:59:59 2020 GMT
  • common name: *.qichacha.com
  • issuer: CN=GeoTrust SSL CA - G3,O=GeoTrust Inc.,C=US
POST /gongsi_getList HTTP/1.1
Host: www.qichacha.com
cookie: acw_tc=AQAAADKhg2r1fgQAzB38csKGgfa3ll5A; PHPSESSID=mr3rtla2pree2kma06in109lp7; UM_distinctid=16409b0ebb924e-01a16023a76872-19336953-13c680-16409b0ebba682; zg_did=%7B%22did%22%3A%20%2216409b0ebcf400-0a47e9ae97aea3-19336953-13c680-16409b0ebd0661%22%7D; _uab_collina=152917094889257593834626; _umdata=535523100CBE37C3B9E8426803FAE682F695DD5C372880100D01308BEF2CB953FEF5024D24D0BA85CD43AD3E795C914C6E418FBD7FCF11CFC02159EA6BDBD805; CNZZDATA1254842228=222723142-1529170913-https%253A%252F%252Fwww.qichacha.com%252F%7C1529479347; hasShow=1; Hm_lvt_3456bee468c83cc63fb5147f119f1075=1529201010,1529202769,1529381570,1529481047; zg_de1d1a35bfa24ce29bbf2c7eb17e6c4f=%7B%22sid%22%3A%201529481036815%2C%22updated%22%3A%201529481052862%2C%22info%22%3A%201529170947031%2C%22superProperty%22%3A%20%22%7B%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22cuid%22%3A%20%22b77823811d3a8fd207eef49092fcf4d6%22%7D; Hm_lpvt_3456bee468c83cc63fb5147f119f1075=1529481053
origin: https://www.qichacha.com
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
content-type: application/x-www-form-urlencoded; charset=UTF-8
accept: /
referer: https://www.qichacha.com/
authority: www.qichacha.com
x-requested-with: XMLHttpRequest
Content-Length: 16
  • upload completely sent off: 16 out of 16 bytes

< HTTP/1.1 200 OK
< Server: Tengine
< Content-Type: text/html; charset=UTF-8
< Content-Length: 20
< Connection: keep-alive
< Date: Wed, 20 Jun 2018 07:53:02 GMT
< Vary: Accept-Encoding
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
< Content-Encoding: gzip
< Via: cache19.l2nu20-3[88,200-0,M], cache15.l2nu20-3[89,0], cache8.cn247[114,200-0,M], cache6.cn247[115,0]
< X-Cache: MISS TCP_MISS dirn:-2:-2 mlen:-1
< X-Swift-SaveTime: Wed, 20 Jun 2018 07:53:02 GMT
< X-Swift-CacheTime: 0
< Timing-Allow-Origin: *
< EagleId: 2a51048615294811825831009e
<

  • Connection #0 to host www.qichacha.com left intact
阅读 3.5k
撰写回答
你尚未登录,登录后可以
  • 和开发者交流问题的细节
  • 关注并接收问题和回答的更新提醒
  • 参与内容的编辑和改进,让解决方法与时俱进
推荐问题
宣传栏