问答博客资讯标签用户活动
logo极客观点logo项目管理logoHarmonyOS
javascript
前端
python
node.js
react
vue.js
php
laravel
go
人工智能
mysql
linux
ios
java
android
css
typescript
spring
程序员
logoONES 研发管理logo思否企业问答logo安谋科技 XPU

如何请求这个数据？

头像
luofeiyu
    1.6k29343706

    目标网页 https://www.bse.cn/nq/listedcompany.html

    第一页数据的post请求：

    import requests
url = "https://www.bse.cn/nqxxController/nqxxCnzq.do?callback=jQuery331_1678283984630"
payload = {
    "page": "0",
    "typejb": "T",
    "xxfcbj[]": "2",
    "xxzqdm": "",
    "sortfield": "xxzqdm",
    "sorttype": "asc",
}
r = requests.post(url, data=payload)
data_text = r.text

    上面的代码为何得不到数据？

    data_text
'<html>\r\n<head><title>403 Forbidden</title></head>\r\n<body>\r\n<center><h1>403 Forbidden</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n'

    使用curl,由于输出的是二进制数据，因此保存到文件

    curl 'https://www.bse.cn/nqxxController/nqxxCnzq.do?callback=jQuery331_1678285734365' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0' -H 'Accept: text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01' -H 'Accept-Language: en-GB,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'X-Requested-With: XMLHttpRequest' -H 'Origin: https://www.bse.cn' -H 'Connection: keep-alive' -H 'Referer: https://www.bse.cn/nq/listedcompany.html' -H 'Cookie: Hm_lvt_ef6193a308904a92936b38108b93bd7f=1678283985; Hm_lpvt_ef6193a308904a92936b38108b93bd7f=1678285736' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-origin' --data-raw 'page=1&typejb=T&xxfcbj%5B%5D=2&xxzqdm=&sortfield=xxzqdm&sorttype=asc'  -o  /tmp/test.data

    可以获得数据，但如何解析这些数据呢？

    python3.x爬虫
    阅读 1.5k
    1 个回答
    得票最新
    头像
    汝何不上九霄
      4.8k61947
      import requests

url = "https://www.bse.cn/nqxxController/nqxxCnzq.do?callback=jQuery331_1678285734365"

payload = "page=1&typejb=T&xxfcbj%5B%5D=2&xxzqdm=&sortfield=xxzqdm&sorttype=asc"
headers = {
  'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/110.0',
  'Accept': 'text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01',
  'Accept-Language': 'en-GB,en;q=0.5',
  'Accept-Encoding': 'gzip, deflate, br',
  'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
  'X-Requested-With': 'XMLHttpRequest',
  'Origin': 'https://www.bse.cn',
  'Connection': 'keep-alive',
  'Referer': 'https://www.bse.cn/nq/listedcompany.html',
  'Cookie': 'Hm_lvt_ef6193a308904a92936b38108b93bd7f=1678283985; Hm_lpvt_ef6193a308904a92936b38108b93bd7f=1678285736',
  'Sec-Fetch-Dest': 'empty',
  'Sec-Fetch-Mode': 'cors',
  'Sec-Fetch-Site': 'same-origin'
}

response = requests.request("POST", url, headers=headers, data=payload)

print(response.text)
      撰写回答
      你尚未登录，登录后可以
      • 和开发者交流问题的细节
      • 关注并接收问题和回答的更新提醒
      • 参与内容的编辑和改进，让解决方法与时俱进
      推荐问题