为何不能使用urllib来构造post请求？

Question

有这样一个post请求
request header:

POST /csindex-home/exportExcel/security-industry-search-excel/CH HTTP/1.1
Host: www.csindex.com.cn
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/json;charset=utf-8
Content-Length: 78
Origin: https://www.csindex.com.cn
Connection: keep-alive
Referer: https://www.csindex.com.cn/
Cookie: aliyungf_tc=0b04a5cc679f3b1e6d9d27a2635a318c4b60b69f0d58a933d780f569a78d043c; ssxmod_itna=iqmxgQDQG=0=oxeq0Lx=S8GO8KDtDcAPcDto=7Qx40Hn0eiODUxn4iaDT=aNBpQ2iwPGbnGar=T+7Gi4Ho/gfgTe87ZeYhoD74i8uDW5DlDDR7NdDlc470DYPG0DiFQDX=Llkqp1IACt2DGCzPqDu24DxxiTDinoq04x0lp7Ih986=xiO6n5NjDrrCxY5ehr8eRbd12IDDcEHChDD===; zg_did=%7B%22did%22%3A%20%221868ba15f2a47b-009a307f25d0578-376a464a-1fa400-1868ba15f2b4c6%22%7D; _uab_collina=167737985610863887816821; ssxmod_itna2=iqmxgQDQG=0=oxeq0Lx=S8GO8KDtDcAPcDto=7DnI3yiqDshKDLe=t7NIQB=AxqidzAPmi+ennDn7UG7FAmGi=es9uIxhej9GB=foqUGA9LPcmQb4ftxPpmiPPvX7Dc8KciqeAM8Ro+YRE378GD8YD0ec34iiWqqGwq7FY4kbAxk+Oe8KDUe7SrSlCDr9fYoA2qh0AYi/+EYjRqeN8deq4h=kc3qiC9Fw8AXFAcCNnKw=fYN/3KIqS=kUdSNubKSDr57Rk2B+YHtBcM8000WytU6=b6en87dmbYV9ztohpGEYToB8CFNxQwOGmiGDx47jI44v7lheo0UaQpoqcDj7jGmlPFgQ8WiGAXt+=4KbPxq4ePc0f8meumLrZqdYj58qFOe+DRuPm2O0o3=xRd1zou6QBDnda0WnfiKvvWmdx49YtjPhleAKgmDOQiKe=1U6KdTILvpiv00UamU8QjRLCPA7BmnI0+79d2azoKcOQ2jvBjyM/foSy/Feu232naAaHchi27rWOyuoKy4ql9TPTT3MWdvfVzExhmQMOZ8qpC5cAeGmmmjmivYFMO8BD7j5ceqx6Ww1wp8kakqeKpRejviMePhF6=c0e+dlpq4aDG2tW4Qz89Wc=D9chE7avY2cu/=iB6orBFcxlmm7c8Ax4DKGRxWo74P8xnzGYDvf=aHaYbeqi7ycDMvuAtLr4pe5RmY4niqjxKy6pGS043YN7Pg37ed2D7i5Qs9GeeP7AXpDD08DG7oGDD=; acw_tc=76b20f8a16773929641405222e21e87483e560370854cd0bb2155ccc8ed2a6; zg_6df0ba28cbd846a799ab8f527e8cc62b=%7B%22sid%22%3A%201677392958817%2C%22updated%22%3A%201677393938767%2C%22info%22%3A%201677379854166%2C%22superProperty%22%3A%20%22%7B%5C%22%E5%BA%94%E7%94%A8%E5%90%8D%E7%A7%B0%5C%22%3A%20%5C%22%E4%B8%AD%E8%AF%81%E6%8C%87%E6%95%B0%E5%AE%98%E7%BD%91%5C%22%7D%22%2C%22platform%22%3A%20%22%7B%7D%22%2C%22utm%22%3A%20%22%7B%7D%22%2C%22referrerDomain%22%3A%20%22%22%2C%22landHref%22%3A%20%22https%3A%2F%2Fwww.csindex.com.cn%2F%23%2FdataService%2FindustryClassification%22%2C%22prePath%22%3A%20%22https%3A%2F%2Fwww.csindex.com.cn%2F%23%2FdataService%2FindustryClassification%22%2C%22duration%22%3A%209749.359999999404%2C%22zs%22%3A%200%2C%22sc%22%3A%200%2C%22firstScreen%22%3A%201677392958817%7D
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin

post data

{"searchInput":"","pageNum":3,"pageSize":10,"sortField":null,"sortOrder":null}

使用requests库构造的请求：

import requests
from urllib.parse import unquote
res = requests.post(
    "https://www.csindex.com.cn/csindex-home/exportExcel/security-industry-search-excel/CH",
    headers={"content-type": "application/json;charset=UTF-8"},
    json={
        "searchInput": "", # 输入的搜索内容
        "pageNum": 1, # 页码
        "pageSize": 10, # 每页条数
        "sortField": None, # 按哪个类别排序
        "sortOrder": None, # 排序方式
    },
)
print(res.status_code, res.url)
200 https://www.csindex.com.cn/csindex-home/exportExcel/security-industry-search-excel/CH

我使用urllib库

import urllib.request
base_url="https://www.csindex.com.cn/csindex-home/exportExcel/security-industry-search-excel/CH"
data = {
    'searchInput' : '',
    'pageNum' : 1,
    'pageSize' : 10,
    'sortField' : None,
    'sortOrder' : None
}
headers={"content-type": "application/json;charset=UTF-8"}
postdata=urllib.parse.urlencode(data).encode('utf-8')
req=urllib.request.Request(url=base_url,headers=headers,data=postdata,method='POST')
response=urllib.request.urlopen(req)
html=response.read()
print(html.decode('utf-8'))
{"code":"500","msg":"服务器异常,请联系管理员","data":null,"success":false}

换一种格式也不行

import urllib.request    
import urllib.parse    
    
url="https://www.csindex.com.cn/csindex-home/exportExcel/security-industry-search-excel/CH"   
params = {
    'searchInput' : '',
    'pageNum' : 1,
    'pageSize' : 10,
    'sortField' : None,
    'sortOrder' : NOne
}
    
query_string = urllib.parse.urlencode( params )    
data = query_string.encode( "ascii" )    
    
with urllib.request.urlopen( url, data ) as response:     
    response_text = response.read()     
    print( response_text ) 

为何服务器异常呢？

Answer 1

没错，就像楼上所说的那样，在第一个使用 urllib 的示例中，postdata 应这样写：

import json
postdata = bytes(json.dumps(data), "utf-8")

另外，这个网址返回的是二进制数据，所以在最后也不能这么写：html.decode('utf-8')

Answer 2

你发送成功的例子里，content-type 是 application/json，body 里放的是 json 格式的数据
所以这个接口接受的是 json 格式的数据

你失败的例子里，content-type 是 application/json，没有问题，body 是 urlencode 的数据，有问题

所以你需要把你的数据用 json 序列化放到 body 里发送，就可以了