POST查询访问网站上的表格数据（Python）

Question

我想用this website来提取数据，但是这个网站没有API来方便数据提取。因此，网站上有一个包含数据的“表格”，但该表格分布在多个页面上，下载选项不支持“批量”下载。所以我想找出一种通过Python下载的方法。
我发现我可以通过发布请求来做到这一点。我所做的是我开始在第1页，然后点击“下一页”按钮，并观看了网络选项卡，这给了我：

Request URL:https://www.patricbrc.org/api/genome/ 
Request Method:POST 
Status Code:200 OK 
Remote Address:128.173.97.11:443 
Referrer Policy:no-referrer-when-downgrade 

Response Headers 
Access-Control-Allow-Origin:https://www.patricbrc.org 
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag 
Content-Encoding:gzip 
Content-Range:items 200-399/45999 
Content-Type:application/json; charset=utf-8 
Date:Sat, 26 Aug 2017 09:36:40 GMT 
ETag:W/"8ac0b-cFdEBUwfdiyTQm/gpJHYzQ" 
Server:nginx/1.9.1 
Transfer-Encoding:chunked 
Vary:Origin, Accept 
Vary:Accept-Encoding 
X-Powered-By:Express 

Request Headers 
Accept:application/javascript, application/json, application/json 
Accept-Encoding:gzip, deflate, br 
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4 
Connection:keep-alive 
Content-Length:82 
Content-Type:application/rqlquery+x-www-form-urlencoded 
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311; _gat=1 
Host:www.patricbrc.org 
Origin:https://www.patricbrc.org 
Range:items=200-399 
Referer:https://www.patricbrc.org/view/Taxonomy/2 
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36 
X-Range:items=200-399 

Request Payload 
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score) 
Name 
genome/ 

要查看页面切换我再次做到这之后有什么变化（所以再次点击“下一页”按钮）。 > 2和第2页 - -

Request URL:https://www.patricbrc.org/api/genome/ 
Request Method:POST 
Status Code:200 OK 
Remote Address:128.173.97.11:443 
Referrer Policy:no-referrer-when-downgrade 

Response Headers 
Access-Control-Allow-Origin:https://www.patricbrc.org 
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag 
Content-Encoding:gzip 
Content-Range:items 400-599/45999 
Content-Type:application/json; charset=utf-8 
Date:Sat, 26 Aug 2017 09:44:24 GMT 
ETag:W/"5dc73-ODopjbbyl5M2vUJvUGN0Gw" 
Server:nginx/1.9.1 
Transfer-Encoding:chunked 
Vary:Origin, Accept 
Vary:Accept-Encoding 
X-Powered-By:Express 

Request Headers 
Accept:application/javascript, application/json, application/json 
Accept-Encoding:gzip, deflate, br 
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4 
Connection:keep-alive 
Content-Length:82 
Content-Type:application/rqlquery+x-www-form-urlencoded 
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311 
Host:www.patricbrc.org 
Origin:https://www.patricbrc.org 
Range:items=400-599 
Referer:https://www.patricbrc.org/view/Taxonomy/2 
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36 
X-Range:items=400-599 

Request Payload 
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score) 

从第1页去当post请求过程中改变（在我看来），所以重要的事情> 3分别为：

• 响应头内容范围：项200-399/45999 - >内容-范围：项400-599/45999
• 请求头范围：项= 200-399 - >范围：项= 400-599
• 请求头 X-范围：项= 200-399 - > X-范围：项= 400-599

这是网站这样说：

• 第1页：1 - 200的45999个结果（但职位0-199，上面没有显示）
• 页2：201 - 400 45999个结果（但职位200-399，见上文）
• 页3：401 - 600 45999个结果（但职位400-599，见上文）

我有零关于阅读这些POST请求的知识。但我想应该说一个方法：

requests.post(some url) and extract the total page numbers (or number of genomes and devide these by the max number per page, which is 200 see post requests). 

然后是这样的：

for page_numb in range(page_numbers): 
    r = requests.post(some_url...... + page_numb) 
    #extract the information from the table 

最后的问题:) 我无法弄清楚如何建立一个POST请像

p1 = requests.post(https://www.patricbrc.org/api/genome/... 0-199) 
p2 = requests.post(https://www.patricbrc.org/api/genome/... 200-399) 
etc... 

（我希望效应初探将包含一些分隔符分隔表数据）

---

的代码我想：

import requests 
import json 

url = 'https://www.patricbrc.org/api/genome/' 
payload = {"eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)"} 
headers = { 
    "X-Range": "items=0-199", 
    "Range": "items=0-199", 
    "content-type": "application/json"} 


r = requests.post(url, data=json.dumps(payload), headers=headers) 
print(r) 

此代码不能正常工作，但我想应该是这样的。
我知道我应该将有效载荷作为JSON发布，但我不知道如何将eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)转换为JSON，并且它是关键字：值对。

Answer 1

这里是一个源代码（其丑陋的代码，但它只是举例来说，我建议你用request.Session做）：

import sys 
import json 
import requests 

URL = 'https://www.patricbrc.org/view/Taxonomy/2' 
API = 'https://www.patricbrc.org/api/genome/' 

# we need to get a Cookie 
response = requests.get(URL) 
if response.status_code != 200: 
    sys.exit('Cookie request failed') 
cookie = response.headers['Set-Cookie'].split(';')[0] 
etag = response.headers['ETag'] 

# now with cookie request a data 
headers = { 
    'Host'   : 'www.patricbrc.org', 
    'Accept'   : 'application/javascript, application/json, application/json', 
    'Accept-Language' : 'en-US,en;q=0.5', 
    'Referer'   : 'https://www.patricbrc.org/view/Taxonomy/2', 
    'X-Range'   : 'items=200-399', # your pages 
    'Range'   : 'items=200-399', # your pages 
    'Content-Type' : 'application/rqlquery+x-www-form-urlencoded', 
    'Cookie'   : '{};'.format(cookie), 
    'ETag'   : etag 
} 

data = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)' 
response = requests.post(API, headers=headers, data=data) 
if response.status_code != 200: 
    sys.exit('API request failed') 
data = json.loads(response.text) 

# test data 
row200 = data[0] 
print 'FAMILY  :', row200['family'] 
print 'GC_CONTENT :', row200['gc_content'] 
print 'GENUS  :', row200['genus'] 
print 'ORDER  :', row200['order'] 

输出test data块：

FAMILY  : Eubacteriaceae 
GC_CONTENT : 38.78 
GENUS  : Eubacterium 
ORDER  : Clostridiales 

编辑 你不需要一个饼干，这工作得很好：

import json 
import requests 

API = 'https://www.patricbrc.org/api/genome/' 

# now with cookie request a data 
headers = { 
    'Host'   : 'www.patricbrc.org', 
    'Accept'   : 'application/javascript, application/json, application/json', 
    'Accept-Language' : 'en-US,en;q=0.5', 
    'Referer'   : 'https://www.patricbrc.org/view/Taxonomy/2', 
    'X-Range'   : 'items=200-399', # your pages 
    'Range'   : 'items=200-399', # your pages 
    'Content-Type' : 'application/rqlquery+x-www-form-urlencoded', 
} 

query = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)' 
response = requests.post(API, headers=headers, data=query) 
data = json.loads(response.text)