无法使用xpath获取脚本标记的内容，使用xpath，lxml

Question

我正在编写一个python程序，并且作为它的一部分，我需要从Google ngram viewer中提取数据。例如，对于搜索：

https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3

我所需要的图形值从这个标签：

var data = [{"ngram": "The Godfather", "type": "NGRAM", "timeseries": [1.4183381225052472e-07, 1.4025288521679614e-07, 1.5749316872870622e-07, 1.618123600824869e-07, 1.7873649125834034e-07, 1.8325580697364785e-07, 1.838378673418057e-07, 1.7964884234191102e-07, 1.7921279850595185e-07, 1.8174738970953642e-07, 1.7919142944070439e-07, 1.7377866307859741e-07, 1.6968103417574249e-07, 1.6785447241675554e-07, 1.698323818085815e-07, 1.7567729011196726e-07, 1.7198440259237812e-07, 1.6793204338227952e-07, 1.7446684604952418e-07, 1.8290416343396438e-07, 1.8512590876136009e-07, 1.8604621183320497e-07, 1.9866517269357636e-07, 1.8994029866397405e-07, 2.0815326909736802e-07, 2.2254876138764042e-07, 2.2457939508058189e-07, 2.5743728875633156e-07, 2.7453169236326046e-07, 2.7866381507075335e-07, 3.0070588609630378e-07, 3.0204388273042629e-07, 2.9686912585345583e-07, 2.9399397760698776e-07, 2.7703356645740013e-07, 2.7225316614476467e-07, 2.6805425434872632e-07], "parent": ""}]; 
    if (data.length > 0) { 
    ngrams.drawD3Chart(data, 1972, 2008, 1.0, "main"); 
    } 

我使用LXML到HTML成树转换试图然后使用从复制的XPath该网站是这样的：

page = requests.get('https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3') 
tree = html.fromstring(page.content) 

nGramData = tree.xpath('//*[@id="container"]/script[11]/text()') 

但是，如果我再尝试打印nGramData我没有得到任何结果，我在做什么错误，或有更好的方法来做到这一点？

Answer 1

使用下面的工作代码：

>>> page = requests.get('https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3') 
>>> tree = html.fromstring(page.content) 
>>> nGramData = tree.xpath('//*[@id="container"]/script') 
>>> nGramData[6].text_content() 
'\n var data = [{"ngram": "The Godfather", "type": "NGRAM", "timeseries": [1.4183381225052472e-07, 1.4025288521679614e-07, 1.5749316872870622e-07, 1.618123600824869e-07, 1.7873649125834034e-07, 1.8325580697364785e-07, 1.838378673418057e-07, 1.7964884234191102e-07, 1.7921279850595185e-07, 1.8174738970953642e-07, 1.7919142944070439e-07, 1.7377866307859741e-07, 1.6968103417574249e-07, 1.6785447241675554e-07, 1.698323818085815e-07, 1.7567729011196726e-07, 1.7198440259237812e-07, 1.6793204338227952e-07, 1.7446684604952418e-07, 1.8290416343396438e-07, 1.8512590876136009e-07, 1.8604621183320497e-07, 1.9866517269357636e-07, 1.8994029866397405e-07, 2.0815326909736802e-07, 2.2254876138764042e-07, 2.2457939508058189e-07, 2.5743728875633156e-07, 2.7453169236326046e-07, 2.7866381507075335e-07, 3.0070588609630378e-07, 3.0204388273042629e-07, 2.9686912585345583e-07, 2.9399397760698776e-07, 2.7703356645740013e-07, 2.7225316614476467e-07, 2.6805425434872632e-07], "parent": ""}];\n if (data.length > 0) {\n ngrams.drawD3Chart(data, 1972, 2008, 1.0, "main");\n }\n' 

Answer 2

您还可以使用BeautifulSoup和JSON：

打开并阅读的链接（可能在Python 3进行的urllib2）：

link=urllib.urlopen("https://books.google.com/ngrams/graph?content=The+Godfather&year_start=1972-&year_end=2008&corpus=15&smoothing=3&direct_url=t1%3B%2CThe%20Godfather%3B%2Cc0") 
content=link.read() 

从BS4导入BS后，将其变为BeautifulSoup元素：

soup=BeautifulSoup(content, "html.parser") 
extract=soup.select('script[type="text/javascript"]')[4].string 

使用正则表达式得到的东西读入JSON（需要进口），然后只提取时间序列：

data=re.findall('(\[.*\])', extract) 

t=json.loads(data[0]) 
result= t[0]['timeseries']