2016-07-29 88 views
2

希望你不需要在这里的整套代码,但我有一个问题,我解析HTML,使用XPath,我没有得到我' d预计:XPath不工作,因为我期望它

# here is the current set of tags I'm interested in 
html = '''<div style="padding-top: 10px; clear: both; width: 100%;"> 
     <a href="http://www.amazon.com/review/R41M1I2K413NG/ref=cm_aya_cmt?ie=UTF8&ASIN=B013IZY7RU#wasThisHelpful" ><img src="http://g-ecx.images-amazon.com/images/G/01/x-locale/communities/discussion_boards/comment-sm._CB192250344_.gif" width="16" alt="Comment" hspace="3" align="absmiddle" height="16" border="0" /></a>&nbsp;<a href="http://www.amazon.com/review/R41M1I2K413NG/ref=cm_aya_cmt?ie=UTF8&ASIN=B013IZY7RU#wasThisHelpful" >Comment</a>&nbsp;|&nbsp;<a href="http://www.amazon.com/review/R41M1I2K413NG/ref=cm_cr_rdp_perm" >Permalink</a>''' 

我试图让第一a标签,这是一个长的URL的href值。为此,我使用以下代码

from lxml import etree 
import StringIO 

parser = etree.HTMLParser(encoding="utf-8") 
tree = etree.parse(StringIO.StringIO(html), parser) 

style = 'padding-top: 10px; clear: both; width: 100%;' 
xpath = "//div[@style='%s']" % style 
xpath += "/a[1]/@href" 

# use the XPath expression above to pull out the href value 
tree.xpath(xpath) 


['http://www.amazon.com/review/R41M1I2K413NG/ref=cm_aya_cmt?ie=UTF8&ASIN=B013IZY7RU#wasThisHelpful'] 

这适用于当我拉出正在使用的零件并将其粘贴为字符串时。这与我使用request.get()调用构建的tree完全不一样,我无法弄清楚为什么?它返回的是:

['http://www.amazon.com/review/R41M1I2K413NG] 

而我不明白为什么。我明白我在这里黑暗中拍摄,但我只是希望有人遇到了“属性截断的XPath返回值”问题。

编辑:

下面是我目前使用的全部代码,但它不工作。它返回上面的截断值。

from lxml import etree 
import requests 
import StringIO 
from requests.packages.urllib3.util.retry import Retry 
from requests.adapters import HTTPAdapter 


session = requests.Session() 
retries = Retry(total=5, backoff_factor=1, status_forcelist=[502, 503, 504]) 
session.mount('http://www.amazon.com', HTTPAdapter(max_retries=retries)) 
parser = etree.HTMLParser(encoding=encoding) 

url = "http://www.amazon.com/gp/cdp/member-reviews/ARPJ98Y7U8K5H?ie=UTF8&display=public&page=3&sort_by=MostRecentReview" 
page = session.get(url, timeout=5) 
tree = etree.parse(StringIO.StringIO(page.text), parser) 

style = 'padding-top: 10px; clear: both; width: 100%;' 
xpath = "//div[@style='%s']" % style 
xpath += "/a[1]/@href" 

# use the XPath expression above to pull out the href value 
tree.xpath(xpath) 

编辑2:

这样确实出于某种原因。而不是创建一个session对象,并使用提交get请求,然后传递到parser,只需将url字符串传递给parser作品:

url = "http://www.amazon.com/gp/cdp/member-reviews/ARPJ98Y7U8K5H?ie=UTF8&display=public&page=3&sort_by=MostRecentReview" 


tree = etree.parse(url, parser) 



for e in tree.xpath("//div[@style='padding-top: 10px; clear: both; width: 100%;']/a[1]/@href"): 

    print e 

据我了解,通过多个网址的循环时会话对象将持续加速进程的连接属性。如果我使用etree.parse(url, parser)方法,我担心我会失去效率。

+1

我们如何重现这一点?请向我们展示返回截断属性值的确切代码。 – mzjn

+0

调用'request.get()'时,你使用的是什么URL? – Markus

+0

http://www.amazon.com/gp/cdp/member-reviews/ARPJ98Y7U8K5H?ie=UTF8&display=public&page=3&sort_by=MostRecentReview –

回答

0

使用您提供的URL,下面的Python代码:

url = "http://www.amazon.com/gp/cdp/member-reviews/ARPJ98Y7U8K5H?ie=UTF8&display=public&page=3&sort_by=MostRecentReview" 

from lxml import etree 
parser = etree.HTMLParser(encoding="utf-8") 
tree = etree.parse(url, parser) 

for e in tree.xpath("//div[@style='padding-top: 10px; clear: both; width: 100%;']/a[1]/@href"): 

    print e 

结果在下面的输出:

> python ~/test.py 

http://www.amazon.com/review/RM8YYCQ57K2CL/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B00J9PAZIO#wasThisHelpful 
http://www.amazon.com/review/R41M1I2K413NG/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B013IZY7RU#wasThisHelpful 
http://www.amazon.com/review/R3DT6VUDGIT9SK/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B000VYD0MA#wasThisHelpful 
http://www.amazon.com/review/RGFW1JM4151MW/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B00TQQN5G0#wasThisHelpful 
http://www.amazon.com/review/R3I9FFX0MVF1BW/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B0048A7NF8#wasThisHelpful 
http://www.amazon.com/review/R24TTSQY34VME8/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B0115ZHH68#wasThisHelpful 
http://www.amazon.com/review/R3C49WWMNQZ007/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B00ABAWHJ6#wasThisHelpful 
http://www.amazon.com/review/R37724EHW829NB/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B00TO5Y3FK#wasThisHelpful 
http://www.amazon.com/review/RQKGM5FRXVYSX/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B0051QUWKG#wasThisHelpful 
http://www.amazon.com/review/R1DW61PMGUDMDJ/ref=cm_aya_cmt/159-5911033-5890330?ie=UTF8&ASIN=B000N8Q2P6#wasThisHelpful 

使用你提供的结果的示例代码:

http://www.amazon.com/review/RM8YYCQ57K2CL 
http://www.amazon.com/review/R41M1I2K413NG 
http://www.amazon.com/review/R3DT6VUDGIT9SK 
http://www.amazon.com/review/RGFW1JM4151MW 
http://www.amazon.com/review/R3I9FFX0MVF1BW 
http://www.amazon.com/review/R24TTSQY34VME8 
http://www.amazon.com/review/R3C49WWMNQZ007 
http://www.amazon.com/review/R37724EHW829NB 
http://www.amazon.com/review/RQKGM5FRXVYSX 
http://www.amazon.com/review/R1DW61PMGUDMDJ 

这是由于以下事实:HTML页面中的任何URL都不会由session.get()h返回任何GET参数;或者是因为在这种情况下服务器不返回带有GET参数的URL,或者因为requests剥离了GET参数。

+0

是的,那正是我正在做的......我必须用新鲜的眼睛回到它。谢谢你的帮助。 –

+0

因此,当我使用'etree.parse(url,parser)'时,它可以工作。但是,如果首先从'session.get(url)'获取HTML并传递'text'属性,比如'etree.parse(page.text,parser)',那么我得到的结果不正确。我想使用'session.get()'b/c它有助于保持请求之间的连接。 –

相关问题