lxml找不到Chrome提供的xpath？

Question

我正在尝试从链接的搜索页面获取公司的行业信息。我使用Chrome的开发工具获取xpath表单，但它会返回空括号。这里似乎是什么问题？

from lxml import html 
import requests 

page = requests.get('https://www.linkedin.com/search/results/companies/?keywords=cisco.com') 
tree = html.fromstring(page.content) 

industry = tree.xpath('//*[@id="ember3734"]/div/div[1]/p[1]') 

print(industry) 

Answer 1

我用selenium和phantomjs制作了脚本，因为网站使用了很多javascript。

#!/usr/bin/env python 
# -*- coding: utf-8 -*- 

import lxml.html 
import re 
from selenium import webdriver 
from time import sleep 
from selenium.webdriver import DesiredCapabilities 
from pprint import pprint 

desired_capabilities = DesiredCapabilities.PHANTOMJS.copy() 
desired_capabilities['phantomjs.page.customHeaders.User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) ' \ 
                    'AppleWebKit/537.36 (KHTML, like Gecko) ' \ 
                    'Chrome/39.0.2171.95 Safari/537.36' 
driver = webdriver.PhantomJS(desired_capabilities=desired_capabilities) 

username = 'email@email.com' 
password = 'password' 


# driver = webdriver.PhantomJS() 
driver.set_window_size(1120, 550) 
driver.get("https://www.linkedin.com") 

driver.find_element_by_id('login-password').send_keys(password) 
driver.find_element_by_id('login-email').send_keys(username) 
driver.find_element_by_id("login-submit").click() 
driver.get("https://www.linkedin.com/search/results/companies/?keywords=cisco.com") 
sleep(3) 
html = driver.page_source 
root = lxml.html.fromstring(html) 

reg = re.compile('ember-view\">\s+<h3\s+class=\"search\-result__title\s+Sans\-17px\-black\-85\%\-semibold-dense\">(.*?)<\/h3>') 
names = reg.findall(html) 

pprint(names) 

driver.quit() 

Answer 2

我觉得这个页面是由JavaScript生成的。由于请求在不执行JavaScript的情况下下载页面，因此只会获得主页面/模板，而不是您期望的数据。

尝试在Chrome浏览器中查看源页面进行确认。