用BeautifulSoup刮时链接丢失

我想从whoscored获得所有玩家的链接。问题是，BS无法找到类player-link的任何链接。什么是获得这些链接的最佳方式？它们存在于检查员身上。用BeautifulSoup刮时链接丢失

这是我到目前为止的代码：

import requests 
from bs4 import BeautifulSoup as BS 
import re 
from incapsula import IncapSession 
import json 

session = IncapSession() 
links = "https://www.whoscored.com/Matches/1080633/Live/England-Premier-League-2016-2017-Manchester-United-Arsenal" 
try: 
    response = session.get(links).text 
except IncapBlocked as e: 
    raise 
soup = BS(response, "html.parser") 
player_site = soup.find("a", text = "Player Statistics") 
player_link = player_site.get("href") 
player_link = "https://www.whoscored.com" + player_link 
try: 
    response_players = session.get(player_link).text 
except IncapBlocked as e: 
    raise 
soup_players = BS(response_players, "html.parser") 
soup_players.find_all("a", class_ ="player-link")

感谢您的帮助。

编辑：

现在我读，我得到不同的结果，因为该页面通过JavaScript部分加载。我该如何解决这个问题？

来源

2017-06-18 Jure Štabuc

你会想使用硒。

>>> from selenium import webdriver 
>>> driver = webdriver.Chrome() 
>>> driver.get('https://www.whoscored.com/Matches/1080633/LiveStatistics/England-Premier-League-2016-2017-Manchester-United-Arsenal') 
>>> links = driver.find_elements_by_xpath('.//a[@class="player-link"]') 
>>> for link in links: 
...  link.get_attribute('href') 
...  
'https://www.whoscored.com/Players/79554/Show/David-de-Gea' 
'https://www.whoscored.com/Players/18296/Show/Antonio-Valencia' 
'https://www.whoscored.com/Players/70050/Show/Marcos-Rojo' 
'https://www.whoscored.com/Players/81726/Show/Phil-Jones' 
'https://www.whoscored.com/Players/23220/Show/Matteo-Darmian' 
'https://www.whoscored.com/Players/97752/Show/Paul-Pogba' 
'https://www.whoscored.com/Players/71174/Show/Ander-Herrera' 
'https://www.whoscored.com/Players/2115/Show/Michael-Carrick' 
'https://www.whoscored.com/Players/25363/Show/Juan-Mata' 
'https://www.whoscored.com/Players/300299/Show/Marcus-Rashford' 
'https://www.whoscored.com/Players/122366/Show/Anthony-Martial' 
'https://www.whoscored.com/Players/29544/Show/Morgan-Schneiderlin' 
'https://www.whoscored.com/Players/70033/Show/Daley-Blind' 
'https://www.whoscored.com/Players/3859/Show/Wayne-Rooney' 
'https://www.whoscored.com/Players/33831/Show/Sergio-Romero' 
'https://www.whoscored.com/Players/110154/Show/Memphis-Depay' 
'https://www.whoscored.com/Players/109000/Show/Jesse-Lingard' 
'https://www.whoscored.com/Players/8166/Show/Ashley-Young' 
'https://www.whoscored.com/Players/6775/Show/Petr-Cech' 
'https://www.whoscored.com/Players/91879/Show/Carl-Jenkinson' 
'https://www.whoscored.com/Players/30051/Show/Laurent-Koscielny' 
'https://www.whoscored.com/Players/80921/Show/Shkodran-Mustafi' 
'https://www.whoscored.com/Players/23072/Show/Nacho-Monreal' 
'https://www.whoscored.com/Players/125209/Show/Mohamed-Elneny' 
'https://www.whoscored.com/Players/69738/Show/Francis-Coquelin' 
'https://www.whoscored.com/Players/13796/Show/Theo-Walcott' 
'https://www.whoscored.com/Players/13756/Show/Mesut-%C3%96zil' 
'https://www.whoscored.com/Players/26820/Show/Aaron-Ramsey' 
'https://www.whoscored.com/Players/25244/Show/Alexis-S%C3%A1nchez' 
'https://www.whoscored.com/Players/24444/Show/Olivier-Giroud' 
'https://www.whoscored.com/Players/89401/Show/Granit-Xhaka' 
'https://www.whoscored.com/Players/84146/Show/Alex-Oxlade-Chamberlain' 
'https://www.whoscored.com/Players/76810/Show/Gabriel' 
'https://www.whoscored.com/Players/36745/Show/David-Ospina' 
'https://www.whoscored.com/Players/136824/Show/Alex-Iwobi' 
'https://www.whoscored.com/Players/27550/Show/Kieran-Gibbs'

来源

2017-06-18 16:14:05

谢谢！完美作品 –

非常欢迎您！ –

用BeautifulSoup刮时链接丢失

回答

相关问题