如何使用python从列表中抓取url

我想抓取列表中的url。基本上我正在刮一个网站，我刮了一个链接，我发现特定的链接刮了这些链接，我搜索另一个特定的链接刮了它。我的代码：如何使用python从列表中抓取url

from bs4 import BeautifulSoup 
import urllib.request 
import re 
r = urllib.request.urlopen('http://i.cantonfair.org.cn/en/ExpExhibitorList.aspx?k=glassware') 
soup = BeautifulSoup(r, "html.parser") 
links = soup.find_all("a", href=re.compile(r"expexhibitorlist\.aspx\?categoryno=[0-9]+")) 
linksfromcategories = ([link["href"] for link in links]) 

string = "http://i.cantonfair.org.cn/en/" 
linksfromcategories = [string + x for x in linksfromcategories] 
subcatlinks = list() 
for link in linksfromcategories: 
    response = urllib.request.urlopen(link) 
    soup2 = BeautifulSoup(response, "html.parser") 
    links2 = soup2.find_all("a", href=re.compile(r"ExpExhibitorList\.aspx\?categoryno=[0-9]+")) 
    linksfromsubcategories = ([link["href"] for link in links2]) 
    subcatlinks.append(linksfromsubcategories) 
responses = urllib.request.urlopen(subcatlinks) 
soup3 = BeautifulSoup(responses, "html.parser") 
print (soup3)

而且我得到的错误

Traceback (most recent call last): 
    File "D:\python\phase2.py", line 46, in <module> 
    responses = urllib.request.urlopen(subcatlinks) 
    File "C:\Users\amanp\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 162, in urlopen 
    return opener.open(url, data, timeout) 
    File "C:\Users\amanp\AppData\Local\Programs\Python\Python35-32\lib\urllib\request.py", line 456, in open 
    req.timeout = timeout 
AttributeError: 'list' object has no attribute 'timeout'

来源

2016-02-25 Aman Kumar

我已经表明了错误，检查它 –

您可以在一个链接一次只能传递给urllib.request.urlopen，而不是他们的整个列表。

所以你需要另一个循环是这样的：

for link in subcatlinks: 
    response = urllib.request.urlopen(link) 
    soup3 = BeautifulSoup(response, "html.parser") 
    print(soup3)

来源

2016-02-25 21:04:49

我试过，但它仍然给我的错误AttributeError的：“名单”对象有没有属性' timeout' –

该数组存储列表为[[“link1”，“link2”]，[“anotherlink1”，“anotherlinks2”]]这个问题？ –

是的，就是这个问题，你必须将一个字符串传递给'urlopen（）'，而不是一个列表。 –

如何使用python从列表中抓取url

回答

相关问题