我想从我的项目中提取Civic Commons Apps链接的数据。我能够获得我需要的页面的链接。但是,当我尝试打开的链接,我得到“的urlopen错误[错误-2]产品名称或服务不知道”Python Web刮 - urlopen错误[Errno -2]名称或服务未知
的网页抓取Python代码:
from bs4 import BeautifulSoup
from urlparse import urlparse, parse_qs
import re
import urllib2
import pdb
base_url = "http://civiccommons.org"
url = "http://civiccommons.org/apps"
page = urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
list_of_links = []
for link_tag in soup.findAll('a', href=re.compile('^/civic-function.*')):
string_temp_link = base_url+link_tag.get('href')
list_of_links.append(string_temp_link)
list_of_links = list(set(list_of_links))
list_of_next_pages = []
for categorized_apps_url in list_of_links:
categorized_apps_page = urllib2.urlopen(categorized_apps_url)
categorized_apps_soup = BeautifulSoup(categorized_apps_page.read())
last_page_tag = categorized_apps_soup.find('a', title="Go to last page")
if last_page_tag:
last_page_url = base_url+last_page_tag.get('href')
index_value = last_page_url.find("page=") + 5
base_url_for_next_page = last_page_url[:index_value]
for pageno in xrange(0, int(parse_qs(urlparse(last_page_url).query)['page'][0]) + 1):
list_of_next_pages.append(base_url_for_next_page+str(pageno))
else:
list_of_next_pages.append(categorized_apps_url)
我得到以下错误:
urllib2.urlopen(categorized_apps_url)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno -2] Name or service not known>
当我执行urlopen时,我应该照顾任何特定的东西吗?因为我没有看到我得到的http链接有问题。
[编辑] 在第二轮我得到了以下错误:
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
raise URLError(err)
相同的代码运行在我朋友的Mac很好,但在我的Ubuntu 12.04将失败。
另外我试着运行scraper wiki中的代码,并成功完成。但是很少有url丢失(与mac相比)。这些行为是否有任何理由?
“categorized_apps_url”在错误点的值是多少? – kojiro 2012-07-23 02:27:25
另外,我知道这种评论通常被认为令人讨厌,但如果您使用[httplib2](http://code.google.com/p/httplib2/)或[请求]( http://docs.python-requests.org/en/latest/index.html)而不是'urllib2'。他们提供了一套更完整的功能来处理http。 – kojiro 2012-07-23 02:29:13
你的脚本按原样从我的电脑上正常运行。我使用python 2.7在Mac上运行,并使用BeautifulSoup 3.2和4.0尝试了它,并且在这两种情况下它都会返回一个包含69个主要链接和117个下一页链接的列表。我怀疑它是你的系统上阻塞python的东西。你是否尝试过直接ping这些网址?也许你有防病毒软件阻止你的脚本? – 2012-07-23 03:24:32