2017-06-16 49 views
0

我收到一条错误消息,当我刮一个配置文件。我假设我使用我的代理错误。但是这里的主要错误是什么?你们能帮不适当的部署Scrapy代理

2017-06-15 21:35:17 [scrapy.proxies] INFO: Removing failed proxy , 12 proxies left 2017-06-15 21:35:17 [scrapy.core.scraper] ERROR: Error downloading https://www.linkedin.com/in/jiajie-jacky-fan-80920083/> Traceback (most recent call last): File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks result = result.throwExceptionIntoGenerator(g) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/twisted/python/failure.py", line 393, in throwExceptionIntoGenerator return g.throw(self.type, self.value, self.tb) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/middleware.py", line 43, in process_request defer.returnValue((yield download_func(request=request,spider=spider))) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/utils/defer.py", line 45, in mustbe_deferred result = f(*args, **kw) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/handlers/init.py", line 65, in download_request return handler.download_request(request, spider) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/handlers/http11.py", line 63, in download_request return agent.download_request(request) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/handlers/http11.py", line 272, in download_request agent = self._get_agent(request, timeout) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/handlers/http11.py", line 252, in _get_agent _, _, proxyHost, proxyPort, proxyParams = _parse(proxy) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/webclient.py", line 37, in _parse return _parsed_url_args(parsed) File "/Users/jiajiefan/data_mining/lib/python2.7/site-packages/Scrapy-1.4.0-py2.7.egg/scrapy/core/downloader/webclient.py", line 21, in _parsed_url_args port = parsed.port File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urlparse.py", line 113, in port port = int(port, 10) ValueError: invalid literal fo r int() with base 10: '178.32.255.199'

回答

0

代理应该有地址以“http”等:

rq.meta['proxy'] = 'http://127.0.0.1:8123' 
+0

我试了一下,这是行不通的。结果显示代理不可用。 – dlfjj