Scrapy通过IP地址抓取本地网站

我仍在尝试Scrapy，并试图抓取本地网络上的网站。该网站的IP地址为192.168.0.185。这是我的蜘蛛：Scrapy通过IP地址抓取本地网站

from scrapy.spider import BaseSpider 
class 192.168.0.185_Spider(BaseSpider): 
     name = "192.168.0.185" 
     allowed_domains = ["192.168.0.185"] 
     start_urls = ["http://192.168.0.185/"] 

     def parse(self, response): 
      print "Test:", response.headers

然后在我的蜘蛛我会执行这个shell命令来运行蜘蛛的同一目录：

scrapy crawl 192.168.0.185

而且我得到一个非常丑陋，无法读取错误信息：

2012-02-10 20:55:18-0600 [scrapy] INFO: Scrapy 0.14.0 started (bot: tutorial) 
2012-02-10 20:55:18-0600 [scrapy] DEBUG: Enabled extensions: LogStats, 
TelnetConsole,  CloseSpider, WebService, CoreStats, MemoryUsage, SpiderState 
2012-02-10 20:55:18-0600 [scrapy] DEBUG: Enabled downloader middlewares:  
HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, 
DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, 
HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats 
2012-02-10 20:55:18-0600 [scrapy] DEBUG: Enabled spider middlewares: 
HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, 
DepthMiddleware 2012-02-10 20:55:18-0600 [scrapy] DEBUG: Enabled item pipelines: 
Traceback (most recent call last): File "/usr/bin/scrapy", line 5, in <module> 
pkg_resources.run_script('Scrapy==0.14.0', 'scrapy') 
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 467, in run_script 
self.require(requires)[0].run_script(script_name, ns) 
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1200, in run_script 
execfile(script_filename, namespace, namespace) 
File "/usr/lib/python2.6/site-packages/Scrapy-0.14.0-py2.6.egg/EGG-INFO/scripts 
/scrapy", line 4, in <module> 
execute() 
File "/usr/lib/python2.6/site-packages/Scrapy-0.14.0-py2.6.egg/scrapy/cmdline.py", 
line 132, in execute 
_run_print_help(parser, _run_command, cmd, args, opts) 
File "/usr/lib/python2.6/site-packages/Scrapy-0.14.0-py2.6.egg/scrapy/cmdline.py", 
line 97, in _run_print_help func(*a, **kw) 
File "/usr/lib/python2.6/site-packages/Scrapy-0.14.0-py2.6.egg/scrapy/cmdline.py", 
line 139, in _run_command cmd.run(args, opts) 
File "/usr/lib/python2.6/site-packages/Scrapy-0.14.0-py2.6.egg/scrapy/commands 
/crawl.py", line 43, in run 
spider = self.crawler.spiders.create(spname, **opts.spargs) 
File "/usr/lib/python2.6/site-packages/Scrapy-0.14.0-py2.6.egg/scrapy 
/spidermanager.py", line 43, in create 
raise KeyError("Spider not found: %s" % spider_name) 
KeyError: 'Spider not found: 192.168.0.185'

所以后来我又蜘蛛，这实际上是一样的第一个，但它采用的是域名而不是IP地址。这个工作得很好。有谁知道交易是什么？我如何才能让Scrapy通过IP地址而不是域名来抓取网站？

from scrapy.spider import BaseSpider 
class facebook_Spider(BaseSpider): 
    name = "facebook" 
    allowed_domains = ["facebook.com"] 
    start_urls = ["http://www.facebook.com/"] 


    def parse(self, response): 
     print "Test:", response.headers

来源

2012-02-11 user1168906

嗯，我必须问 - 为什么你会*使用IP地址来描述主机？它们不像主机名那样自然而然地描述，所以我建议谨慎使用它们。 – 2012-02-11 03:24:23

我建议你在使用像scrapy，django等复杂框架之前学习Python。你可以从[Python wiki]（http://wiki.python.org/moin/BeginnersGuide/Programmers）选择教程 – reclosedev 2012-02-11 04:50:15

class 192.168.0.185_Spider(BaseSpider): 
    ...

不能使用与数字开头或包含在Python点的类名。见文档Identifiers and keywords

您可以创建这种蜘蛛有正确的名称：

$ scrapy startproject testproj 
$ cd testproj 
$ scrapy genspider testspider 192.168.0.185 
    Created spider 'testspider' using template 'crawl' in module: 
    testproj.spiders.testspider

蜘蛛定义看起来就像这样：

class TestspiderSpider(CrawlSpider): 
    name = 'testspider' 
    allowed_domains = ['192.168.0.185'] 
    start_urls = ['http://www.192.168.0.185/'] 
    ...

，也许你应该从start_urls删除www。要开始爬行，使用蜘蛛名代替主机：

$ scrapy crawl testspider

$ scrapy crawl testspider

来源

2012-02-11 04:44:45 reclosedev

也许它会是有助于在scrapy中添加一些检查以防止创建无效的类名称。 – 2012-02-19 07:01:23

Scrapy通过IP地址抓取本地网站

回答

相关问题