2013-04-09 73 views
0
from scrapy import project, signals 
from scrapy.crawler import Settings 
from scrapy.crawler import CrawlerProcess 
from scrapy.xlib.pydispatch import dispatcher 
from multiprocessing.queues import Queue 
import multiprocessing 

class CrawlerWorker(multiprocessing.Process): 

    def __init__(self, spider, result_queue): 
     multiprocessing.Process.__init__(self) 
     self.result_queue = result_queue 

     self.crawler = Crawler(Settings()) 
     if not hasattr(project, 'crawler'): 
      self.crawler.install() 
     self.crawler.configure() 

     self.items = [] 
     self.spider = spider 
     dispatcher.connect(self._item_passed, signals.item_passed) 

    def _item_passed(self, item): 
     self.items.append(item) 

    def run(self): 
     self.crawler.crawl(self.spider) 
     self.crawler.start() 
     self.crawler.stop() 
     self.result_queue.put(self.items) 

尝试从scrapy.conf.settings只使用CrawlerProcess(设置)我得到一个错误,它看起来像有之间的scrapy医生说这里什么http://doc.scrapy.org/en/latest/topics/practices.htmlScrapy:导入错误:无法导入名称设置

的差异

我正在遵循一个较旧的scrapy版本,我正在试图使它与0.16的scrapy一起工作。

这是运行python脚本后的错误。

Traceback (most recent call last): 
    File "server.py", line 5, in <module> 
    from scraper import Scraper 
    File "/home/me/spider/spider/scraper.py", line 6, in <module> 
    from crawlerworker import CrawlerWorker 
    File "/home/me/spider/spider/crawlerworker.py", line 2, in <module> 
    from scrapy.crawler import Settings 
ImportError: cannot import name Settings 
+0

你能错误添加到问题吗? – Talvalin 2013-04-09 10:10:33

+0

更新了有错误的问题 – KJW 2013-04-09 15:25:21

回答

3

尝试:

from scrapy.settings import Settings 
相关问题