如何将2个蜘蛛的所有结果与scrapy放在一个XML中？

Question

我用Scrapy制作了2个蜘蛛，我需要将它们放在一个脚本中，并将所有结果放在一个XML中。

以下是一些管理2个蜘蛛的方法，但我无法将结果放在一个XML中。

http://doc.scrapy.org/en/latest/topics/practices.html

有什么办法，推出1个脚本2个蜘蛛和收集在一个文件中所有的结果？

Answer 1

在你scrapy项目中创建一个名为script.py添加蜘蛛文件code.Assuming名 以下行Python脚本是spider_one.py和spider_two.py和你的蜘蛛分别SpiderOne和SpiderTwo。 所以在你script.py中添加。

from spiders.spider_one import SpiderOne 
from spiders.spider_two import SpiderTwo 

# scrapy api 
from scrapy import signals, log 
from twisted.internet import reactor 
from scrapy.crawler import Crawler 
from scrapy.settings import Settings 


file = "your_file.json" #your results 
TO_CRAWL = [SpiderOne,SpiderTwo] 


# list of crawlers that are running 
RUNNING_CRAWLERS = [] 

def spider_closing(spider): 
    """Activates on spider closed signal""" 
    log.msg("Spider closed: %s" % spider, level=log.INFO) 
    RUNNING_CRAWLERS.remove(spider) 
    if not RUNNING_CRAWLERS: 
     reactor.stop() 

log.start(loglevel=log.DEBUG) 
for spider in TO_CRAWL: 
    settings = Settings() 
    settings.set("USER_AGENT", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36") 
    settings.set("FEED_FORMAT",'json') 
    settings.set("FEED_URI",file) 
    # settings.set("ITEM_PIPELINES",{ 'pipelines.CustomPipeline': 300}) 
    settings.set("DOWNLOAD_DELAY",1) 
    crawler = Crawler(settings) 
    crawler_obj = spider() 
    RUNNING_CRAWLERS.append(crawler_obj) 

    # stop reactor when spider closes 
    crawler.signals.connect(spider_closing, signal=signals.spider_closed) 
    crawler.configure() 
    crawler.crawl(crawler_obj) 
    crawler.start() 

# blocks process so always keep as the last statement 
reactor.run() 

该示例适用于json，但也可以适用于xml。