传递使用XPath作为参数传递给Scrapy

Question

我试图写一个网页被称为具有下列参数的通用履带：

• 允许域
• URL被抓取
• 的XPath提取网页内的价格

URL和允许的域参数似乎工作正常，但我无法让xPath参数工作。

我猜我需要声明一个变量来保持它正确，因为其他两个参数被分配给现有的类元素。

这里是我的蜘蛛：

import scrapy 
from Spotlite.items import SpotliteItem 

class GenericSpider(scrapy.Spider): 
    name = "generic" 

    def __init__(self, start_url=None, allowed_domains=None, xpath_string=None, *args, **kwargs): 
     super(GenericSpider, self).__init__(*args, **kwargs) 
     self.start_urls = ['%s' % start_url] 
     self.allowed_domains = ['%s' % allowed_domains] 
     xpath_string = ['%s' % xpath_string] 

    def parse(self, response): 
     self.logger.info('Hi, this is an item page! %s', response.url) 
     item = SpotliteItem() 
     item['url'] = response.url 
     item['price'] = response.xpath(xpath_string).extract() 
     return item 

我得到以下错误：

Traceback (most recent call last): 
    File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks 
    current.result = callback(current.result, *args, **kw) 
    File "/home/ubuntu/spotlite/spotlite/spiders/generic.py", line 23, in parse 
    item['price'] = response.xpath(xpath_string).extract() 

NameError：全局名称 'xpath_string' 没有定义

任何援助将不胜感激！

感谢，

迈克尔

Answer 1

有xpath_string作为实例变量代替：

import scrapy 
from Spotlite.items import SpotliteItem 

class GenericSpider(scrapy.Spider): 
    name = "generic" 

    def __init__(self, start_url=None, allowed_domains=None, xpath_string=None, *args, **kwargs): 
     super(GenericSpider, self).__init__(*args, **kwargs) 
     self.start_urls = ['%s' % start_url] 
     self.allowed_domains = ['%s' % allowed_domains] 
     self.xpath_string = xpath_string 

    def parse(self, response): 
     self.logger.info('Hi, this is an item page! %s', response.url) 
     item = SpotliteItem() 
     item['url'] = response.url 
     item['price'] = response.xpath(self.xpath_string).extract() 
     return item 

Answer 2

添加变量初始类的声明解决了这一问题。

import scrapy 
from spotlite.items import SpotliteItem 


class GenericSpider(scrapy.Spider): 
    name = "generic" 
    xpath_string = "" 

    def __init__(self, start_url, allowed_domains, xpath_string, *args, **kwargs): 
     super(GenericSpider, self).__init__(*args, **kwargs) 
     self.start_urls = ['%s' % start_url] 
     self.allowed_domains = ['%s' % allowed_domains] 
     self.xpath_string = xpath_string 

    def parse(self, response): 
     self.logger.info('URL is %s', response.url) 
     self.logger.info('xPath is %s', self.xpath_string) 
     item = SpotliteItem() 
     item['url'] = response.url 
     item['price'] = response.xpath(self.xpath_string).extract() 
     return item