在Scrapy中创建可编辑的CrawlSpider规则

Question

我一直在尝试创建一个简单的Scrapy CrawlSpider脚本，该脚本可以很容易地进行更改，但我无法弄清楚如何让链接提取器规则正常工作。

这里是我的代码：

class LernaSpider(CrawlSpider): 
"""Our ad-hoc spider""" 

name = "lerna" 

def __init__(self, url, allow_follow='.*', deny_follow='', allow_extraction='.*', deny_extraction=''): 
    parsed_url = urlparse(url) 
    domain = str(parsed_url.netloc) 
    self.allowed_domains = [domain] 
    self.start_urls = [url] 
    self.rules = (
     # Extract links 
     # and follow links from them (since no callback means follow=True by default). 
     Rule(SgmlLinkExtractor(allow=(allow_follow,), deny=(deny_follow,))), 

     # Extract links and parse them with the spider's method parse_item 
     Rule(SgmlLinkExtractor(allow=(allow_extraction,), deny=(deny_extraction,)), callback='parse_item'), 
    ) 

    super(LernaSpider, self).__init__() 

def parse_item(self, response): 

    print 'Crawling... %s' % response.url 
    # more stuff here 

我有这样的代码，但我从来没有得到过允许/拒绝规则，以正常工作，我真的不知道为什么。是否留下空弦使它拒绝一切？我认为，因为这是一个RE，如果我输入'。*'或其他什么，它只会做一个全面的否定。

任何帮助，将不胜感激。

Answer 1

你是在自己实例化蜘蛛吗？是这样的：

spider = LernaSpider('http://example.com') 

，否则如果你是从你使用的URL作为构造函数的第一个参数不正确的命令行运行$scrapy crawl lerna（应该是名字），你也不会再传递到超。也许试试这个：

class LernaSpider(CrawlSpider): 
    """Our ad-hoc spider""" 

    name = "lerna" 

    def __init__(self, name=None, url=url, allow_follow='.*', deny_follow='', allow_extraction='.*', deny_extraction='', **kw): 
     parsed_url = urlparse(url) 
     domain = str(parsed_url.netloc) 
     self.allowed_domains = [domain] 
     self.start_urls = [url] 
     self.rules = (
      # Extract links 
      # and follow links from them (since no callback means follow=True by default). 
      Rule(SgmlLinkExtractor(allow=allow_follow, deny=deny_follow)), 

      # Extract links and parse them with the spider's method parse_item 
      Rule(SgmlLinkExtractor(allow=allow_extraction, deny=deny_extraction), callback='parse_item'), 
     ) 
     super(LernaSpider, self).__init__(name, **kw) 

    def parse_item(self, response): 
     print 'Crawling... %s' % response.url 
     # more stuff here 

正则表达式的东西看起来很好：空值允许所有和拒绝没有。