尝试/探测与scapy的请求

我想知道是否有可能使用scrapy请求在进行页面的实际处理之前检查url的有效性（url不是事先知道的，而是它们的不同模式出现可能会被测试）。失败的示例代码如下。（所使用的变量reties为简单起见，该测试条件也可以像s.th if response.code != 200 ）尝试/探测与scapy的请求

代码失败，因为在第二回调的端部（parse_page_2）控制不返回到第一回调（ parse_page_1），即使发出新的请求，其回调为parse_page_1。这是为什么发生？我知道基于urllib2的解决方案表示为here，只是检查是否可以严格在scrapy环境中完成。

import scrapy 

from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors import LinkExtractor 
from scrapy.http import Request 

class MySpider(CrawlSpider): 
    name = 'alexa' 
    allowed_domains = ['alexa.com'] 
    start_urls = ['http://www.alexa.com'] 
    retries = 0 
    rules = (
     # Extract links matching 'category.php' (but not matching 'subsection.php') 
     # and follow links from them (since no callback means follow=True by default). 
     # Rule(LinkExtractor(allow=('topsites',))), 

     # Extract links matching 'item.php' and parse them with the spider's method parse_item 
     Rule(LinkExtractor(allow=('topsites',)), callback='parse_page1'), 
    ) 

    def parse_page1(self, response): 
     if self.retries < 5: 
      self.retries += 1 
      print 'Retries in 1: ', self.retries 
      return scrapy.Request("http://www.alexa.com/siteieekeknfo/google.com", 
           meta={'dont_merge_cookies': True, 
           'dont_redirect': False, 
           "handle_httpstatus_list": [301, 302, 303, 404]}, 
           callback=self.parse_page2) 
     else: 
      print "Finished in 1" 

    def parse_page2(self, response): 
     if self.retries < 5: 
      self.retries += 1 
      print 'Retries in 2: ', self.retries 
      return scrapy.Request("http://www.alexa.com/siteieekeknfo/google.com", 
           meta={'dont_merge_cookies': True, 
           'dont_redirect': False, 
           "handle_httpstatus_list": [301, 302, 303, 404]}, 
           callback=self.parse_page1) 
     else: 
      print "Finished in 2"

将抓取结果粘贴here。

来源

2015-10-06 pkaramol

你可能使用了错误的蜘蛛。尝试蜘蛛，蜘蛛和直接从start_requests产量，因为你已经知道你的网址。 http://scrapy.readthedocs.org/zh/latest/topics/spiders.html#scrapy-spider – digenishjkl

递归回调似乎工作：

import random 


from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors import LinkExtractor 
from scrapy.http import Request 

class MySpider(CrawlSpider): 
    name = 'alexa' 
    allowed_domains = ['alexa.com'] 
    start_urls = ['http://www.alexa.com']  
    rules = (
     Rule(LinkExtractor(allow=('topsites',)), callback='parse_page1'), 
    ) 

    _retries = 0 

    _random_urls = [ 
     'http://www.alexa.com/probablydoesnotexist', 
     'http://www.alexa.com/neitherdoesthis', 
     'http://www.alexa.com/siteinfo/google.com' 
    ] 

    def parse_page1(self, response): 
     print "Got status: ", response.status 
     if self._retries == 0 or response.status != 200: 
      self._retries += 1 
      print 'Retries in 1: ', self._retries 
      return Request(random.choice(self._random_urls), 
           meta={"handle_httpstatus_list": [301, 302, 303, 404]}, 
           callback=self.parse_page1) 
     else: 
      print "Exiting"

来源

2015-10-09 16:03:55 pkaramol

尝试/探测与scapy的请求

回答

相关问题