Scrapy抓取多个页面错误过滤重复

刚开始使用scrapy，我试图通过整个数据库一页一页地做一个通用的搜索引擎，并抓住某些我需要的链接，但我得到这个错误当我尝试去下一页。不完全确定如何去下一页，会感谢任何帮助正确的方法！Scrapy抓取多个页面错误过滤重复

这是我的代码：（？）

class TestSpider(scrapy.Spider): 

    name = "PLC" 
    allowed_domains = ["exploit-db.com"] 

    start_urls = [ 
     "https://www.exploit-db.com/local/" 
    ] 

    def parse(self, response): 
     filename = response.url.split("/")[-2] + '.html' 
     links = response.xpath('//tr/td[5]/a/@href').extract() 
     description = response.xpath('//tr/td[5]/a[@href]/text()').extract()  

     for data, link in zip(description, links): 
      if "PLC" in data: 
       with open(filename, "a") as f: 
        f.write(data+'\n') 
        f.write(link+'\n\n') 
        f.close() 


      else: 
       pass 

     next_page = response.xpath('//div[@class="pagination"][1]//a/@href').extract() 
     if next_page: 
      url = response.urljoin(next_page[0]) 
      yield scrapy.Request(url, callback=self.parse)

但我在控制台上收到此错误

2016-06-08 16:05:21 [scrapy] INFO: Enabled item pipelines: 
[] 
2016-06-08 16:05:21 [scrapy] INFO: Spider opened 
2016-06-08 16:05:21 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 
2016-06-08 16:05:21 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023 
2016-06-08 16:05:22 [scrapy] DEBUG: Crawled (200) <GET https://www.exploit-db.com/robots.txt> (referer: None) 
2016-06-08 16:05:22 [scrapy] DEBUG: Crawled (200) <GET https://www.exploit-db.com/local/> (referer: None) 
2016-06-08 16:05:23 [scrapy] DEBUG: Crawled (200) <GET https://www.exploit-db.com/local/?order_by=date&order=desc&pg=2> (referer: https://www.exploit-db.com/local/) 
2016-06-08 16:05:23 [scrapy] DEBUG: Crawled (200) <GET https://www.exploit-db.com/local/?order_by=date&order=desc&pg=1> (referer: https://www.exploit-db.com/local/?order_by=date&order=desc&pg=2) 
2016-06-08 16:05:23 [scrapy] DEBUG: Filtered duplicate request: <GET https://www.exploit-db.com/local/?order_by=date&order=desc&pg=2> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates) 
2016-06-08 16:05:23 [scrapy] INFO: Closing spider (finished) 
2016-06-08 16:05:23 [scrapy] INFO: Dumping Scrapy stats: 
{'downloader/request_bytes': 1162, 
'downloader/request_count': 4, 
'downloader/request_method_count/GET': 4, 
'downloader/response_bytes': 40695, 
'downloader/response_count': 4, 
'downloader/response_status_count/200': 4, 
'dupefilter/filtered': 1, 
'finish_reason': 'finished', 
'finish_time': datetime.datetime(2016, 6, 8, 8, 5, 23, 514161), 
'log_count/DEBUG': 6, 
'log_count/INFO': 7, 
'request_depth_max': 3, 
'response_received_count': 4, 
'scheduler/dequeued': 3, 
'scheduler/dequeued/memory': 3, 
'scheduler/enqueued': 3, 
'scheduler/enqueued/memory': 3, 
'start_time': datetime.datetime(2016, 6, 8, 8, 5, 21, 561678)} 
2016-06-08 16:05:23 [scrapy] INFO: Spider closed (finished)

无法检索下一个页面，很想解释为什么TT

来源

2016-06-09 dumdum

您可以在您的请求中使用参数dont_filter = True：

if next_page: 
    url = response.urljoin(next_page[0]) 
    yield scrapy.Request(url, callback=self.parse, dont_filter=True)

但是你会遇到一个无限循环，因为它似乎您的xpath检索相同的链接两次（检查每页上的寻呼机，因为.pagination的第二个元素可能并不总是“下一页” 。

next_page = response.xpath('//div[@class="pagination"][1]//a/@href').extract()

而且，如果他们开始使用引导或相似，他们补充BTN BTN-默认链接的类？

我建议使用

selector.css(".pagination").xpath('.//a/@href')

代替

来源

2016-06-09 10:44:16

Scrapy抓取多个页面错误过滤重复

回答

相关问题