Scrapy SgmlLinkExtractor referer none

Question

我想让我的蜘蛛工作。 这是我在蜘蛛代码：

start_urls = ["http://www.khmer24.com/"] 

rules = (
Rule(SgmlLinkExtractor(allow=(r'ad/\w+/67-\d+\.html',), 
    ), 
    callback='parse_items'), 
) 

样品网址的样子： http://www.khmer24.com/ad/honda-click-2012-98/67-258149.html

我想保持“广告”和“67-”

的scrapy crawl khmer24输出是：

Crawled (200) <GET http://www.khmer24.com/> (referer: None) 

我想不通为什么 这里是我的整个代码：

from scrapy.contrib.spiders import CrawlSpider, Rule 
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor 
from scrapy.selector import HtmlXPathSelector 


class MySpider(CrawlSpider): 
    name = "khmer24" 
    allowed_domains = ["khmer24.com"] 
    start_urls = ["http://www.khmer24.com/"] 

    rules = (
    Rule(SgmlLinkExtractor(allow=(r'ad/\w+/67-\d+\.html',), 
     ), 
     callback='parse_items'), 
    ) 

    def parse_items(self, response): 
     hxs = HtmlXPathSelector(response) 
     titles = hxs.select("//div[@class='innerbox']/h1/text()") 
     return(titles) 

Answer 1

所以你的问题是，“为什么我的引荐没有？从start_urls制成“

从您的日志输出

Crawled (200) <GET http://www.khmer24.com/> (referer: None) 

从start_urls来，而不是链接提取行。请求默认不包含参照标头，您可以通过发出手动添加页眉requests yourself。