我试图在使用Splash发出请求后访问cookie。 以下是我如何构建请求。从Splash请求中读取cookies
script = """
function main(splash)
splash:init_cookies(splash.args.cookies)
assert(splash:go{
splash.args.url,
headers=splash.args.headers,
http_method=splash.args.http_method,
body=splash.args.body,
})
assert(splash:wait(0.5))
local entries = splash:history()
local last_response = entries[#entries].response
return {
url = splash:url(),
headers = last_response.headers,
http_status = last_response.status,
cookies = splash:get_cookies(),
html = splash:html(),
}
end
"""
req = SplashRequest(
url,
self.parse_page,
args={
'wait': 0.5,
'lua_source': script,
'endpoint': 'execute'
}
)
该脚本是Splash文档的精确副本。
所以我试图访问在网页上设置的cookie。当我不使用Splash时,下面的代码按照我的预期工作,但在使用Splash时不起作用。
self.logger.debug('Cookies: %s', response.headers.get('Set-Cookie'))
这同时使用飞溅返回:
2017-01-03 12:12:37 [spider] DEBUG: Cookies: None
当我不使用飞溅此代码的工作,并返回该网页提供的饼干。
飞溅的文档显示该代码例如:
def parse_result(self, response):
# here response.body contains result HTML;
# response.headers are filled with headers from last
# web page loaded to Splash;
# cookies from all responses and from JavaScript are collected
# and put into Set-Cookie response header, so that Scrapy
# can remember them.
我不知道我是否正确地理解这一点,但我要说,我应该能够访问在同一饼干就像我不使用Splash一样。
中间件设置:
# Download middlewares
DOWNLOADER_MIDDLEWARES = {
# Use a random user agent on each request
'crawling.middlewares.RandomUserAgentDownloaderMiddleware': 400,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': 700,
# Enable crawlera proxy
'scrapy_crawlera.CrawleraMiddleware': 600,
# Enable Splash to render javascript
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
所以我的问题是:我怎么在使用飞溅请求访问饼干?
我已将端点添加到请求但没有结果。 response.headers.get('Set-Cookie')仍然返回一个NoneType。对于response.cookiejar,我得到一个错误:AttributeError:'SplashTextResponse'对象没有属性'cookiejar' – Casper
@Casper - 你确定所有描述的选项都设置在settings.py中吗? scrapy_splash.SplashCookiesMiddleware添加到'DOWNLOADER_MIDDLEWARES'吗? –
我用DOWNLOADER_MIDDLEWARES设置变量更新了这个问题。 – Casper