以编程方式在R内响应报头

Question

我试图访问突出显示的响应标头：位置下面的屏幕截图中的文本仅使用R及其基于卷页的webscraping库。可以通过访问http://www.worldvaluessurvey.org/WVSDocumentationWVL.jsp，点击任何数据文件的下载并填写协议表单，在任何网络浏览器中轻松获得此点。下载在Web浏览器中自动开始。

我相信，以获得有效的cookie中的唯一方法是library(curlconverter)（见How to download a file behind a semi-broken javascript asp function with R），但这个问题的答案似乎没有足够的以编程方式确定该文件的HTTP URL，只一旦它已经知道，就下载压缩文件。

我已经粘贴下面的一些代码有不同HTTR和我周围玩curlconverter代码，但是我在这里失去了一些东西。再次，唯一的目标是以编程方式完全在R（跨平台）内确定突出显示的文本。

library(curlconverter) 
library(httr) 

browserPOST <- 
    "curl 'http://www.worldvaluessurvey.org/AJDownload.jsp' 
    -H 'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' 
    -H 'Accept-Encoding:gzip, deflate' 
    -H 'Accept-Language:en-US,en;q=0.8' 
    -H 'Cache-Control:max-age=0' 
    --compressed -H 'Connection:keep-alive' 
    -H 'Content-Length:188' 
    -H 'Content-Type:application/x-www-form-urlencoded' 
    -H 'Cookie:ASPSESSIONIDCASQAACD=IBLGBFOAEHFILMMJJCFEOEMI; JSESSIONID=50DABDEDD0B2FC370C415B4BD1855260; __atuvc=13%7C45; __atuvs=58224f37d312c42400c' 
    -H 'Host:www.worldvaluessurvey.org' 
    -H 'Origin:http://www.worldvaluessurvey.org' 
    -H 'Referer:http://www.worldvaluessurvey.org/AJDownloadLicense.jsp' 
    -H 'Upgrade-Insecure-Requests:1' 
    -H 'User-Agent:Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36'" 

form_data <- 
    list( 
     ulthost = "WVS" , 
     CMSID = "" , 
     LITITLE = "" , 
     LINOMBRE = "fas" , 
     LIEMPRESA = "asf" , 
     LIEMAIL = "asdf" , 
     LIPROJECT = "asfd" , 
     LIUSE = "1" , 
     LIPURPOSE = "asdf" , 
     LIAGREE = "1" , 
     DOID = "3996" , 
     CndWAVE = "-1" , 
     SAID = "-1" , 
     AJArchive = "WVS Data Archive" , 
     EdFunction = "" , 
     DOP = "" 
    ) 



getDATA <- (straighten(browserPOST) %>% make_req)[[1]]() 

a <- VERB(verb = "POST", url = "http://www.worldvaluessurvey.org/AJDownload.jsp", 
    httr::add_headers(Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 
     `Accept-Encoding` = "gzip, deflate", `Accept-Language` = "en-US,en;q=0.8", 
     `Cache-Control` = "max-age=0", Connection = "keep-alive", 
     `Content-Length` = "188", Host = "www.worldvaluessurvey.org", 
     Origin = "http://www.worldvaluessurvey.org", Referer = "http://www.worldvaluessurvey.org/AJDownloadLicense.jsp", 
     `Upgrade-Insecure-Requests` = "1", `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36"), 
    httr::set_cookies(`Cookie:ASPSESSIONIDCASQAACD` = "IBLGBFOAEHFILMMJJCFEOEMI", 
     JSESSIONID = "50DABDEDD0B2FC370C415B4BD1855260", `__atuvc` = "13%7C45", 
     `__atuvs` = "58224f37d312c42400c"), encode = "form",body=form_data) 

Answer 1

这是一个很好的挑战！

的问题是不相关的R输入语言。如果我们只是试图将一些数据发布到下载脚本中，我们将在任何语言中获得相同的结果。我们必须在这里处理某种安全“模式”。该网站限制用户检索文件的URL，并要求他们填写表格以提供这些链接。如果浏览器可以检索这些链接，那么我们也可以通过编写正确的HTTP调用。事情是，我们需要确切地知道我们必须打出哪些电话。为了找到这一点，我们需要看到有人点击下载时，网站会进行个别调用。以下是我发现了几个电话成功302 AJDownload.jspPOST呼叫前：

我们可以清楚地看到它，如果我们看一下AJDocumentation.jsp源，它通过使用jQuery $.get使这些调用：

$.get("http://ipinfo.io?token=xxxxxxxxxxxxxx", function (response) { 
    var geodatos=encodeURIComponent(response.ip+"\t"+response.country+"\t"+response.postal+"\t"+ 
    response.loc+"\t"+response.region+"\t"+response.city+"\t"+ 
    response.org); 

    $.get("jdsStatJD.jsp?ID="+geodatos+ 
     "&url=http%3A%2F%2Fwww.worldvaluessurvey.org%2FAJDocumentation.jsp&referer=null&cms=Documentation", 
     function (resp2) { 
    }); 
}, "jsonp"); 

然后，下面打了几个电话，我们可以看到成功POST /AJDownload.jsp与状态302 Moved Temporarily，并在其响应头的希望Location：

HTTP/1.1 302 Moved Temporarily 
Content-Length: 0 
Content-Type: text/html 
Location: http://www.worldvaluessurvey.org/wvsdc/CO00001/F00003724-WVS_Longitudinal_1981-2014_stata_dta_v_2015_04_18.zip 
Server: Microsoft-IIS/7.5 
X-Powered-By: ASP.NET 
Date: Thu, 01 Dec 2016 16:24:37 GMT 

所以，这是这个网站的安全机制。它使用ipinfo.io来存储关于其IP，位置甚至ISP组织的访客信息，就在用户即将通过点击链接发起下载之前。接收这些数据的脚本是/jdsStatJD.jsp。我没有使用ipinfo。io，也没有这个服务的API密钥（将它隐藏在我的截图中），而是我创建了一个虚拟的有效数据序列，以验证请求。 “受保护”文件的表单数据根本不需要。可以在不发布这些数据的情况下下载文件。

另外，curlconverter库不是必需的。我们所要做的只是通过使用httr库来简化GET和POST请求。我想指出的一个重要组成部分，是为了防止httrPOST功能从以下我们最后一次通话与302状态收到Location头，我们需要使用的配置设置config(followlocation = FALSE)这当然会阻止它继Location让我们从头文件中获取Location。

输出

我的[R脚本可以在命令行中运行，它可以接受DOID数值为参数，以获得所需要的文件。例如，如果我们想要得到的链接文件WVS_Longitudinal_1981-2014_stata_dta_v_2015_04_18，那么我们就必须增加其DOID（这是3724）我们的脚本的末尾使用Rscript命令调用它时：

Rscript wvs_fetch_downloads.r 3724 
[1] "http://www.worldvaluessurvey.org/wvsdc/CO00001/F00003724-WVS_Longitudinal_1981-2014_stata_dta_v_2015_04_18.zip" 

我创建的R功能通过刚好路过DOID得到你想要的每个文件地点：

getFileById <- function(fileId) 

您可以删除命令行参数解析，并通过直接传递DOID使用的功能：

#args <- commandArgs(TRUE) 
#if(length(args) == 0) { 
# print("No file id specified. Use './script.r ####'.") 
# quit("no") 
#} 

#fileId <- args[1] 
fileId <- "3724" 

# DOID=3843 : WVS_EVS_Integrated_Dictionary_Codebook v_2014_09_22 (Excel) 
# DOID=3844 : WVS_Values Surveys Integrated Dictionary_TimeSeries_v_2014-04-25 (Excel) 
# DOID=3725 : WVS_Longitudinal_1981-2014_rdata_v_2015_04_18 
# DOID=3996 : WVS_Longitudinal_1981-2014_sas_v_2015_04_18 
# DOID=3723 : WVS_Longitudinal_1981-2014_spss_v_2015_04_18 
# DOID=3724 : WVS_Longitudinal_1981-2014_stata_dta_v_2015_04_18 

getFileById(fileId) 

;最终R工作脚本

library(httr) 

getFileById <- function(fileId) { 
    response <- GET(
     url = "http://www.worldvaluessurvey.org/AJDocumentation.jsp?CndWAVE=-1", 
     add_headers(
      `Accept` = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", 
      `Accept-Encoding` = "gzip, deflate", 
      `Accept-Language` = "en-US,en;q=0.8", 
      `Cache-Control` = "max-age=0", 
      `Connection` = "keep-alive", 
      `Host` = "www.worldvaluessurvey.org", 
      `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0", 
      `Content-type` = "application/x-www-form-urlencoded", 
      `Referer` = "http://www.worldvaluessurvey.org/AJDownloadLicense.jsp", 
      `Upgrade-Insecure-Requests` = "1")) 

    set_cookie <- headers(response)$`set-cookie` 
    cookies <- strsplit(set_cookie, ';') 
    cookie <- cookies[[1]][1] 

    response <- GET(
     url = "http://www.worldvaluessurvey.org/jdsStatJD.jsp?ID=2.72.48.149%09IT%09undefined%0941.8902%2C12.4923%09Lazio%09Roma%09Orange%20SA%20Telecommunications%20Corporation&url=http%3A%2F%2Fwww.worldvaluessurvey.org%2FAJDocumentation.jsp&referer=null&cms=Documentation", 
     add_headers(
      `Accept` = "*/*", 
      `Accept-Encoding` = "gzip, deflate", 
      `Accept-Language` = "en-US,en;q=0.8", 
      `Cache-Control` = "max-age=0", 
      `Connection` = "keep-alive", 
      `X-Requested-With` = "XMLHttpRequest", 
      `Host` = "www.worldvaluessurvey.org", 
      `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0", 
      `Content-type` = "application/x-www-form-urlencoded", 
      `Referer` = "http://www.worldvaluessurvey.org/AJDocumentation.jsp?CndWAVE=-1", 
      `Cookie` = cookie)) 

    post_data <- list( 
     ulthost = "WVS", 
     CMSID = "", 
     CndWAVE = "-1", 
     SAID = "-1", 
     DOID = fileId, 
     AJArchive = "WVS Data Archive", 
     EdFunction = "", 
     DOP = "", 
     PUB = "") 

    response <- POST(
     url = "http://www.worldvaluessurvey.org/AJDownload.jsp", 
     config(followlocation = FALSE), 
     add_headers(
      `Accept` = "*/*", 
      `Accept-Encoding` = "gzip, deflate", 
      `Accept-Language` = "en-US,en;q=0.8", 
      `Cache-Control` = "max-age=0", 
      `Connection` = "keep-alive", 
      `Host` = "www.worldvaluessurvey.org", 
      `User-Agent` = "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0", 
      `Content-type` = "application/x-www-form-urlencoded", 
      `Referer` = "http://www.worldvaluessurvey.org/AJDocumentation.jsp?CndWAVE=-1", 
      `Cookie` = cookie), 
     body = post_data, 
     encode = "form") 

    location <- headers(response)$location 
    location 
} 

args <- commandArgs(TRUE) 
if(length(args) == 0) { 
    print("No file id specified. Use './script.r ####'.") 
    quit("no") 
} 

fileId <- args[1] 

# DOID=3843 : WVS_EVS_Integrated_Dictionary_Codebook v_2014_09_22 (Excel) 
# DOID=3844 : WVS_Values Surveys Integrated Dictionary_TimeSeries_v_2014-04-25 (Excel) 
# DOID=3725 : WVS_Longitudinal_1981-2014_rdata_v_2015_04_18 
# DOID=3996 : WVS_Longitudinal_1981-2014_sas_v_2015_04_18 
# DOID=3723 : WVS_Longitudinal_1981-2014_spss_v_2015_04_18 
# DOID=3724 : WVS_Longitudinal_1981-2014_stata_dta_v_2015_04_18 

getFileById(fileId) 

Answer 2

据the source of the underlying httr::request_perform，你从VERB()获取对象看起来是这样的：

res <- response(
    url = resp$url, 
    status_code = resp$status_code, 
    headers = headers, 
    all_headers = all_headers, 
    cookies = curl::handle_cookies(handle), 
    content = resp$content, 
    date = date, 
    times = resp$times, 
    request = req, 
    handle = handle 
) 

所以，你有兴趣在其headers或all_headers（response is but a structure）。如果重定向参与，all_headers将拥有多套头由curl::parse_headers()返回的，headers总是决胜盘。