如何让cURL绕过这个僵硬的僵尸程序？

Question

我需要能够以编程方式读或刮擦该URL的内容：

https://www.toysrus.com/product?productId=122872496

我试过设置用户代理，饼干，各种。我也试着在Chrome“复制为卷曲”从开发者工具 - 命令>网络，这给下面卷曲bash命令：

curl 'https://www.toysrus.com/product?productId=122872496' -H 'if-none-match: W/"2318d-c7pvMOCypyagcnOWwd+nRwWuY3M"' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-GB,en;q=0.8,en-US;q=0.6' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'cache-control: max-age=0' -H 'authority: www.toysrus.com' -H 'cookie: GUID=15F299082881F0F; AMCVS_B9241CFE53309B6A0A490D45%40AdobeOrg=1; _abck=CA9FF35985405E426CA113364E2843B95C7A361ECC7F0000CAC6E559CE126315~0~gKgTXIlJpZdnvAbG5IkQwQDNivNRDOZlT2fWOD3JtYs=~-1~-1; entry_domain=toysrus; UrCapture=32c9b0fc-a726-20df-9a6f-62e5b646ae80; Omni_ShopperID=59e5c6d2f0ada2826d8ffee0; invodoSession=qooDReJQgHIivnpF47JvZK; invodoVisitor=A2PNHsy8xb7HkbYFs0WmFv; RES_TRACKINGID=582138174429307; ResonanceSegment=1; ak_bmsc=07593E6331CACE711A54A233912585F35C7A361E8B6000009068E85921E49529~pl51CsTeE5lRXhxDt77+3bSwhUsz0NHEbHDZviNsf5zsoLDiMPES8naaZBbn66eCN80VGsaW+FKJpBRsZmnYSIdqgplNwdRljB7RHos/Mz880QrJOIiIMx8wkCaJ2s5NYFQ3idkdIeql4t4RztiH2oRraA8Bue0r0Ck9wsc8yXmtDZ1hDdGPn4+LqZtrMhlDxZ6cwu1TODBrQSBd5Ni5eGTRvJe1/oauUHr9ztu6ayRaw=; bm_sz=3234D15C4A9BAF5E11789569DCD4FD6C~QAAQHjZ6XO9Ayy5fAQAAt3TYMwFkUQkzB9j2O8d5YvvKQfRZdHumrTmjvdQ1iPQKYi+F0p7HcKT0cVl7JYImlqc9DVfB6C3At4C+9S4b+B4iLA2IAi9fU7gcH6GfDoQxi0iPBbQLk98gkY1cqAPfQbUsjwcUbJ0l+JYY7QjWSET3e054Al8z5jVSTGpFVqlx; AMCV_B9241CFE53309B6A0A490D45%40AdobeOrg=-1330315163%7CMCIDTS%7C17459%7CMCMID%7C52999333410090394193783160145267952168%7CMCAAMLH-1508835659%7C6%7CMCAAMB-1509008146%7CxfQZlSXQfPyHZHNXzhww8jeiMq5K2sHrXyNvM9qY7m_hTGU%7CMCOPTOUT-1508410546s%7CNONE%7CMCAID%7CNONE; TRUSESSIONID=0KYz7VGR4y7wsZpUXcvpdLFQfnRdJxJJf1w6M9JbLJVNqzu1xIPV!-1065198256; mbox=PC#7086875003bc497a93590bb613c34b50.17_21#1509615639|session#888c37d2b3124c40878eb0a1792e3265#1508407899|check#true#1508406099; s_ppvl=en_US%253A%2520TRU%253A%2520Product%2520Detail%253A%2520122872496%253A%2520LEGO%2520City%2520Jungle%2520Explorers%2520Jungle%2520Halftrack%2520Mission%2520%252860159%2529%2C21%2C21%2C759%2C1536%2C759%2C1536%2C864%2C1.25%2CL; s_cc=true; __utma=136881205.1239421491.1508230865.1508350392.1508403351.3; __utmc=136881205; __utmz=136881205.1508403351.3.3.utmcsr=lookingforagooddeal.com|utmccn=(referral)|utmcmd=referral|utmcct=/product/test-product/; __pr.7tp=rgrgcsjgwm; utag_main=v_id:015f29909284001f515df5e18dfc04073003306b00978$_sn:3$_ss:0$_st:1508407846067$ses_id:1508403348621%3Bexp-session$_pn:5%3Bexp-session; _cp_dtm_store_name=toysrus; OAS_SC1=1508406046406; IR_PI=1508230866697.bml86prs4m; _br_uid_2=uid%3D6871658334943%3Av%3D12.0%3Ats%3D1508230867182%3Ahc%3D10; PrefID=21-5485403154; s_nr=1508407773155-Repeat; s_previousPageName=en_US%3A%20TRU%3A%20Product%20Detail%3A%20122872496%3A%20LEGO%20City%20Jungle%20Explorers%20Jungle%20Halftrack%20Mission%20%2860159%29; s_cpm=%5B%5B%27ch%27%2C%271508407773158%27%5D%5D; TRUSESSIONID_SET_TIMESTAMP=1508407838524; bm_sv=08D1BE1D2408D1E12F94213DC168DABC~B3fBu73AESdKjl6l/qB2U3aCBKuZw4SCcjmyc+h+CWYxT1NJ0fXXP6GlFba3e+tKujhnRf47tnMJHAn83pK816cqr6bxeL5m0yn1yo9SPOiYNXSZsE35r/9sYgAQhRWDl7Gd4AljS43B80m36ONZMb6q9kL9aT3oJ3sooY69nAE=; s_ppv=en_US%253A%2520TRU%253A%2520Product%2520Detail%253A%2520122872496%253A%2520LEGO%2520City%2520Jungle%2520Explorers%2520Jungle%2520Halftrack%2520Mission%2520%252860159%2529%2C21%2C21%2C759%2C1536%2C419%2C1536%2C864%2C1.25%2CL; RT="sl=0&ss=1508406037346&tt=0&obo=0&sh=&dm=toysrus.com&si=d98b4ff0-ddd2-47e6-8a7c-761ce2d21e15&bcn=%2F%2F36e4f0e7.akstat.io%2F&ld=1508406046108&r=https%3A%2F%2Fwww.toysrus.com%2Fproduct%3F328e8990a18940b61934f007b9211205&ul=1508407960823"' -H 'referer: https://foundthru.com/product/lego-city-jungle-explorers/' --compressed 

连接发起，但它只是超时。

我也试过：

wget http://www.toysrus.com 

其中给出这个，然后就挂断永远（除非我添加了超时）：

我也使用节点尝试。 js，phantomjs和horseman使用这样的代码：

horseman 
    .cookies(params.cookies_file_path) 
    .open(params.url) 
    .status() 
    .then(function(status) { 
     horseman.html() 
     .then(function(html) { 
       callback({status: status, data: html}); 
       return horseman.close(); 
     }); 
    }) 
    .catch(function (err) { 
     console.log(err); 
     horseman.close(); 
     callback(false); 
    }); 

我错过了什么？如何获取cURL或phantomJS来获取此页面？

Answer 1

我在PHP中使用curl很多。您可能会将我在此发布的内容适应您的环境。

没有bot拦截器。这是一个简单的https：//获取
你很可能得到挂了没有做重定向到https：//

更改URL为https：//
使用http：//导致重定向到https：//开头
然后你不想SSL证书问题：
添加curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

该代码测试和工程：

<?php 
header('Content-Type: text/plain; charset=utf-8'); 
$request = array(); 
$request[] = 'host:www.toysrus.com'; 
$request[] = 'Connection: keep-alive'; 
$request[] = 'Pragma: no-cache'; 
$request[] = 'Cache-Control: no-cache'; 
$request[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'; 
$request[] = 'User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36'; 
$request[] = 'DNT: 1'; 
$request[] = 'Accept-Encoding: gzip, deflate'; 
$request[] = 'Accept-Language: en-US,en;q=0.8'; 


$url = 'https://www.toysrus.com/product?productId=122872496'; 
$ch = curl_init($url); 

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false); 
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
curl_setopt($ch, CURLOPT_POST, false); 
curl_setopt($ch, CURLOPT_HTTPHEADER, $request); 
curl_setopt($ch, CURLOPT_ENCODING,""); 
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); 
curl_setopt($ch, CURLOPT_TIMEOUT,10); 
curl_setopt($ch, CURLOPT_FAILONERROR,true); 
curl_setopt($ch, CURLOPT_ENCODING,""); 
$data = curl_exec($ch); 
if (curl_errno($ch)){ 
    echo 'Retreive Base Page Error: ' . curl_error($ch); 
} 
else { 
    $info = rawurldecode(var_export(curl_getinfo($ch),true)); 
    echo $data; 

} 
?>