用cURL登录到网页与PHP

Question

我买了一本书在网上抓取php。其中作者登录到https://www.packtpub.com/。这本书已经过时了，所以我无法真正测试出想法，因为网页自发布以来已经发生了变化。这是我正在使用的修改后的代码，但登录失败，我从“帐户选项”字符串中得出的结论不在$results变量中。我应该改变什么？我相信错误来自错误地指定目的地。

<?php 
// Function to submit form using cURL POST method 
function curlPost($postUrl, $postFields, $successString) { 
    $useragent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; 
     en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3'; // Setting useragent of a popular browser 
    $cookie = 'cookie.txt'; // Setting a cookie file to storecookie 
    $ch = curl_init(); // Initialising cURL session 
    // Setting cURL options 
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); // PreventcURL from verifying SSL certificate 
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0); 
    curl_setopt($ch, CURLOPT_FAILONERROR, TRUE); // Script shouldfail silently on error 
    curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE); // Use cookies 
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE); // FollowLocation: headers 
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE); // Returningtransfer as a string 
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie); // Settingcookiefile 
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie); // Settingcookiejar 
    curl_setopt($ch, CURLOPT_USERAGENT, $useragent); // Settinguseragent 
    curl_setopt($ch, CURLOPT_URL, $postUrl); // Setting URL to POSTto 
    curl_setopt($ch, CURLOPT_POST, TRUE); // Setting method as POST 
    curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($postFields)); // Setting POST fields as array 
      $results = curl_exec($ch); // Executing cURL session 
      $httpcode = curl_getinfo($ch,CURLINFO_HTTP_CODE); 
       echo "$httpcode"; 
      curl_close($ch); // Closing cURL session 
      // Checking if login was successful by checking existence of string 
      if (strpos($results, $successString)) { 
       echo "I'm in."; 
       return $results; 
      } else { 
       echo "Nope, sth went wrong."; 
       return FALSE; 
      } 
} 

$userEmail = 'youremail@email.com'; // Setting your email address for site login 
$userPass = 'yourpass'; // Setting your password for sitelogin 
$postUrl = 'https://www.packtpub.com'; // Setting URL toPOST to 
// Setting form input fields as 'name' => 'value' 
$postFields = array(
     'email' => $userEmail, 
     'password' => $userPass, 
     'destination' => 'https://www.packtpub.com', 
     'form_id' => 'packt-user-login-form' 
); 
$successString = 'Account Options'; 
$loggedIn = curlPost($postUrl, $postFields, $successString); //Executing curlPost login and storing results page in $loggedIn 

编辑：POST请求：

我取代了线

'destination' => 'https://www.packtpub.com' 
with  

'op' => 'Login' 

，加入

'form_build_id' => '' 

和编辑

$postUrl = 'https://www.packtpub.com/register'; 

因为这是我在选择复制为cURL并在编辑器中粘贴时所得到的URL。

我仍然在“没有，出错了信息”。我认为这是因为$successString首先不会被存储在curl中。应该设置的form-b​​uild-id是什么？它每次登录时都在变化。

Answer 1

你正在使用的书是旧的，Packt Publishing已经改变了他们的网站。它现在包含一个CSRF令牌，如果不通过这个，您将永远无法登录。

我开发了一个工作解决方案。它使用pQuery来解析HTML。您可以使用Composer安装它，或者下载该软件包并将其包含到您的应用程序中。如果这样做，请删除require __DIR__ . '/vendor/autoload.php';，并将其替换为系统上pquery软件包的位置。

要通过命令行进行测试，只需运行：php packt_example.php。

您还会注意到许多头文件甚至都不需要，比如useragent。我已经离开了这些。

<?php 

require __DIR__ . '/vendor/autoload.php'; 

$email = 'myemail@gmail.com'; 
$password = 'mypassword'; 

# Initialize a cURL session. 
$ch = curl_init('https://www.packtpub.com/register'); 

# Set the cURL options. 
$options = [ 
    CURLOPT_COOKIEFILE  => 'cookies.txt', 
    CURLOPT_COOKIEJAR  => 'cookies.txt', 
    CURLOPT_RETURNTRANSFER => 1 
]; 

# Set the options 
curl_setopt_array($ch, $options); 

# Execute 
$html = curl_exec($ch); 

# Grab the CSRF token from the HTML source 
$dom = pQuery::parseStr($html); 
$csrfToken = $dom->query('[name="form_build_id"]')->val(); 

# Now we have the form_build_id (aka the CSRF token) we can 
# proceed with making the POST request to login. First, 
# lets create an array of post data to send with the POST 
# request. 
$postData = [ 
    'email'   => $email, 
    'password'  => $password, 
    'op'   => 'Login', 
    'form_build_id' => $csrfToken, 
    'form_id'  => 'packt_user_login_form' 
]; 


# Convert the post data array to URL encoded string 
$postDataStr = http_build_query($postData); 

# Append some fields to the CURL options array to make a POST request. 
$options[CURLOPT_POST] = 1; 
$options[CURLOPT_POSTFIELDS] = $postDataStr; 
$options[CURLOPT_HEADER] = 1; 

curl_setopt_array($ch, $options); 

# Execute 
$response = curl_exec($ch); 

# Extract the headers from the response 
$headerSize = curl_getinfo($ch, CURLINFO_HEADER_SIZE); 
$headers = substr($response, 0, $headerSize); 

# Close cURL handle 
curl_close($ch); 

# If login is successful, the headers will contain a location header 
# to the url http://www.packtpub.com/index 
if(!strpos($headers, 'packtpub.com/index')) 
{ 
    print 'Login Failed'; 
    exit; 
} 

print 'Logged In'; 

Answer 2

我在发布这个答案，因为我认为它可能会在将来遇到这样的问题时帮助你。我在写网络刮板时会做很多事情。

1. 打开Firefox。按CTRL + SHIFT + Q
2. 新闻网络标签
3. 转到网站。您将注意到正在监视HTTP请求
4. 成功登录，同时监视HTTP请求
5. 登录后，右键单击用于登录的HTTP请求，然后复制为CURL。

现在您有CURL请求。使用PHP的cURL复制HTTP请求。并再次测试。

对于网页抓取，您应该非常熟悉监视HTTP标头。您可以使用：

• 网络监控器（Chrome，火狐）

• 提琴手

• Wiresharp

• MITMProxy

• 查尔斯

等...