同时HTTP请求与卷曲

Question

我试图采取域的一个相当大的列表查询每个军衔使用compete.com API这里看到的 - >https://www.compete.com/developer/documentation

我写的剧本需要一个数据库我填充的域名，并发起cURL请求，争夺网站的排名。我很快意识到这是非常缓慢的，因为每次请求都是一次发送一次。我做了一些搜索，并遇到了这个post->http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/，它解释了如何用cURL在PHP中执行同时的HTTP请求。

不幸的是，脚本将采用25,000个域的数组并尝试一次处理它们。我发现一批1000个工作很好。

任何想法如何发送1,000个查询到conte.com，然后等待完成并发送下一个1000，直到数组为空？这里就是我，因此到目前为止，干活：

<?php 

//includes 
include('includes/mysql.php'); 
include('includes/config.php'); 

//get domains 
$result = mysql_query("SELECT * FROM $tableName"); 
while($row = mysql_fetch_array($result)) { 
    $competeRequests[] = "http://apps.compete.com/sites/" . $row['Domain'] . "/trended/rank/?apikey=xxx&start_date=201207&end_date=201208&jsonp="; 
} 

//first batch 
$curlRequest = multiRequest($competeRequests); 
$j = 0; 
foreach ($curlRequest as $json){ 
    $j++; 
    $json_output = json_decode($json, TRUE); 
    $rank = $json_output[data][trends][rank][0][value]; 

    if($rank) { 
     //Create mysql query 
     $query = "Update $tableName SET Rank = '$rank' WHERE ID = '$j'"; 

     //Execute the query 
     mysql_query($query); 
     echo $query . "<br/>"; 
    } 
} 


function multiRequest($data) { 
    // array of curl handles 
    $curly = array(); 
    // data to be returned 
    $result = array(); 

    // multi handle 
    $mh = curl_multi_init(); 

    // loop through $data and create curl handles 
    // then add them to the multi-handle 
    foreach ($data as $id => $d) { 

    $curly[$id] = curl_init(); 

    $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d; 
    curl_setopt($curly[$id], CURLOPT_URL,   $url); 
    curl_setopt($curly[$id], CURLOPT_HEADER,   0); 
    curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1); 

    // post? 
    if (is_array($d)) { 
     if (!empty($d['post'])) { 
     curl_setopt($curly[$id], CURLOPT_POST,  1); 
     curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']); 
     } 
    } 

    curl_multi_add_handle($mh, $curly[$id]); 
    } 

    // execute the handles 
    $running = null; 
    do { 
    curl_multi_exec($mh, $running); 
    } while($running > 0); 

    // get content and remove handles 
    foreach($curly as $id => $c) { 
    $result[$id] = curl_multi_getcontent($c); 
    curl_multi_remove_handle($mh, $c); 
    } 

    // all done 
    curl_multi_close($mh); 

    return $result; 

} 
?> 

Answer 1

而不是

//first batch 
$curlRequest = multiRequest($competeRequests); 

$j = 0; 
foreach ($curlRequest as $json){ 

你可以这样做：

$curlRequest = array(); 

foreach (array_chunk($competeRequests, 1000) as $requests) { 
    $results = multiRequest($requests); 

    $curlRequest = array_merge($curlRequest, $results); 
} 

$j = 0; 
foreach ($curlRequest as $json){ 
    $j++; 
    // ... 

这大阵分成1000块，将这1000个值传递给您的multiRequest函数，该函数使用cURL执行这些请求。