java httpurlconnection中断html

Question

嘿，我试图从Twitter的个人资料页获取html，但httpurlconnection只返回一小段html。我的代码

for(int i = 0; i < urls.size(); i++) 
{ 
URL url = new URL(urls.get(i)); 
HttpURLConnection connection = (HttpURLConnection) url.openConnection(); 
connection.setRequestProperty("User-Agent","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6"); 
System.out.println(connection.getResponseCode()); 
String line; 
StringBuilder builder = new StringBuilder(); 
BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream())); 
while((line = reader.readLine()) != null) 
{ 
    builder.append(line); 
} 
String html = builder.toString(); 
} 

我总是得到200作为每次调用的响应代码。然而，大约1/3的时间整个html文档被返回，另一半只有前几百行。 html截断时返回的金额并不总是相同的。

任何想法？谢谢你的帮助！

附加信息：查看标题后，似乎我得到重复的内容长度标题。第一个是全长，另一个是短得多（可能代表我获得一些时间的长度）如何处理重复的标题？

Answer 1

这对我工作得很好，我加builder.append(line);后换行，使其在控制台更具可读性，但除此之外，它返回所有的HTML此页：

import java.io.BufferedReader; 
import java.io.IOException; 
import java.io.InputStreamReader; 
import java.net.HttpURLConnection; 
import java.net.URL; 
import java.util.ArrayList; 
import java.util.List; 

public class RetrieveHTML { 

    public static void main(String[] args) throws IOException { 
     List<String> urls = new ArrayList<String>(); 
     urls.add("http://stackoverflow.com/questions/3285077/java-httpurlconnection-cutting-off-html"); 

     for (int i = 0; i < urls.size(); i++) { 
      URL url = new URL(urls.get(i)); 
      HttpURLConnection connection = (HttpURLConnection) url.openConnection(); 
      connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6"); 
      System.out.println(connection.getResponseCode()); 
      String line; 
      StringBuilder builder = new StringBuilder(); 
      BufferedReader reader = new BufferedReader(new InputStreamReader(connection.getInputStream())); 
      while ((line = reader.readLine()) != null) { 
       builder.append(line); 
       builder.append("\n"); 
      } 
      String html = builder.toString(); 
      System.out.println("HTML " + html); 
     } 

    } 
} 

Answer 2

看看我的HTTP类

https://stackoverflow.com/questions/9349378/java-net-httpurlconnection-returning-your-browsers-cookie-functionality-has-be

基于此API。随意改变一些东西。