用BufferedWriter在Java中获取353个网页的更快方法

Question

我检查了其他链接，但是我找不到解决方案。 我可以连接到353个链接，并在大约7分钟内抓取我需要的数据。我需要把时间缩短到不到1分钟。

我在下面列入了我的代码。

URL urlChartLink; 
URLConnection urlconn; 

try { 
    Class.forName(driver).newInstance(); 
    Connection mysqlconn = DriverManager.getConnection(url + dbName, userName, password); 
    Statement st1 = mysqlconn.createStatement(); 
    Connection mysqlconn2 = DriverManager.getConnection(url + dbName, userName, password); 
    ResultSet rs1 = st1.executeQuery(strSQL); 

    while (rs1.next()) 
    { 
     sElementID = rs1.getString(1); 
     sSymbol = rs1.getString(2); 
     sChartLink = rs1.getString(3); 

     urlChartLink = new URL(sChartLink); 
     urlconn = urlChartLink.openConnection(); 
     urlconn.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"); 

     sCurrentPrice = ""; 
     sPriceChange = ""; 

     try { 
      BufferedReader in = new BufferedReader(new InputStreamReader(urlconn.getInputStream(), "UTF-8"));     
      String currentLine; 

      int iLine = 0; 

      while ((currentLine = in.readLine()) != null) { 
       //Get data from page 
       iLine += 1; 

      } 
      in.close(); 
     } catch (IOException e) { 

    } 

    st1.close(); 

    mysqlconn.close(); 
    mysqlconn2.close(); 

} 

我试过了没有URLConnection，但我得到了403错误。

如果有人能给我一个更好的解决方案，这将是伟大的！

Eddi Rae

Answer 1

对线程使用ExecutorService。此代码不是100％语法错误免费，但它应该给你正确的想法。

Class.forName(driver).newInstance(); 
Connection mysqlconn = DriverManager.getConnection(url + dbName, userName, password); 
Statement st1 = mysqlconn.createStatement(); 
Connection mysqlconn2 = DriverManager.getConnection(url + dbName, userName, password); 
final ResultSet rs1 = st1.executeQuery(strSQL); 
ExecutorService service = ExecutorService.newFixedThreadPool(30); 

while (rs1.next()) 
{ 
    service.execute(new Runnable() { 
     public void run() { 
      String sElementID = rs1.getString(1); 
      String sSymbol = rs1.getString(2); 
      String sChartLink = rs1.getString(3); 

      URL urlChartLink = new URL(sChartLink); 
      URLConnection urlconn = urlChartLink.openConnection(); 
      urlconn.addRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"); 

      sCurrentPrice = ""; 
      sPriceChange = ""; 

      try { 
       BufferedReader in = new BufferedReader(new InputStreamReader(urlconn.getInputStream(), "UTF-8"));     
       String currentLine; 


       while ((currentLine = in.readLine()) != null) { 
        //Call syncronized method to perform operations that need to be thread safe 
        addLine() 

       } 
       in.close(); 
      } catch (IOException e) { 

      } 
     } 
    }); 
} 

executor.shutdown(); 
while (!executer.isShutdown() { 
    Thread.sleep(100); 
} 

st1.close(); 

mysqlconn.close(); 
mysqlconn2.close(); 

public void addLine() { 
    syncronized (OBJECT_LOCK) { 
     iLine++; 
    } 
}