更快创建CSV文件

Question

我通过获取数据表然后遍历该数据表并写入CSV文件的每一行来创建一个CSV文件。我的数据源通常有大约65,000行。这个过程需要几分钟才能从浏览器下载。问题是本地和开发它不会花太长时间，但在客户端，他们通常会超时。

有没有更快的方式来生成这个文件？

Function GenerateCSVFile() As String 
    Dim stuPro As New studentProvider.StudentProvider 
    Dim emailCenter As New EmailCenter 
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "") 
    Dim strResult As String = "" 

    Dim dtStudent As Data.DataTable 
    Dim paymentYear As String = "" 
    dtStudent = stuPro.generateDataFile() 

    If dtStudent.Rows.Count > 0 Then 

     Using sw As New System.IO.StreamWriter(Server.MapPath("Temp/" + strFileName + ".csv")) 
      Try 
       Dim lineValue As String = "" 

       lineValue += "Academic Year, StudentID, SSN, First, Middle, Last" 

       sw.WriteLine(lineValue) 

       For i As Integer = 0 To dtStudent.Rows.Count - 1 

        lineValue = dtStudent.Rows(i)("fy").ToString 
        lineValue += "," & dtStudent.Rows(i)("uniq_stu_id").ToString 
        lineValue += "," & dtStudent.Rows(i)("ssn").ToString 
        lineValue += "," & dtStudent.Rows(i)("fname").ToString 
        lineValue += "," & dtStudent.Rows(i)("mname").ToString 
        lineValue += "," & dtStudent.Rows(i)("lname").ToString 
        sw.WriteLine(lineValue) 

       Next 
      Catch ex As Exception 
       strResult += ex.ToString 
      Finally 
       sw.Close() 
      End Try 

     End Using 

     Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv" 

     If String.IsNullOrEmpty(strResult) Then 

      Dim fs As System.IO.FileStream = Nothing 

      fs = System.IO.File.Open(Server.MapPath("Temp/" + strFileName + ".csv"), System.IO.FileMode.Open) 
      Dim btFile(fs.Length) As Byte 
      fs.Read(btFile, 0, fs.Length) 
      fs.Close() 

      With Response 
       .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName) 
       .ContentType = "application/octet-stream" 
       .BinaryWrite(btFile) 
       .End() 
      End With 
     End If 
    Else 
     strResult = "No records found for specified academic year" 
    End If 

    Return strResult 
End Function 

更新的代码

Function GenerateCSVFile() As String 
    Dim startDate As Date = Date.Now 
    Dim enddate As Date = Nothing 
    Dim stuPro As New studentProvider.StudentProvider 
    Dim emailCenter As New EmailCenter 
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "") 
    Dim strResult As String = "" 

    Dim dtStudent As Data.DataTable 
    Dim paymentYear As String = "" 
    dtStudent = stuPro.generateDataFile(Session("VendorID"), txtAcademicYear.Text.Trim) 

    If dtStudent.Rows.Count > 0 Then 

     With Response 

      Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv" 
      .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName) 
      .ContentType = "application/octet-stream" 

      Dim lineValue As StringBuilder = New StringBuilder 

      lineValue.Append("Academic Year, StudentID, SSN, First, Middle, Last") 

      .Write(lineValue.ToString) 

      For i As Integer = 0 To dtStudent.Rows.Count - 1 

       lineValue = New StringBuilder 
       lineValue.Append(dtStudent.Rows(i)("fy").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("uniq_stu_id").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("ssn").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("fname").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("mname").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("lname").ToString) 

       .Write(lineValue.ToString) 

      Next 
      enddate = Date.Now 

      MsgBox(DateDiff(DateInterval.Second, startDate, enddate)) 

      .End() 
     End With 
    Else 
     strResult = "No records found for specified academic year" 
    End If 
    Return strResult 
End Function 

Answer 1

有一些选项，以加快这：

• 不要的StreamWriter写入文件来响应，写入页面直接回应。
• 寻找一种使用数据读取器而不是数据表来循环访问数据的解决方案，使用这些数据可能会更快。它也会降低内存使用量（不会在内存中加载整个表）。
• 如果连接很多字符串使用StringBuilder，或者在这种情况下使用String.Join来轻松创建整行。

的string.join例如：

For Each row in dtStudent.Rows 
    Dim line as new List(of String) 
    line.Add(row("fy").ToString) 
    line.Add(row("uniq_stu_id").ToString) 
    line.Add(-etc-) 

    Response.Write(String.Join(",", line.ToArray) & vbcrlf) 
Next 

Answer 2

您正在编写到一个临时文件，读取该文件中，他们编写文件的内容到响应。跳过临时文件并一次直接写入响应一行。这将阻止浏览器认为你的服务器超时，让事情更快，并减少你的应用程序消耗的内存量。

之后，请查看如何在此Web请求上启用缓存，以便ASP.NET在短时间内如果有多个用户要求时不必重新创建CSV。

Answer 3

你应该使用StringBuilder而不是串联。

Answer 4

除了@Robert利维的建议，您如何使用字符串变量要小心。你会更好地使用这些线上使用stringbuilder服务：

 dim sbTemp as new StringBuilder() 
     For i As Integer = 0 To dtStudent.Rows.Count - 1 

      sbTemp.Append(dtStudent.Rows(i)("fy").ToString) 
      sbTemp.Append(",") 
      sbTemp.Append(dtStudent.Rows(i)("uniq_stu_id").ToString) 

      'etc 
      sw.WriteLine(lineValue) 

     Next 

Answer 5

有一件事，你可以看看是生产者/消费者的设计模式。这可以让你做的是有一个（或多个）线程提供一个包含需要写入csv文件的数据的队列和另一个执行实际写入的线程（或多个）。