Java使用Byteoutputstream写入大文件

Question

我正在尝试使用ByteArrayOutputStream写入大小在1kb到10GB之间的文件，但会引发下面的异常。我正在使用jdk 6.请建议任何更好的高性能Api。我正在使用相同的网络盒来读取和写入。

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit 
     at java.util.Arrays.copyOf(Unknown Source) 
     at java.io.ByteArrayOutputStream.grow(Unknown Source) 
     at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source) 
     at java.io.ByteArrayOutputStream.write(Unknown Source) 
     at java.io.OutputStream.write(Unknown Source) 
     at 

代码：

import java.io.BufferedOutputStream; 
import java.io.ByteArrayOutputStream; 
import java.io.File; 
import java.io.FileInputStream; 
import java.io.FileOutputStream; 


public class PrepareFile { 

    /** 
    * @param args 
    * @throws Exception 
    */ 
    public static void main(String[] args) throws Exception { 
     // TODO Auto-generated method stub 

     new PrepareFile().constructFile("f:\\hello","f:\\output",10000000); 

    } 

    //Writes a large file of 10 GB using input file data of small size by duplicating 
    public void constructFile(String fileName, String outPath, int multiplier) throws Exception { 
     BufferedOutputStream fos = null; 
     FileInputStream fis = null; 

     final File inputFile = new File(fileName); 
     String path = inputFile.getParent(); 
     if (outPath != null && !outPath.isEmpty()) { 
      path = outPath; 
     } 

     fis = new FileInputStream(fileName); 

     try { 



      // read the transactions in the input file. 
      byte[] txnData = new byte[(int) inputFile.length()]; 
      fis.read(txnData); 

      final File outFile = new File(path, "Myfile"); 
      fos = new BufferedOutputStream(new FileOutputStream(outFile)); 
      final ByteArrayOutputStream baos = new ByteArrayOutputStream(); 
      final ByteArrayOutputStream baos1 = new ByteArrayOutputStream(); 

      //multiplier if input file size is 1 KB and output file is 10 GB, then multiplier value is (1024*1024) 

      for (long i = 1; i <= multiplier; i++) { 

       if(i >=40000 && i % 40000==0){ 
        System.out.println("i value now: "+i); 
        baos.writeTo(fos); 
        baos.reset(); 
        //baos.write(txnData); 
       } 

       // write transactions 
       baos.write(txnData); 
       baos1.write(txnData); //Exception is coming at this line 
      } 

      int Padding = myCustomMethod(baos1.toByteArray()); 

      // write all out data to the output stream 
      baos.writeTo(fos); 

      baos.flush(); 
      baos1.flush(); 
     } catch(Exception e){ 
      e.printStackTrace(); 
     }finally { 
      fos.close(); 
      fis.close(); 
     } 

    } 

    public int myCustomMethod(byte[] b){ 

     //Need complete bytes to prepare the file trailer 
     return 0; 
    } 


} 

Answer 1

你不能有一个ByteArrayOutputStream 2 GB或更大的缓冲区大小为32位有符号。

如果你想要的性能，我会逐步处理文件，并避免如此大的内存拷贝，因为它们非常昂贵。

顺便说一句我有一个库Chronicle Bytes它支持大于2 GB的缓冲区，并且可以使用本地内存并映射到文件以避免使用堆并且可能比主内存大。

但是，如果逐步处理数据，则不需要如此大的缓冲区。

我也建议你使用Java 8，因为它的性能比Java 6中（这是发布在十年前）

---

编辑基于您的代码更好的64位操作，没有必要使用ByteArrayOutputStream并且您可以逐步准备文件。

//Writes a large file of 10 GB using input file data of small size by duplicating 
public void constructFile(String fileName, String outFileName, int multiplier) throws IOException { 
    byte[] bytes; 
    try (FileInputStream fis = new FileInputStream(fileName)) { 
     bytes = new byte[fis.available()]; 
     fis.read(bytes); 
    } 

    try (FileOutputStream fos = new FileOutputStream(outFileName)) { 
     for (int i = 0; i < multiplier; i++) { 
      fos.write(bytes); 
     } 
    } 

    // now process the file "outFileName" 
    // how depends on what you are trying to do. 
    // NOTE: It is entirely possible the file should be processed as it is written. 
}