Hadoop Zlib和JDK的比较性能Gzip

我正在做一些单线程压缩编解码器的基准测试，我看到Zlib的性能似乎远高于您对单线程的预期。我已经使用了org.apache.hadoop.io.compress.zlib.ZlibCompressor用于Zlib压缩器实现，并且java.util.zip.Deflate用于与Gzip实现进行比较。Hadoop Zlib和JDK的比较性能Gzip

在Hadoop中以某种方式提供的ZLib压缩器（包装器）是多线程的，可能通过JNI接口？

的Zlib：

import org.apache.hadoop.io.compress.zlib.*; 
protected final zlibCompressor = new ZlibCompressor(ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION, ZlibCompressor.CompressionStrategy.DEFAULT_STRATEGY, ZlibCompressor.CompressionHeader.DEFAULT_HEADER, DEFAULT_BUFFER_SIZE); 
protected final zlibDecompressor = new ZlibDecompressor(ZlibDecompressor.CompressionHeader.DEFAULT_HEADER, DEFAULT_BUFFER_SIZE); 

//compress 
zlibCompressor.setInput(uncompressed, 0, uncompressed.length); 
zlibCompressor.finish(); 
int n = zlibCompressor.compress(compressBuffer, 0, compressBuffer.length); 

//decompress 
zlibCompressor.reset(); 
zlibDecompressor.setInput(compressed, 0, compressed.length); 
int n = zlibDecompressor.decompress(uncompressBuffer, 0, uncompressBuffer.length);

Gzip已：

import java.util.zip.*; 
protected final deflater = new Deflater(COMPRESSION_LEVEL, NO_WRAP); 
protected final inflater = new Inflater(NO_WRAP); 

//compress 
int n = compressBlockUsingStream(uncompressed, compressBuffer); 

//decompress 
inflater.reset(); 
int n = uncompressBlockUsingStream(new InflaterInputStream(new ByteArrayInputStream(compressed), _inflater), uncompressBuffer);

gzip的辅助功能：甲

protected int compressBlockUsingStream(byte[] uncompressed, byte[] compressBuffer) throws IOException 
{ 
     ByteArrayOutputStream out = new ByteArrayOutputStream(compressBuffer); 
     compressToStream(uncompressed, out); 
     return out.length(); 
}  

protected int uncompressBlockUsingStream(InputStream in, byte[] uncompressBuffer) throws IOException 
{ 
      ByteArrayOutputStream out = new ByteArrayOutputStream(uncompressBuffer); 
      byte[] buffer = new byte[4096]; 
      int count; 
      while ((count = in.read(buffer)) >= 0) { 
       out.write(buffer, 0, count); 
      } 
      in.close(); 
      out.close(); 
      return out.length(); 
}

吞吐量：

的Zlib /块 - 143.902 MBps的

Gzip已/ JDK /流 - 22.573 Mbps的

任何人有一个想法，为什么zlib的是如此之快（本地使用所有内核）？该代码预计将运行单线程。任何人都可以复制类似的结果？

来源

2016-11-23 nikk

java.util.zip使用zlib。

你确定你在两个版本中使用相同的压缩级别吗？ COMPRESSION_LEVEL是否等于ZlibCompressor.CompressionLevel.DEFAULT_COMPRESSION？

来源

2016-11-23 17:37:39

OK，你知道什么样的价值“，‘DEFAULT_COMPRESSION’是[0 - 9]。喜欢明确设置它 – nikk

不能打印 –

好吧，如果你看一看Hadoop的zlibcompressor GitHub上的代码？，你会看到它被设置为-15的整数值，-15是什么意思呢？据我所知，压缩级别从0到9范围内。 – nikk

Hadoop Zlib和JDK的比较性能Gzip

回答

相关问题