Smalltalk › Squeak › Squeak - Dev

[ANN] Sqnappy - a fast compressor wrapper

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

4 messages Options

Masashi UMEZAWA-2

[ANN] Sqnappy - a fast compressor wrapper

Hi all,

Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy
compressor library.
https://github.com/mumez/sqnappy

About snappy:
https://code.google.com/p/snappy/

Sqnappy is easy to use. You can just send #compress:, #uncompress to SnappyCore:
compressed := SnappyCore compress: data.
uncompressed := SnappyCore uncompress: compressed.

With a simple test, Sqnappy was 10.8x faster than the existing
GZipWriteStream/GZipReadStream.

Additionally, Sqnappy implements snappy framing format, so that it can
treat big data with a small memory allocation.

About framing format:
https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt

Tested from a workspace, 1.3 GB pg_dump file was compressed in around
10 seconds and decompressed in 6.5 seconds . No annoying GCs. It was
comfortable.

Enjoy!
--
[:masashi | ^umezawa]

Chris Muller-3

Re: [ANN] Sqnappy - a fast compressor wrapper

Very impressive! Especially if not creating additional garbage during
compress or decompress (although I suppose the original input string
would be garbage right? It doesn't output the compressed into the
original String/ByteArray or does it?). I'm curious whether I could
use this to reduce the sizes of Magma DB's -- if the decompression is
fast enough then it shouldn't appreciably affect read performance.

Only downside is the need for the additional VM plug-in.

On Thu, Apr 25, 2013 at 9:49 PM, Masashi UMEZAWA
<[hidden email]> wrote:

> Hi all,
>
> Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy
> compressor library.
> https://github.com/mumez/sqnappy
>
> About snappy:
> https://code.google.com/p/snappy/
>
> Sqnappy is easy to use. You can just send #compress:, #uncompress to SnappyCore:
> compressed := SnappyCore compress: data.
> uncompressed := SnappyCore uncompress: compressed.
>
> With a simple test, Sqnappy was 10.8x faster than the existing
> GZipWriteStream/GZipReadStream.
>
> Additionally, Sqnappy implements snappy framing format, so that it can
> treat big data with a small memory allocation.
>
> About framing format:
> https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
>
> Tested from a workspace, 1.3 GB pg_dump file was compressed in around
> 10 seconds and decompressed in 6.5 seconds . No annoying GCs. It was
> comfortable.
>
> Enjoy!
> --
> [:masashi | ^umezawa]
>

Colin Putney-3

Re: [ANN] Sqnappy - a fast compressor wrapper

In reply to this post by Masashi UMEZAWA-2

On Thu, Apr 25, 2013 at 7:49 PM, Masashi UMEZAWA <[hidden email]> wrote:

Hi all,

Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy
compressor library.
https://github.com/mumez/sqnappy

Bravo! Thanks for sharing!

Colin

Masashi UMEZAWA-2

Re: [ANN] Sqnappy - a fast compressor wrapper

In reply to this post by Chris Muller-3

Hi,

Basically Compression/uncompression results are copied from the
original, so there are some overheads. But if you use SnappyFraming
(Stream API), copies are minimized until you retrieve the final result
from the stream. Compression/uncompression are done in chunks and the
same small buffer is reused while processing. So we can expect pretty
linear performance.

LevelDB already utilizes snappy library for reducing its database
size, so I think Magma with Sqnappy is a great idea.

Best regards,

2013/4/27 Chris Muller <[hidden email]>:

> Very impressive! Especially if not creating additional garbage during
> compress or decompress (although I suppose the original input string
> would be garbage right? It doesn't output the compressed into the
> original String/ByteArray or does it?). I'm curious whether I could
> use this to reduce the sizes of Magma DB's -- if the decompression is
> fast enough then it shouldn't appreciably affect read performance.
>
> Only downside is the need for the additional VM plug-in.
>
> On Thu, Apr 25, 2013 at 9:49 PM, Masashi UMEZAWA
> <[hidden email]> wrote:
>> Hi all,
>>
>> Recently I've developed Sqnappy - Squeak/Pharo binding of the snappy
>> compressor library.
>> https://github.com/mumez/sqnappy
>>
>> About snappy:
>> https://code.google.com/p/snappy/
>>
>> Sqnappy is easy to use. You can just send #compress:, #uncompress to SnappyCore:
>> compressed := SnappyCore compress: data.
>> uncompressed := SnappyCore uncompress: compressed.
>>
>> With a simple test, Sqnappy was 10.8x faster than the existing
>> GZipWriteStream/GZipReadStream.
>>
>> Additionally, Sqnappy implements snappy framing format, so that it can
>> treat big data with a small memory allocation.
>>
>> About framing format:
>> https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
>>
>> Tested from a workspace, 1.3 GB pg_dump file was compressed in around
>> 10 seconds and decompressed in 6.5 seconds . No annoying GCs. It was
>> comfortable.
>>
>> Enjoy!
>> --
>> [:masashi | ^umezawa]
>>
>

--
[:masashi | ^umezawa]