lesson learned trying external configuration of compression stream classes

<a href="https://www.instantiations.com/docs/92/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/compression2.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.instantiations.com%2Fdocs%2F92%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fcompression2.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFZFYiyh9F6wJzJ6pf3HRrJoV6mHQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.instantiations.com%2Fdocs%2F92%2Fwwhelp%2Fwwhimpl%2Fjs%2Fhtml%2Fwwhelp.htm%23href%3Dpr%2Fcompression2.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFZFYiyh9F6wJzJ6pf3HRrJoV6mHQ&#39;;return true;">https://www.instantiations.com/docs/92/wwhelp/wwhimpl/js/html/wwhelp.htm#href=pr/compression2.html

Because the best compression choice depends on circumstances, I was tempted to not hard-code a decision on which read & write stream classes to use.  But I found at least for LZ4, the lz4Compress and lz4Decompress methods are much faster than the equivalent general stream code shown on that page.  Just FYI.

Still liking the idea of being able to configure the class, I may add my own methods on the class side of the stream classes, for instance in LZ4WriteStream I would add a method like #compress: which would simply use lz4Compress

Although part of a performance study might involve changing the compression level via #compressionLevel: of the codec, where supported.

Thanks. Makes sense.

I was thinking Brotli was real slow, then I realized the default is maximum compression.

What would be cool is a way for a client to send a compressed string to a server, and the server to write the uncompressed string to a file, without the server needing the full uncompressed string in its memory.

Scratch my socket thought.  I now remember my thinking.

If a server was sent (say) a zstdCompressed string which was originally 1GB, and the server wants to write the uncompressed string to a file, the server essentially has to do:

myCfsWriteStream nextPutAll: compressedBytes zstdUncompressed

thereby getting that whole 1GB string in memory.

Hi Wayne,

To answer your first question, yes, you need an SstSocketStream from Server Smalltalk - TCP Communications as the source/sink to wrap an EsCompressionStream around.

As to your second point, I'm saying that it can be arranged such that 'compressedBytes' can be pulled off the socket incrementally and pushed to the file stream in parts.
It's certainly easier, I won't deny that to do what you showed.  But you can arrange things such that the server has a socket, wrapped by a socket stream.

So getting data on the server-side is a bunch of #next, #next:, #nextLine... actions of internally buffered data in the socket stream instead of a bunch of #recv calls on a primitive socket.

Once you know the the next number of bytes to get, which may be zstdCompressed...then wrap an ZstdReadStream around the socket stream and go into a
next loop, grabbing bits of uncompressed zstd data and pushing it into a file stream.
Here is a really primitive example to make the point...and this is possibly blocking on zstdStream next since it wraps a socket stream.

bytesExpected timesRepeat: [fileStream nextPut: zstdStream next]

- Seth

The trick here is to make sure you don't overread into the zstd stream since it is internally buffered.
If you don't want to mess around with

On Tuesday, December 10, 2019 at 6:56:48 AM UTC-5, Wayne Johnston wrote:

Scratch my socket thought.  I now remember my thinking.

If a server was sent (say) a zstdCompressed string which was originally 1GB, and the server wants to write the uncompressed string to a file, the server essentially has to do:

myCfsWriteStream nextPutAll: compressedBytes zstdUncompressed

thereby getting that whole 1GB string in memory.