ZLib streams

Hi!

I'm sorry to disturb you further, but I found something weird
while trying what we discussed:

   Eval [
      PackageLoader fileInPackage: #TCP.
      PackageLoader fileInPackage: #ZLib.
   ]
   Eval [
      |zr zw sock|
      sock := TCP.Socket remote: 'localhost' port: 12324.

      zw := RawDeflateStream on: sock.
      zr := RawInflateStream on: sock.

      zw nextPutAll: 'TESTABC'; syncFlush.

      [sock atEnd not] whileTrue: [
         | hunk |
         hunk := zr nextHunk.
         'hunk!' displayNl.
         hunk inspect.
      ].
   ]

I tested this with socat like this:

   socat TCP4-LISTEN:12324,reuseaddr PIPE

The problem is that this line blocks/hangs:

   zr := RawInflateStream on: sock.

It blocks because RawInflateStream>>#on: calls ZlibReadStream>>#fillBuffer
a few method calls down the stack. And fillBuffer actually does
call nextHunk to wait for data to arrive on the line.

If I rearrange the lines btw.:

      zw nextPutAll: 'TESTABC'; syncFlush.
      zr := RawInflateStream on: sock.

Then everything seems to be fine (because there is data for nextHunk).

Would it be possible to prevent the call to fillBuffer to not have
a blocking RawInflateStream creation and only have it block when _I_
call RawInflateStream>>#nextHunk?


Thanks!

Robin


_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

> Would it be possible to prevent the call to fillBuffer to not have
> a blocking RawInflateStream creation and only have it block when _I_
> call RawInflateStream>>#nextHunk?

Seems easy...


--- orig/packages/zlib/ZLibReadStream.st
+++ mod/packages/zlib/ZLibReadStream.st
@@ -145,9 +145,9 @@ position
  !ZlibReadStream methodsFor: 'private'!

  resetBuffer
+    ptr := 0.
      delta := 0.
-    endPtr := 0.
-    self fillBuffer!
+    endPtr := 0!

  initialize: aStream
      super initialize: aStream.


Paolo


_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

On Thu, Aug 23, 2007 at 09:55:30AM +0200, Paolo Bonzini wrote:
Ok, thats easy. But maybe a bit too easy :-)

That worked for my small example. But I've tried to implement it in my program
now and ran into some other problem:

  [socket atEnd not] whileTrue: [self handleData: inputStream nextHunk]

(Please note that I tried to use '[inputStream atEnd not]' first, but that
seemed to run into the same problem I'm going to describe below as a
call to ZlibReadStream>>#atEnd results in a call to ZlibReadStream>>#fillBuffer)

nextHunk does not seem to return properly. I guess the problem is in
 ZlibReadStream>>#fillBuffer:
       "Fill the output buffer, supplying data to zlib until it can actually
        produce something."
       | flush |
       delta := delta + endPtr.
       ptr := 0.
       [
           inBytes isNil ifTrue: [
               inBytes := self stream atEnd
                   ifTrue: [ #[] ]
                   ifFalse: [ self stream nextHunk ] ].

           flush := self stream atEnd ifTrue: [ 4 ] ifFalse: [ 0 ].
           endPtr := self processInput: flush size: inBytes size.
           endPtr = 0 ] whileTrue.

       "End of data, or zlib error encountered."
       endPtr = -1 ifTrue: [ self checkError ]! !

The following line completes when it received the syncFlushed data,
and an inspect revealed that there really was data.

           ifFalse: [ self stream nextHunk ] ].

However, I put in some displayNls and observed that the call:

           flush := self stream atEnd ifTrue: [ 4 ] ifFalse: [ 0 ].

Seemed to block. That wouldn't be a problem if there would be further
data waiting now, but it isn't (due to the fact that the first packet was a
'HELO' in my protocol and my software waits now for further input which
wont come).

You may also note that the call to ZlibReadStream>>#processInput:size: which
will result in a call to RawInflateStream>>#processInput:size: will receive the
wrong flag (0 Z_NO_FLUSH or 4 Z_FINISH, and not 2 Z_SYNC_FLUSH) when atEnd
finally returns.

I'm not familiar enough and currently don't have enough time to try to fix it
or propose a solution to this.


Robin


_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

> That worked for my small example. But I've tried to implement it in my program
> now and ran into some other problem:
>
>   [socket atEnd not] whileTrue: [self handleData: inputStream nextHunk]
>
> Seemed to block.

You cannot know if the stream is atEnd unless a) it closes, b) you have
one more byte.  So yes, #atEnd blocks (by design).

I would do something like

    [
      socket atEnd ifTrue: [ ^self ]
      self handleData: inputStream nextHunk
    ] whileFalse.

with #handleData: returning true when it finishes reading a packet.

> You may also note that the call to ZlibReadStream>>#processInput:size: which
> will result in a call to RawInflateStream>>#processInput:size: will receive the
> wrong flag (0 Z_NO_FLUSH or 4 Z_FINISH, and not 2 Z_SYNC_FLUSH) when atEnd
> finally returns.

As far as I could see from the zlib source code, Z_SYNC_FLUSH only
matters for deflating.  On inflation, it only looks for Z_NO_FLUSH,
Z_FINISH, Z_BLOCK.

Paolo


_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

On Thu, Aug 23, 2007 at 03:14:20PM +0200, Paolo Bonzini wrote:
Thanks okay. But what I mean was that nextHunk blocked even after it
received a SYNC_FLUSHed packet from the other end. See the rest of the
mail I wrote there.

It's also fine in that loop when atEnd blocks, if it comes back after data
arrived or the stream closed or whatever, but inputStream (being (a
RawInflateStream (which I maybe missed to say, sorry)) needs to give me the
packet it got (and it actually really got a _complete_ packet, which I found
out while debugging). The problem was that atEnd (on the socket) which is
called in fillBuffer blocks _after_ the compressed data was read from the socket.
Which resulted in nextHunk not to give me the finished packet it got.

In fact nextHunk returned the packet when I shut down one end of the TCP connection
(which resulted in the atEnd in fillBuffer to return).

> I would do something like
>
>    [
>      socket atEnd ifTrue: [ ^self ]
>      self handleData: inputStream nextHunk
>    ] whileFalse.
>
> with #handleData: returning true when it finishes reading a packet.

handleData accumulates the data it got and maybe finished parsing none,
one, two, ..., N protocol messages. And I don't see why I would like to
end my read-loop when I got packets from the socket.

Maybe my approach of accumulating the bytes from the network in a buffer
and reparse it when more bytes arrive to detect completed packets is not
practical in smalltalk.

My logic was: "read all packets from the socket until we get a disconnect":

    [inputStream "(or socket)" atEnd not]
        whileTrue: [self handleData: inputStream nextHunk]

I don't know about the zlib source code, but I read the manual:
   http://www.zlib.net/manual.html#inflate

   If the parameter flush is set to Z_SYNC_FLUSH, inflate flushes as much output
   as possible to the output buffer.
And:
   However if all decompression is to be performed in a single step (a single
   call of inflate), the parameter flush should be set to Z_FINISH. In this case
   all pending input is processed and all pending output is flushed ;

The manual might be wrong, I don't know.


Robin


_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

> Thanks okay. But what I mean was that nextHunk blocked even after it
> received a SYNC_FLUSHed packet from the other end. See the rest of the
> mail I wrote there.

Ah, I see now.  Could you try this further patch?

Paolo

--- orig/packages/zlib/ZLibReadStream.st
+++ mod/packages/zlib/ZLibReadStream.st
@@ -161,12 +161,10 @@ fillBuffer
     delta := delta + endPtr.
     ptr := 0.
     [
-        inBytes isNil ifTrue: [
-    inBytes := self stream atEnd
-        ifTrue: [ #[] ]
-        ifFalse: [ self stream nextHunk ] ].
+        (inBytes isNil and: [ self stream atEnd not ])
+    ifTrue: [ inBytes := self stream nextHunk ].
 
- flush := self stream atEnd ifTrue: [ 4 ] ifFalse: [ 0 ].
+ flush := inBytes isNil ifTrue: [ 4 ] ifFalse: [ 0 ].
         endPtr := self processInput: flush size: inBytes size.
  endPtr = 0 ] whileTrue.
 

_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

On Thu, Aug 23, 2007 at 04:45:23PM +0200, Paolo Bonzini wrote:
>
> >Thanks okay. But what I mean was that nextHunk blocked even after it
> >received a SYNC_FLUSHed packet from the other end. See the rest of the
> >mail I wrote there.
>
> Ah, I see now.  Could you try this further patch?

With pleasure!

Yes, it seems to work just fine now (together with the patch for
resetBuffer). I'm however still a bit concerned about the value
for 'flush', but if you think 4 and 0 are fine there I'll trust
you :-)

Thanks!


Robin


_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk

I think so, yes.  You might have problems with lookahead there, too, but
apart from that it would be okay.

Paolo


_______________________________________________
help-smalltalk mailing list
[hidden email]
http://lists.gnu.org/mailman/listinfo/help-smalltalk