Strange socket behavior

Hi all,

    we have some strange results using dolphin sockets.

    The socket is returning true to #hasInput evaluation and also is
returning true to #isOpen evaluation, the SocketReadStram atEnd evaluation
was returning false, but when we ask for #readPage this stream we receive a
Remote Socket Closed exception.

    Anybody has an idea to make it to work right?

    thanks,

Seb

Sebastián,

>     we have some strange results using dolphin sockets.
>
>     The socket is returning true to #hasInput evaluation and also is
> returning true to #isOpen evaluation, the SocketReadStram atEnd evaluation
> was returning false, but when we ask for #readPage this stream we receive a
> Remote Socket Closed exception.

Could it simply be that that you have a condition in which there is data
in the buffer, but #readPage is "too much to ask"?  Note that #atEnd
might have already sent #readPage.  I'm not saying it well, but I wonder
whether there could be data in buffer but the socket itself has indeed
closed.

Does something like

     aSocket readStream next:5

work when #readPage would signal the error?  If so, it would be reading
data from the buffer vs. reading from the socket.

What kind of protocol are you using?  I use mostly binary protocols
(sending a message size followed by what amounts to a byte array of that
size), but have also used cr/lf etc. terminated protocols at various times.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

> Could it simply be that that you have a condition in which there is data
> in the buffer, but #readPage is "too much to ask"?  Note that #atEnd might
> have already sent #readPage.  I'm not saying it well, but I wonder whether
> there could be data in buffer but the socket itself has indeed closed.

I've evaluated "socket receiveByteArrayPartial: 1" with the same results.

> work when #readPage would signal the error?  If so, it would be reading
> data from the buffer vs. reading from the socket.

The buffer I think is the SocketReadStream that work on that socket, and in
this case, as I said it is answering false to atEnd message. It even say
false, having that at that point the stream's position is equal to the
readLimit value. Don't you think that #atEnd message should answer true in
this case?

> What kind of protocol are you using?  I use mostly binary protocols
> (sending a message size followed by what amounts to a byte array of that
> size), but have also used cr/lf etc. terminated protocols at various
> times.

Not binary protocol in this case. I'm making an FTPClient all with ST (only
the socket uses the winInet API). The Sockets are the ordinary dolphin
Socket class (a TCP socket).

    regards,

Seb

Sebastián Sastre wrote:

> The buffer I think is the SocketReadStream that work on that socket, and
> in this case, as I said it is answering false to atEnd message. It even
> say false, having that at that point the stream's position is equal to the
> readLimit value. Don't you think that #atEnd message should answer true in
> this case?

There may be some deeper problem, but this sounds like normal Dolhpin + TCP/IP
behaviour to me.

The way that TCP works there can be no general test for connection-closed that
you can issue, and then be completely confident that there will be data to read
later.  I.e. the standard Smalltalk streams reading loop

    [aStream atEnd] whileFalse: [nextByte := aStream next].

simply cannot work.  It's form is incompatible with TCP, since the stream may
be remotely closed between the #adEnd returning true and the call to #next.
It's probably possible to make #atEnd smarter, but it can never be made
reliable; arguably it should be #shouldNotImplement.

For this reason, it is important that SocketReadStream overrides #do: (and
#upToEnd: and so on) so that they work correctly with TCP.  Unfortunately, and
despite repeated requests (from me anyway) OA have never finished that aspect
of the stream implementation.

So, in the end, you cannot do TCP stream handling without explicitly catching
the connection-closed error (not even if your code is supposed to handle
general streams).  The above loop has to be recoded as something like:

    [[aStream atEnd] whileFalse: [nextByte := aStream next]]
        on:SocketError
        do: [:err | "finished now"]

HTH

    -- chris

Hi OA team,

SocketReadStream>>readPage
    "Read ahead, a whole page, into the buffer."

    collection := (socket receiveByteArrayPartial: self bufferSize)
asString.

    readLimit := collection size.

    position := 0.

    #todo "Detect when the socket hits its end"

_________________________________

    I saw this method in SocketReadStream and I'm specially concern about
the #todo comment. Are there any improvement on this regard?

    I also saw that the socket usually answer true to #isOpen message, but
next action prooves that is not open. That's normal? I think that *at least*
that is leading to SocketReadStream>>atEnd to answer incorrect status.

    There is a way to make Socket #isOpen more trustable?

    regards,

--
Sebastián Sastre
Seaswork
Special Software Solutions
www.seaswork.com.ar


"Sebastián Sastre" <[hidden email]> escribió en el mensaje
news:[hidden email]...

Sebastián,

>>Could it simply be that that you have a condition in which there is data
>>in the buffer, but #readPage is "too much to ask"?  Note that #atEnd might
>>have already sent #readPage.  I'm not saying it well, but I wonder whether
>>there could be data in buffer but the socket itself has indeed closed.
>
>
> I've evaluated "socket receiveByteArrayPartial: 1" with the same results.

In the condition I'm describing, that would fail - you are ignoring the
buffer contents and asking for read from the socket.  If you are going
to do that, the stream is pointless, and in fact harmful.

Use things like #next, #next:, and #nextAvailable:, and it should work
provided the connection remains in tact and the other side holds up its
end of the contract.



> The buffer I think is the SocketReadStream that work on that socket, and in
> this case, as I said it is answering false to atEnd message. It even say
> false, having that at that point the stream's position is equal to the
> readLimit value. Don't you think that #atEnd message should answer true in
> this case?

As Chris points out, #atEnd is always going to be at the mercy of "the
network".


> Not binary protocol in this case.

Are you certain of that?  FTP certainly has a binary mode, though I do
not recall whether it uses 7-bit clean transmission (which would require
encoding, and make your life even harder).


 > I'm making an FTPClient all with ST (only
> the socket uses the winInet API).

IMHO, that is a contradiction.  If you use WinINet, you are almost
certainly getting encoding, decoding, request/reply, state, etc.
decisions that are made by Microsoft programmers.  There is nothing
wrong with that if it what you want and you are willing to take the bad
with the good; but, it is not an "all Smalltalk" result.


 > The Sockets are the ordinary dolphin
> Socket class (a TCP socket).

The following applies if you control the sockets and the protocol:

While it is of course possible that you have hit a bug in Dolphin's
sockets classes, it seems much more likely that you have missed a
termination condition or otherwise deviated from the protocol.  I know
(all too well<g>) how frustrating it can be to isolate such problems,
but I'm betting that the problem is in your FTP client.

Squeak includes an FTP client, and it might give you some help with the
details of the protocol.

Have a good one,

Bill


--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Dear Chris, all,

    we have considered before the use of error handling, now you convinced
us :)

    So I've made a patch for socket and an extension of socket streams
includig a TextSocketReadStream to work better with ascii.

    Not sure if is the definitive solution to this but I've made a solution
for waht I need.

    If anyone is interested I can send it by email.

    regards,


--
Sebastián Sastre
Seaswork
Special Software Solutions
www.seaswork.com.ar



"Chris Uppal" <[hidden email]> escribió en el
mensaje news:[hidden email]...

Sebastián Sastre wrote:

Not sure about #hasInput, I'll have to look at it,
but I suspect it is lying - er, being precise in that
it should give same result as (#atEnd not).

Q: When is a stream atEnd?
A: When there is not now, and never will be,
    any more data to read.

Q: How do we know if there is any more data
    to read from a socket?
A: Try reading.  If it works, there was data.
    If not, the socket might be closed.

Q: How do we know if a socket is closed?
A: We get a signal from the far end.

Q: How is that signal sent?
A: In the exact same way that more data is sent.
    Well, almost.  Actually, it is a 'FIN' bit
    in the last packet sent from the other end.
    The hitch is, there are two cases:
    a) The "all in one packet" case -
       Here is the last byte of data I will send,
       and I am marking this packet with a 'FIN'.
    b) The "takes two to tango" case -
       Packet One: Here is all the data I have so far.
       (time passes...)
       Packet two: Oh, I just now determined
                   that I am done sending, so
                   so I am marking this (otherwise empty)
                   packet with a 'FIN'.

Q: So we ALWAYS have to start reading from a socket
    to discover the end of data signal?
A: Right!

Bottom line - we do not know there is no more data
until we try to read some.

One could get around this in the case of streams,
but not for sockets in general - and you would need
two different implementations of #hasInput,
one for direct socket access (#atEnd -> #hasInput not)
and one for stream access (#hasInput -> #atEnd not).

The first is the current implementation (non-buffered).
The second requires #hasInput (and by implication, #atEnd)
to read one character ahead, into the stream buffer,
(protected with on: Error do: [:ex| self beAtEnd])
before answering.

Also, for streams, #do: should be implemented as
     self contents do:
so that stream position is not used and/or changed.

If one actually wants to #do: the entire contents
of a socket based stream, one should first obtain it -
   contents := stream upToEnd.
   contents do: ...

Relatedly, file based streams should implement #contents
as
   | was all |
    was := self position.
    all := self reset; upToEnd.
    self position: was.
    ^all

--

The only runintended consequence of doing
all this is that algorithms which attempt
to do useful work between checking #atEnd
and (possibly) blocking on a read would
require an alternate approach.  Still...


Regards,

-cstb

> > I'm making an FTPClient all with ST (only
>> the socket uses the winInet API).
>
> IMHO, that is a contradiction.  If you use WinINet, you are almost
> certainly getting encoding, decoding, request/reply, state, etc. decisions
> that are made by Microsoft programmers.

Yes, but that is only if you make use of WinInet API *dedicated to* FTP.
For this client, we are using raw Dolphin Sockets (that also make use of
WinInet API). Beside for this raw sockets, all the rest of the ftp client is
in Dolphin hands,

regards,

--
Sebastián Sastre
Seaswork
Special Software Solutions
www.seaswork.com.ar