Next stupd question: how to get a String from a Socket?

So, I can connect to my news server, and read a byte array back (although only with readPage which
bothers me because I don't want to mess about with buffer handling).

Is there any convenient way of getting a stream connected to the socket which works like one over a
String? Ideally, I'd like to be able to able to use nextLine etc, or perhaps a FileStream

What I have is:

address := InternetAddress host: 'localhost'.
socket := Socket port: 119 address: address.
socket connect.
w:= socket writeStream.
w  nextPutAll: 'USER PAUL' asByteArray.
out:= socket readStream readPage;contents asString.

... but that's going to get pretty inconvenient. If I can't use nextLine etc, I 'll need pretty efficient single-
character I/O (which suggests a FileStream over the socket?). I'd still like exceptions to be raised when the
socket is closed by the other end, or times out.

This must be pretty common in Internet programming. What have other people done?

P.

Paul Hudson <[hidden email]> writes:

> This must be pretty common in Internet programming. What have other
> people done?

Note that you haven't read the first line response from the NNTP
server before you send your 'USER' command. You might want to do that
as I think NNTP servers send a confirmation that you've connected. You
also need to put a carriage return/line feed combo at the end of your
write and do a flush. Not necessarily the best example of style, but
I've used:

  writeLine: aLine to: aSocket
    |byteLine|
    byteLine := OrderedCollection new.
    aLine do: [ :c | byteLine add: c asInteger ].
    aSocket writeStream
      nextPutAll: byteLine;
      nextPut: 13;
      nextPut: 10.

To read a line I use:

  readLineFrom: aSocket
    | eol1 eol2 result |
    eol1 := 13.
    eol2 := 10.
    result := WriteStream with: String new.
    [ |ch readResult|
      readResult := self readUpTo: eol1 from: aSocket.
      result nextPutAll: readResult.
      ch := aSocket readStream next.
      ch = eol2 ifFalse: [
        result nextPut: eol1 asCharacter.
        result nextPut: ch asCharacter
      ].
      ch = eol2.
    ] whileFalse.
    ^result contents.

    readUpTo: aChar from: aSocket
      |result|
      result := WriteStream with: String new.
      [ |ch|
        ch := aSocket readStream next.
        ch = aChar ifFalse: [ result nextPut: ch asCharacter ].
        ch = aChar
      ] whileFalse.
      ^result contents.

As is probably obvious, I'm not a dolphin or smalltalk experienced
user though.

Chris.
--
http://www.double.co.nz/smalltalk

Paul,

> So, I can connect to my news server, and read a byte array back (although
only with readPage which
> bothers me because I don't want to mess about with buffer handling).
>
> Is there any convenient way of getting a stream connected to the socket
which works like one over a
> String? Ideally, I'd like to be able to able to use nextLine etc, or
perhaps a FileStream

#readStream will probably do what you want...


> What I have is:
>
> address := InternetAddress host: 'localhost'.
> socket := Socket port: 119 address: address.
> socket connect.
> w:= socket writeStream.
> w  nextPutAll: 'USER PAUL' asByteArray.
> out:= socket readStream readPage;contents asString.

...and you've obviously found it.


> ... but that's going to get pretty inconvenient. If I can't use nextLine
etc, I 'll need pretty efficient single-
> character I/O (which suggests a FileStream over the socket?).

Now I see your problem: you're in denial<g> about your need to enter the
world of multi-threaded programming.  Making a long story short, I've had
good results with "blocking" reads from socket streams, but, always on a
background thread.  That's important because otherwise, you'll block the
main thread which runs the message loop; then new data will arrive, but,
your app won't be able to process it.  Whether the background thread reads a
fixed number of bytes, a number of bytes specified by a DWORD at the
beginning of a message, or up to a terminating character depends on the
protocol; since you're asking about reading lines, it will likely be the
latter.  FWIW, I use BlockingCallMonitor to listen for connections, and
create home-grown threads for everything else.  I created ProcessViewer to
help with debugging; it's still available on my web site, but, Dolphin now
includes a tool that you might like better.  The one advantage of
ProcessViewer is that it shows text call stacks without having to open a
debugger, which is useful for finding deadlocks.

On Win98 anyway (and probably Win9x in general), you should probably avoid
trying to time out a connect call sooner than 30 seconds.  I had an
interesting bug for a while that would sometimes crash apps.  It appears to
have resulted from a race condition between my 30 second timeout and
Microsoft's 30 second timeout in connect(). Caveat emptor: I've never been
able to prove that, but, it seems to fit the facts.  My suggestion is to
either do the connect on "yet another thread" that you never terminate; just
let it go away when it's done, or set your timeouts to at least 35-40
seconds.  If you use a separate thread, the thread that forks it will have
to wait on a semaphore that the connect thread signals on a establishing a
connection.  You should be able to terminate the "forking" (not the forked)
thread any time you want.  You would most likely ignore the failure of the
connect, because the timeout (or user cancel) would already have indicated
that something went wrong.

The options for the Debugger now include a choice of whether or not to
terminate (including an option to prompt) a thread on close of a debugger.
The other thing that might make opening a debugger troublesome is whether or
not it interferes with waiting semaphores.  IIRC, closing a debugger on a
thread at least had the effect of signalling a semaphore (if any) on which
the thread was waiting.  That's ok for the Dolphin threads, because they'll
discover there is nothing to do and go back to sleep.  My threads are (part
by design and part by laziness) very particular about things happening in
the sequence they expect; they would consider an extra signal to be a
"fatal" error.  I worked out my debugging procedures a long time ago in
Dolphin years, and haven't had need to change them.  Anyway, that's the
Cliff's Notes (perhaps a somewhat dated copy) from the school of hard knocks
:)


> I'd still like exceptions to be raised when the
> socket is closed by the other end, or times out.

Wouldn't we all :)   I'm still waiting for somebody to put me straight on
the fine points of this, but, my feeble observations suggest that one can't
rely solely on socket errors.  TCP stream sockets do a great job of not
misrepresenting the validity of data they deliver (those are carefully
chosen words); they sometimes (maybe even most of the time) can tell when
the other side closed.  Likewise, they can also sometimes know when the
other side crashed.  However, if you you don't enforce some kind of timeout,
you can find yourself waiting on data that will never arrive.

Recently, I searched for the Microsoft KB article on keep alive packets.
It's still on their web site; I urge you to ignore it.  The impression I got
from reading stuff from DejaNews (sadly, those posts are probably "gone"
now) is that, when first proposed, the feature was so unpopular that it was
hobbled from the beginning.  The default settings take _hours_ (40 IIRC) to
time out a connection.  That's if you can guarantee that both sides are
configured correctly, and IF you can guarantee that both sides support keep
alive packets.

Re timeouts, I have a bleeding-edge goodie for timeouts using a progress
dialog; it's in my web site but separate from the 4.0 goodies.  I'd
appreciate comments, suggestions and bug reports.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Chris, Paul,

> Note that you haven't read the first line response from the NNTP
> server before you send your 'USER' command. You might want to do that
> as I think NNTP servers send a confirmation that you've connected. You
> also need to put a carriage return/line feed combo at the end of your
> write and do a flush.

Good point.  In fact, it's common for servers to be quite strict about
ignoring extra commands that haven't been acknowledged.  For example,
suppose commands are separated by |.  Then sending

    Hello|Do something|Do something else|

at one shot would probably result in Hello being processed, and the other
two commands ignored.  The server would typically send an ack for Hello and
ignore the others.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

"Bill Schwab" <[hidden email]> wrote in news:9trpvk$qf5$[hidden email]:

>> Note that you haven't read the first line response from the NNTP
>> server before you send your 'USER' command.[snip]
>
> Good point

Don't pay too much attention to the details of my example in this respect. It was all wrong, but that wasn't the
point of my post. For the record, I've got authentication working just fine now.

P.

"Bill Schwab" <[hidden email]> wrote in news:9trpi7$13c$[hidden email]:

> #readStream will probably do what you want...

Not really. It gives me a stream over a ByteArray (i.e. contains SmallIntegers) and I want one over a String
(containing characters). And I'd like it buffered (so I'd like a FileStream, but over a socket, not a real file).
Which of course is pretty trivial with C and the stdio library....

> Now I see your problem: you're in denial<g> about your need to enter the
> world of multi-threaded programming.

If you say so. I didn't have this problem when programming in C, though! I don't think I am in denial, either. I'm
just after a buffered socket returning characters. Multi-threading not really needed - the only underlying
functionality needed is already there  (Socket>>receivePartialByteArray)  I had hoped with Smalltalk I'd be
spared all this messing about with fixed-size buffers, but apparently not.

> Wouldn't we all :)   I'm still waiting for somebody to put me straight on
> the fine points of this, but, my feeble observations suggest that one can't
>rely solely on socket errors.  TCP stream sockets do a great job of not
>misrepresenting the validity of data they deliver (those are carefully
>chosen words); they sometimes (maybe even most of the time) can tell when
>the other side closed.  Likewise, they can also sometimes know when the
>other side crashed.  However, if you you don't enforce some kind of timeout,
>you can find yourself waiting on data that will never arrive.

My experience in C is a bit better. I don't have a problem with sockets generally - I've done enough
programming with them in past lives. My problem is with how they're integrated into Dolphin.

P. (currently resigned to re-implementing chunks of FileStream)

Paul,

> > #readStream will probably do what you want...
>
> Not really. It gives me a stream over a ByteArray (i.e. contains
SmallIntegers) and I want one over a String
> (containing characters). And I'd like it buffered (so I'd like a
FileStream, but over a socket, not a real file).
> Which of course is pretty trivial with C and the stdio library....

No real argument there: it be nice to be able to change the text/binary
nature of a socket stream.  When working with text, I typically convert the
charmed characters to integers and parse using them.  When a whole buffer
arrives (as a byte array), I send #asString to it and pass it along.  It's
not been a big problem, though there is room for improvement.


> > Now I see your problem: you're in denial<g> about your need to enter the
> > world of multi-threaded programming.
>
> If you say so. I didn't have this problem when programming in C, though! I
don't think I am in denial, either. I'm
> just after a buffered socket returning characters. Multi-threading not
really needed - the only underlying
> functionality needed is already there  (Socket>>receivePartialByteArray)
I had hoped with Smalltalk I'd be
> spared all this messing about with fixed-size buffers, but apparently not.

That's the big benefit of reading on another thread: you can simply block
for what you need, and the problems generally go away.  All you need to do
is synchronize the transfer of results to other threads; that might be as
simple as using a SharedQueue.


> My experience in C is a bit better. I don't have a problem with sockets
generally - I've done enough
> programming with them in past lives. My problem is with how they're
integrated into Dolphin.

I could be wrong, but, I think you might be transferring the complexity of a
C program over to Smalltalk.  In C, one would think nothing of having some
kind of polling loop and managing memory to go with it - it's the nature of
the language.  A background thread that blocks until it has what it needs
can result in much cleaner code and is well worth the extra trouble of
thread synchronization, espcially given how easy it is to fork a block and
create/share synchronization objects in Smalltalk.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Paul,

> Not really. It gives me a stream over a ByteArray (i.e. contains
SmallIntegers) and I want one over a String
> (containing characters). And I'd like it buffered (so I'd like a
FileStream, but over a socket, not a real file).
> Which of course is pretty trivial with C and the stdio library....
...
>  I'm just after a buffered socket returning characters.

I'm sure I've missed something in this conversation but SocketReadStream
_is_ buffered with a default of 4096 characters per buffering - or do you
all mean some other kind, maybe at the socket level?  As for characters it
wouldn't take much. Depending on what you want I would have thought that
adding #nextChar, #nextLine and maybe #contents to SocketReadStream would
provide all the options?

nextChar
    ^Character codePoint: self next

nextLine
    | line |
    line := ByteArray writeStream.
    [##(10 13) includes: self peek] whileFalse: [line add: self next].
    [##(10 13) includes: self peek] whileTrue: [self next].
    ^line contents asString

contents
    ^self contents asString

Regards
    Ian

Paul Hudson <[hidden email]> writes:

> Not really. It gives me a stream over a ByteArray (i.e. contains
> SmallIntegers) and I want one over a String (containing
> characters). And I'd like it buffered (so I'd like a FileStream, but
> over a socket, not a real file).  Which of course is pretty trivial
> with C and the stdio library....

As Ian has already noted, it is buffered. To get things out as a
character array (or String) is pretty simple to. Once that is done
where is the problem. I've haved no problems at all using Dolphin
Sockets using the HTTP protocol, which follows the same type of
principle as NNTP (line based request and response). Once I'd got the
readLine/writeLine behaviour I wanted based on characters rather than
integers of course!

Chris.
--
http://www.double.co.nz/smalltalk

Ian,

> I'm sure I've missed something in this conversation but SocketReadStream
> _is_ buffered with a default of 4096 characters per buffering - or do you
> all mean some other kind, maybe at the socket level?  As for characters it
> wouldn't take much. Depending on what you want I would have thought that
> adding #nextChar, #nextLine and maybe #contents to SocketReadStream would
> provide all the options?

[snip]

> contents
>     ^self contents asString

#contents is tricky because the stream is potentially bottomless.  There are
some previous posts on this topic.  IIRC, the best solution seems to be to
wait for the socket to close, at which point #contents can give a
well-defined result.  I've typically focused on termination conditions or
sending "packets" that are prefaced by the (dword) number of bytes to be
read.  That way, #contents isn't needed and the socket can remain open.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Bill,

> #contents is tricky because the stream is potentially bottomless.  There
> are some previous posts on this topic.  IIRC, the best solution seems to
> be to wait for the socket to close, at which point #contents can give a
> well-defined result.

I was thinking more in terms of when you want to get a lot of characters
from the SocketReadStream at the same time, rather than singly or a whole
line. After a #readPage for example.  On further reflection though it would
probably be better to override #upToEnd so that it returns a String as using
#readPage looks a bit destructive.  Because of this I don't think
SocketReadStream can ever give a "well defined result" to #contents as
#readPage overwrites each collection with the next, rather than
concatenating it to earlier buffers.

Regards
    Ian

"Ian Bartholomew" <[hidden email]> wrote in
news:BtAM7.7507$ur6.443950@wards:

> I'm sure I've missed something in this conversation but
> SocketReadStream _is_ buffered with a default of 4096 characters per
> buffering - or do you all mean some other kind, maybe at the socket
> level?  As for characters it wouldn't take much. Depending on what you
> want I would have thought that adding #nextChar, #nextLine and maybe
> #contents to SocketReadStream would provide all the options?

<embarrassed shuffle>

Think I misread something somewhere, or looked a something else. I'm sure I looked at something and thought
it wasn't buffered.

>    [##(10 13) includes: self peek] whileFalse: [line add: self next].
>    [##(10 13) includes: self peek] whileTrue: [self next].

Looks like this will skip over blank lines?

"Bill Schwab" <[hidden email]> wrote in
news:9tug6s$50eil$[hidden email]:

> I could be wrong, but, I think you might be transferring the
> complexity of a C program over to Smalltalk.  In C, one would think
> nothing of having some kind of polling loop and managing memory to go
> with it - it's the nature of the language.  A background thread that
> blocks until it has what it needs can result in much cleaner code and
> is well worth the extra trouble of thread synchronization, espcially
> given how easy it is to fork a block and create/share synchronization
> objects in Smalltalk.

Maybe. I'm not sure. But I should give this technique a try, anyway.

Paul,

> >    [##(10 13) includes: self peek] whileFalse: [line add: self next].
> >    [##(10 13) includes: self peek] whileTrue: [self next].
>
> Looks like this will skip over blank lines?

Yes. It wasn't intended to but it does! - a demonstration of why you
shouldn't add code to a reply without testing it :-(

The idea of the original was to allow line terminations of either lf or crlf
but, as you spotted, it then doesn't allow for blank lines. The following
does (and I've added the test framework to prove it!) but will need some
extra work to allow for things like partial lines.

Evaluate one of the five test lines to set the input and then evaluate the
rest of the code to extract the lines.

input := #[71 72 73 10 74 75 76 10 77 78 79 10]. "lf - no blank line"
input := #[71 72 73 10 74 75 76 10 10 77 78 79 10]. "lf - blank line"
input := #[71 72 73 13 10 74 75 76 13 10 77 78 79 13 10]. "crlf - no blank
line"
input := #[71 72 73 13 10 74 75 76 13 10 13 10 77 78 79 13 10]. "crlf -
blank line"
input := #[71 72 73 13 10 74 75 76 13 10 13 10 77 78 79]. "bang"

inputStream := input readStream.
lines := OrderedCollection new.
[inputStream atEnd] whileFalse: [
    line := ByteArray writeStream.
    [#[10 13] includes: inputStream peek]
        whileFalse: [line nextPut: inputStream next].
    inputStream peek == 13
        ifTrue: [inputStream next: 2]
        ifFalse: [inputStream next].
    lines add: line contents asString].
lines inspect

Regards
    Ian

Paul,

> Is there any convenient way of getting a stream connected to the socket
which works like one over a
> String? Ideally, I'd like to be able to able to use nextLine etc, or
perhaps a FileStream

Nobody else has mentioned this so far; I'm sorry it's so vague.

Somewhere in the past there was an early port of Swazoo (or some such name);
it -- IIRC -- had code to switch a SocketStream in and out of binary (i.e.
#next answers an Integer rather than Character) mode.

Unfortunately I'm away from home (and have been, and shall be, for some
time) so I have no ready Web access and can't get at my main filestore, so I
can't say for sure.

However, I suspect that a glance at the dolphinharbour site
(www.dolphinharbor.org), where, I believe, the swazoo stuff now lives, would
repay the time.

> P.

    -- chris

Hi Chris and Paul,

Swazoo does have a couple of Stream classes that have a ReadWrite protocol,
and can answer either bytes/byteArrays or characters/strings.

The classes are in the "Swazoo Sockets" package in the Spray download [1]

There is also a wiki [2] which is a start at documenting Swazoo.

Thanks,
Steve
www.dolphinharbor.org

[1] http://www.dolphinharbor.org/qad/spray/sprayDownload.html
[2] http://www.dolphinharbor.org:8080/SwazooDocumentation