Sockets in Pharo CollaborActive Book

Hi!

On 09/01/2010 10:04 AM, Stéphane Ducasse wrote:
>> 2. I haven't looked at Andreas' new WebServer but I presume it is a "standard" forking server just like the old Comanche/KomHttpServer codebase is. A while back I wrote Blackfoot - a SimpleCGI implementation and then I extracted the essential "forking Socket server" code from KomHttpServer so that it can be easily understood and reused. I tend to point to Blackfoot as a very good "example" to use in this regard. We could even split out the generic part as a "standard class" to use. In fact, Stephen Pair did this with the TcpServices library a long time ago when he rewrote Comanche into KomHttpServer BUT... I think it got a bit overengineered and would instead point to Blackfoot as a good candidate for such a "SocketServer" class.
>
> sound cool do you have a little 2 pages intro to blackfoot?

Blackfoot in itself is a SimpleCGI implementation, I presented it on an
OOPSLA back in... 2006 or so:

http://goran.krampe.se/Blackfoot.pdf

...but my point was that it can easily serve as a good SIMPLE example of
a forking Socket server using SocketStream. It is VERY small and IIRC
the base "forking Socket server" is separate.

regards, Göran

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Hi Bill,

On 1 sept. 2010, at 05:59, Schwab,Wilhelm K wrote:

> The deeper question is whether we want to document how things are, or they they *should be* - which is VERY much different from the eventless and timeout ridden design that we have now.  It desperately needs to be redesigned.
>
I agree that the current status of the networking libraries is horrible.
But, to learn the current status and think of the alternatives, I felt the need to understand how thinks work now.

As Stéf said, Luc and I started the Ocean project.
Thanks you Bill and to Guille for your proposal to help.
It's a pitty that you don't attend to the ESUG conf (Your names are not on the registrations list) otherwise we could discuss this better. Cause Luc and I will give a short presentation describing the current status of Ocean and where it is headed.
Now, we need to think how to split the project in order to have other contributions.

Note BTW, our goal is to have a low-level Pharo wrappers of the networking facilities of the OS.
We started using Alien to avoid using a plugin and ease portability.
Once we get this low-level layer, we'll go on and use it in an OOP middleware like rST and UbiquiTalk.
Which is after all, the right level to work with since we want to deal with objects and message sends.
A direct use of Sockets isn't really OO most of the time.

Noury


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Noury,

First, please don't assume that you have to change course just because we offered to help.  If you are close to having a major improvement, then it might be best to just power through it.  If there are things holding you up, we might be able to help there.

Using Alien concerns me a little, as the last time I checked, it was not available on Linux??  Failure to support Linux would be a complete show-stopper for me.  Regardless, I would like to see your code as I have read some comments suggesting that Alien is cumbersome to use; you might show me that all is well.  My interest in Alien is primarily to support callbacks for numerical analysis.  So far, I have been adding functions to a shared library, which is easy enough, ultimately translates formulas using C++ (which is better at it than Smalltalk), likely makes things go faster than they would with Smalltalk in the loop, but is far from elegant.  Until Alien exists on Linux, it does not exist :(

Back to sockets, if the current situation were to stand, for whatever reason, I would consider tackling the problem using a plugin to get OS threads, and use them to achieve the non-blocking operation that is essential for interactive software.  In short, everything blocks its calling Smalltalk Process; nothing blocks the entire image.  Hopefully you recognize that a server listening with timeouts is hideous design; just start and stop the server as needed, and it should be harmless to the image.  If the server is not harmless to the image, it needs to be fixed.  Clients having timeouts isn't much better.  If you want to add helper methods with timeouts, who am I to tell others what to do, but we must not force timeouts on clients because there is no correct answer to how long something should wait, so we should not look for one.  What we should do is make sure that we never block the entire image, and let people make their own decisions.

Why would I consider a plugin?  In large part because there are hints that I would find examples to follow, and because OpenSSL has a very cumbersome non-blocking interface, at least that was my impression of it.  A plugin that I would write would hopefully support TCP and OpenSSL sockets, and hopefully IrDA.  Running those things on OS threads and signaling semaphores in the image is likely to be simpler than working out the details of non-blocking calls in the separate worlds - I think.  I can understand why you are not thrilled about creating a plugin.

The sad part of this is that the current sockets system appears to demonstrate all of the essential pieces to make something that really works; it just hasn't been done.  On attempting it, I would no doubt run across things that you found and led you to create a new system.  Creating something new allows it to coexist with the current code until it is ready, at which point it could be integrated into Pharo.

Thanks for working on it.  Is the code on Squeaksource representative of your current progress?  Please do not let my interest slow your efforts.  If you are indeed doing all of the right things, it might be that my efforts should be directed toward getting Alien going on Linux.  It might also be that I can be of most help simply by doing to your code what I did to DCOM long ago.  Our network was a harsh landscape, and I got fairly good bringing code to its knees on intranets.  This involves creating forked processes doing active communication across a hub that coincidentally gets unpowered w/o warning :)  Threads must block or give errors, but the images must continue to be responsive and other threads must run as expected.  Until we reliably pass that test, this is all just a game.

Bill


________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Noury Bouraqadi [[hidden email]]
Sent: Saturday, September 04, 2010 5:31 AM
To: [hidden email]
Subject: Re: [Pharo-project] Sockets in Pharo CollaborActive Book

Hi Bill,

On 1 sept. 2010, at 05:59, Schwab,Wilhelm K wrote:

> The deeper question is whether we want to document how things are, or they they *should be* - which is VERY much different from the eventless and timeout ridden design that we have now.  It desperately needs to be redesigned.
>
I agree that the current status of the networking libraries is horrible.
But, to learn the current status and think of the alternatives, I felt the need to understand how thinks work now.

As Stéf said, Luc and I started the Ocean project.
Thanks you Bill and to Guille for your proposal to help.
It's a pitty that you don't attend to the ESUG conf (Your names are not on the registrations list) otherwise we could discuss this better. Cause Luc and I will give a short presentation describing the current status of Ocean and where it is headed.
Now, we need to think how to split the project in order to have other contributions.

Note BTW, our goal is to have a low-level Pharo wrappers of the networking facilities of the OS.
We started using Alien to avoid using a plugin and ease portability.
Once we get this low-level layer, we'll go on and use it in an OOP middleware like rST and UbiquiTalk.
Which is after all, the right level to work with since we want to deal with objects and message sends.
A direct use of Sockets isn't really OO most of the time.

Noury


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Noury,

I was struggling a little with a stream for serial ports, and thought you might have tackled the problem of a generating stream (one that can grow in the background).  Now that I type that, maybe Nile has the answer??

Re Ocean, I grabbed a .mcz from SqueakSource and simply browsed the main package.  Some comments:

(1) no socket stream :(   Streams are really slick things, and good read and write stream semantics should exist for sockets.  FWIW, Dolphin separates the socket read and write streams, and we might do well to follow their lead.

(2) ByteArray>>asAlien loops over the bytes.  Do all the bounds checking you want up front (once) and then use memcpy() or something to actually transfer the data.  It recently came to my attention that Squeak has no formalized approach to access, copy and move external memory; Pharo needs to offer that, and this is a good place to start.

(3) You are planning to attack ipv4 vs. 6 with polymorphism - good call!!

(4) OCNTcpSocket>>send: sizes a buffer to match the data provided to it.  I'm not sure I like that.  You might consider renaming your current #send: to #basicSend: and adding something like

send:blob
| in |
in := blob readStream.
[ in atEnd ] whileFalse:[
   self basicSend:( in nextAvailable:self alienDataBufferSize ).
].

so that a fixed buffer (which will have to be allocated when the socket is opened) is used to send large blobs in buffer-sized chunks.  Better still would be to use array indexing to cope with the blob in pieces but in place to skip the copying.

Bill


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On 5 sept. 2010, at 03:22, Schwab,Wilhelm K wrote:

> Noury,
>
> I was struggling a little with a stream for serial ports, and thought you might have tackled the problem of a generating stream (one that can grow in the background).  Now that I type that, maybe Nile has the answer??
>
> Re Ocean, I grabbed a .mcz from SqueakSource and simply browsed the main package.  Some comments:
>
> (1) no socket stream :(   Streams are really slick things, and good read and write stream semantics should exist for sockets.  FWIW, Dolphin separates the socket read and write streams, and we might do well to follow their lead.
>
Not yet ;-)
So far, we want to have a clean socket. And then we'll rebuild a SocketStream on top of it.

> (2) ByteArray>>asAlien loops over the bytes.  Do all the bounds checking you want up front (once) and then use memcpy() or something to actually transfer the data.  It recently came to my attention that Squeak has no formalized approach to access, copy and move external memory; Pharo needs to offer that, and this is a good place to start.
>
Thanks for the hint.

> (3) You are planning to attack ipv4 vs. 6 with polymorphism - good call!!
>
Yes. One reason behind doing Ocean is that we found existing library too dirty to improve.
So, we try to have a well designed tested library.
We discussed this with Luc. And the rational behind our decision to do it this way is that Socket is but a basic wrapper for the actual OS socket.
It shouldn't provide any extra features.

Spliting a blob is something I'd rather do by using a SocketStream.

Thanx for the feedback,
Noury



_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project