Xtreams : first embryonary port on Squeak

So, I started a quick port of VW Xtreams to Squeak this evening.
Only the Xtreams-Core and the trivial Xtreams-Terminals (no
file/pipe/socket/pointer).
You'll find code at http://www.squeaksource.com/XTream.
My previous experimental Xtream project has been renamed SqueaXTream
to reduce confusion, but still sits in the same project.
Please find my report below.

cheers

Nicolas

--------------------- REPORT  ---------------------

THE CONTENTS:

This first port is made of 3 packages
- Xtreams-Core
- Xtreams-Terminals
- Xtreams-VWCompatible
Plus 2 tests
- Xtreams-CoreTests
- Xtreams-TerminalsTests
Please notice I removed one hyphen from test package names for MC compatibility.

THE PROCEDURE:

I didn't use FileOut30 not any more advanced tool, but just a simple
copy/paste strategy.
This gave me a chance to do a fast review of code.
For porting the future evolutions of VW Xtreams, more advanced tools
will be necessary, the manual approach does not scale.

THE MODIFICATIONS:

The changes I made are quite restricted, which indicates that Xtreams
is not that hard to port.
Main changes are due to lack of namespace in Squeak:
- I added a XT prefix to every Xtreams classes (no Namespace in Squeak)
- I added XTPools, a SharedPool for holding the single Xtreams
namespace variable DefaultBufferSize
- I transformed class SharedVariables into class variables and added
proper class side initialization for these in XTWriteStream

For the rest, I tried to not change Xtreams contents. but rather did
implement VW compatible messages when possible in the
Xtreams-VWCompatible package when they did not exist in Squeak.
These message are a full rewrite and are also released under MIT license.
The few exceptions are:
- I modified originator -> self originator in XTIncomplete.
- I changed the senders of #newInFixedSpace: to rather send an #error:
'not implemented in Squeak' in XTRecyclingBuffer
- I replaced #waitIfCurtailedSignal/#signal with a #critical: section
in XTRecyclingBuffer
- I replaced  Core.Timer after:do: with self after:do: in Xtreams-CoreTests

A DISCUSSION OF TECHNICAL DETAILS:

I first attempted to replace #growToAtLeast: with #grownBy: because
Squeak become: is notoriously inefficient.
But obviously, this did not work, the tests rely on destination
preserving its identity.
This clearly is going to degrade micro benchmarks when the destination
collection capacity is not well adjusted in advance.
I think we should discuss this particular point. Is identity
preservation absolutely required, or just convenient ?

THE STATUS OF TESTS:

Tests do not all pass. There seems to be a bug in Squeak
#replace:from:to:with: when the replacement is the collection itself,
moved to the right (this is with a COG VM).

THE FUTURE:

I will probably continue porting with reduced activity, so I invite
any interested person to help. Right now, the project is world
writeable, code commited here falling under MIT license.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

On Sun, 10 Oct 2010, Nicolas Cellier wrote:

> So, I started a quick port of VW Xtreams to Squeak this evening.
> Only the Xtreams-Core and the trivial Xtreams-Terminals (no
> file/pipe/socket/pointer).
> You'll find code at http://www.squeaksource.com/XTream.
> My previous experimental Xtream project has been renamed SqueaXTream
> to reduce confusion, but still sits in the same project.
> Please find my report below.

Great. :)

If you can create a diff from the VW changes, then porting "by hand"
shouldn't be hard.

I've never seen a real world example where this identity preservation was
useful, but it causes trouble when literals aren't immutable. IMHO the
#become: send is just convenient with VW's stream implementation and it
has no advantage over Squeak's approach.

>
> THE STATUS OF TESTS:
>
> Tests do not all pass. There seems to be a bug in Squeak
> #replace:from:to:with: when the replacement is the collection itself,
> moved to the right (this is with a COG VM).

#replace:from:to:with: is not supposed to work for moves. IIRC the
primitive version uses memcpy (or strncpy), so moving to the left works
just because the undelying C function supports it. I used to (ab)use this
feature though.


Levente

>
> THE FUTURE:
>
> I will probably continue porting with reduced activity, so I invite
> any interested person to help. Right now, the project is world
> writeable, code commited here falling under MIT license.
>
>

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

FWIW, SIF might help you:

   http://www.pocketsmalltalk.com/sif/

I have put a lot of code through it in the past couple of years.


________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Nicolas Cellier [[hidden email]]
Sent: Saturday, October 09, 2010 6:33 PM
To: The general-purpose Squeak developers list; Pharo Development; [hidden email]
Subject: [Pharo-project] Xtreams : first embryonary port on Squeak

So, I started a quick port of VW Xtreams to Squeak this evening.
Only the Xtreams-Core and the trivial Xtreams-Terminals (no
file/pipe/socket/pointer).
You'll find code at http://www.squeaksource.com/XTream.
My previous experimental Xtream project has been renamed SqueaXTream
to reduce confusion, but still sits in the same project.
Please find my report below.

cheers

Nicolas

--------------------- REPORT  ---------------------

THE CONTENTS:

This first port is made of 3 packages
- Xtreams-Core
- Xtreams-Terminals
- Xtreams-VWCompatible
Plus 2 tests
- Xtreams-CoreTests
- Xtreams-TerminalsTests
Please notice I removed one hyphen from test package names for MC compatibility.

THE PROCEDURE:

I didn't use FileOut30 not any more advanced tool, but just a simple
copy/paste strategy.
This gave me a chance to do a fast review of code.
For porting the future evolutions of VW Xtreams, more advanced tools
will be necessary, the manual approach does not scale.

THE MODIFICATIONS:

The changes I made are quite restricted, which indicates that Xtreams
is not that hard to port.
Main changes are due to lack of namespace in Squeak:
- I added a XT prefix to every Xtreams classes (no Namespace in Squeak)
- I added XTPools, a SharedPool for holding the single Xtreams
namespace variable DefaultBufferSize
- I transformed class SharedVariables into class variables and added
proper class side initialization for these in XTWriteStream

For the rest, I tried to not change Xtreams contents. but rather did
implement VW compatible messages when possible in the
Xtreams-VWCompatible package when they did not exist in Squeak.
These message are a full rewrite and are also released under MIT license.
The few exceptions are:
- I modified originator -> self originator in XTIncomplete.
- I changed the senders of #newInFixedSpace: to rather send an #error:
'not implemented in Squeak' in XTRecyclingBuffer
- I replaced #waitIfCurtailedSignal/#signal with a #critical: section
in XTRecyclingBuffer
- I replaced  Core.Timer after:do: with self after:do: in Xtreams-CoreTests

A DISCUSSION OF TECHNICAL DETAILS:

I first attempted to replace #growToAtLeast: with #grownBy: because
Squeak become: is notoriously inefficient.
But obviously, this did not work, the tests rely on destination
preserving its identity.
This clearly is going to degrade micro benchmarks when the destination
collection capacity is not well adjusted in advance.
I think we should discuss this particular point. Is identity
preservation absolutely required, or just convenient ?

THE STATUS OF TESTS:

Tests do not all pass. There seems to be a bug in Squeak
#replace:from:to:with: when the replacement is the collection itself,
moved to the right (this is with a COG VM).

THE FUTURE:

I will probably continue porting with reduced activity, so I invite
any interested person to help. Right now, the project is world
writeable, code commited here falling under MIT license.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

2010/10/10 Michael Lucas-Smith <[hidden email]>:
This is in:

CollectionReadingWritingTest>>setUp
       | buffer |
       buffer := XTElasticBuffer on: ByteArray new.
       input := buffer reading.
       output := buffer writing.

The expectation is that the buffer will be shared between input and output.
If output modifies a copy of buffer, then the tests fail (input is
empty or partial or too long if WriteStream>>#close did not perform a
#become: )

Nicolas

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

2010/10/10 Levente Uzonyi <[hidden email]>:
OK, since we use memcpy rather than memmove in Squeak VM, I
implemented the trivial work around - move through a motionBuffer
temporary copy.

I now have a single test failing,
XTCollectionWriteStream>>#testReadWriteLargeAmount and this must be
due to my #after:do: simplistic implementation.
Messing with #fork in SUnit TestCase is tricky, and I need help on this point.

Nicolas

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

Nicolas,

#fork in a test?  Do you mean like the code below?  The thread running the test waits on something signaled toward the end of the forked thread.  

If you want a watchdog timer, you could fork another thread that waits on a delay, sets an error flag (best to use a shared queue or a mutex to avoid weirdness) and then signals the test semaphore.  You might also want to use #ifCurtailed: to set an error flag and wrap that with #ensure: to signal the semaphore.  You don't necessarily need to be perfect about signaling the semaphore because failure to do so is not much different from your test going into an infinite loop (actually more benign because the stack isn't growing); you'll realize it, break, and figure out what didn't happen or at least "that it didn't" and try again forewarned.  

Does that help?

Bill


testLocalServerAndClient
        "11-09 - try a server and a client in the same image."

                | server port client serverSocket serverSocketStream text thread done read |

        port := 4900.
        done := Semaphore new.

        "11-09 - gag the prompter below"
        serverSocket := nil.

        server := ConnectionQueue portNumber:port queueLength:5.
        [
                client := SocketStream openConnectionToHostNamed:'localhost' port:port.
                [ serverSocket isNil ] whileTrue:[
                        serverSocket := server getConnectionOrNil.
                ].

                serverSocketStream := SocketStream on:serverSocket.
                text := 'Long live Tux!'.
                serverSocketStream nextPutAll:text; flush.

                "11-09 - this hung up w/o the thread.
                11-09 - as you were; the problem was a lack of #flush on the send side, so there was
                nothing for the client side to read."
                thread := [
                        read := client nextMany:text size.
                        done signal.
                ] forkAt:Processor userBackgroundPriority.
                thread name:'READ'.

                done wait.
                self should:[ read = text ].
                Transcript nextPutAll:'Just to make the point, we read: '; nextPutAll:read; cr; flush.

        ] ensure:[
                server destroy.
                serverSocketStream close.
                client close.
        ].

        "11-09 - no prompts"
        serverSocket := serverSocket.
        client := client.
        serverSocketStream := serverSocketStream.
        thread := thread.

        "
                NetworkSmokeTestCase prod:#testLocalServerAndClient.
        "



________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Nicolas Cellier [[hidden email]]
Sent: Sunday, October 10, 2010 7:05 AM
To: The general-purpose Squeak developers list
Cc: Pharo Development
Subject: Re: [Pharo-project] [squeak-dev] Xtreams : first embryonary port on    Squeak

2010/10/10 Levente Uzonyi <[hidden email]>:
OK, since we use memcpy rather than memmove in Squeak VM, I
implemented the trivial work around - move through a motionBuffer
temporary copy.

I now have a single test failing,
XTCollectionWriteStream>>#testReadWriteLargeAmount and this must be
due to my #after:do: simplistic implementation.
Messing with #fork in SUnit TestCase is tricky, and I need help on this point.

Nicolas

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

"Schwab,Wilhelm K"<[hidden email]> wrote:
> FWIW, SIF might help you:
>
>    http://www.pocketsmalltalk.com/sif/
>
> I have put a lot of code through it in the past couple of years.

I'd recommend giving the Monticello tools in VW a try before trying to put something together from lower level tools. It's used to keep Seaside in sync in VW, so it should handle most of the obvious obstacles. With the help of your Xtreams-VWCompatibility package on Squeak side and maybe some adjustments in the portable code and maybe moving some of the non-portable code into packages of their own, we should be able to make this largely automatic.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

This still seems confusing. Could we just have a new SqueakSource
project called "Xtreams" (small T + an S)? It's not like projects are
expensive to create...

Julian

On Sat, Oct 9, 2010 at 11:33 PM, Nicolas Cellier
<[hidden email]> wrote:
> So, I started a quick port of VW Xtreams to Squeak this evening.
> Only the Xtreams-Core and the trivial Xtreams-Terminals (no
> file/pipe/socket/pointer).
> You'll find code at http://www.squeaksource.com/XTream.
> My previous experimental Xtream project has been renamed SqueaXTream
> to reduce confusion, but still sits in the same project.
> Please find my report below.

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project