Re: [Pharo-project] XTream was Re: Ocean (was Re: Less plugins, more Smalltalk code!)

I saw this thread on the pharo list and figured that a bit of feedback from Michael and myself might be helpful. Please, forgive the cross-posting, this concerns both dialects and I'd like to make this discussion visible on the VW side as well.

First of all, let me thank everyone for the "xtreamly" positive feedback. It sure feels good to get a thumbs up once in a while. So, thanks!

Since there was a question earlier about availability of my ESUG slides, I'll also add that a rather crude and plain but usable pdf export of those is available on the ESUG slideshare. I'd be happy to forward the pdf to anyone interested as well. It's full of examples and it certainly should be easier to copy those from there than from the video or memory :-).

As far as the future plans for Xtreams go (at least on VW side), we'd primarily like to validate the design with actual use, and fix any outstanding issues of course. We are certainly open towards collaboration with others on further development or porting efforts. I'd be happy to help sort out any porting issues (at least from the VW side).

Regarding the general direction of the project, we are skeptical about the possibility of maintaining backward compatibility with the classic streams and yet being able to make the kind of changes that we're experimenting with in Xtreams. At this point we believe that the only practical approach is to let them live side by side and let people chose whichever fits better their needs. If Xtreams ever do gain significant acceptance there certainly is a very real possibility that we'll end up having to live with both frameworks, maybe forever. But we don't think there's practical way to somehow evolve the classic streams into something like Xtreams without wreaking havoc.

That's also partly why we took the liberty of changing even the basic APIs. They don't behave quite the same way (intentionally) in a number of cases and given that the two will likely have to coexist in the same context, it might be actually beneficial to make them distinctly different, so that it's clear which one is used where. Anyway, what I'm getting at is that some of the choices we've already made with Xtreams might be hard for us to compromise on, but we can certainly discuss the options. On the other hand, implementation changes that aid portability and don't compromise the basic goals are certainly a fair game. Ideally we'll be able to agree on an arrangement that would allow the project to co-evolve on multiple-dialects without the need to effectively fork the code-base. That would certainly be worth-while and we're definitely interested in trying to achieve that. I'll be at the Smalltalks 2010 conference in november and would be happy to discuss any details if th!
 ere are people interested in this.

Obviously, given the license, people are free to fork the code or start from scratch or whatever and take it in whatever direction they see fit. I just want to make it clear that we are open to collaboration as long as we can agree on general direction. So, no hard feelings Nicolas :-), although we may want to sort out the naming eventually just for the sake of avoiding confusion in the Squeak and Pharo world (if Xtreams ever get traction there).

Anyway, I'm now subscribed to the pharo list so I'm reachable both there and on vwnc.

Martin

"Nicolas Cellier"<[hidden email]> wrote:

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Hi Martin, Michael.

thanks for this feedback, that's usefull.

2010/10/8  <[hidden email]>:
> I saw this thread on the pharo list and figured that a bit of feedback from Michael and myself might be helpful. Please, forgive the cross-posting, this concerns both dialects and I'd like to make this discussion visible on the VW side as well.
>
> First of all, let me thank everyone for the "xtreamly" positive feedback. It sure feels good to get a thumbs up once in a while. So, thanks!
>
> Since there was a question earlier about availability of my ESUG slides, I'll also add that a rather crude and plain but usable pdf export of those is available on the ESUG slideshare. I'd be happy to forward the pdf to anyone interested as well. It's full of examples and it certainly should be easier to copy those from there than from the video or memory :-).
>

More example than the site http://code.google.com/p/xtreams/ ? If so,
I'm interested.

> As far as the future plans for Xtreams go (at least on VW side), we'd primarily like to validate the design with actual use, and fix any outstanding issues of course. We are certainly open towards collaboration with others on further development or porting efforts. I'd be happy to help sort out any porting issues (at least from the VW side).
>
> Regarding the general direction of the project, we are skeptical about the possibility of maintaining backward compatibility with the classic streams and yet being able to make the kind of changes that we're experimenting with in Xtreams. At this point we believe that the only practical approach is to let them live side by side and let people chose whichever fits better their needs. If Xtreams ever do gain significant acceptance there certainly is a very real possibility that we'll end up having to live with both frameworks, maybe forever. But we don't think there's practical way to somehow evolve the classic streams into something like Xtreams without wreaking havoc.
>

Yes, I think this is even worse in Squeak, because of proliferation of
subclasses. I played with it, but that can't be a transparent
replacement. Rather a huge rewrite (not counting external packages...)

> That's also partly why we took the liberty of changing even the basic APIs. They don't behave quite the same way (intentionally) in a number of cases and given that the two will likely have to coexist in the same context, it might be actually beneficial to make them distinctly different, so that it's clear which one is used where. Anyway, what I'm getting at is that some of the choices we've already made with Xtreams might be hard for us to compromise on, but we can certainly discuss the options. On the other hand, implementation changes that aid portability and don't compromise the basic goals are certainly a fair game. Ideally we'll be able to agree on an arrangement that would allow the project to co-evolve on multiple-dialects without the need to effectively fork the code-base. That would certainly be worth-while and we're definitely interested in trying to achieve that. I'll be at the Smalltalks 2010 conference in november and would be happy to discuss any details if there are people interested in this.
>

Yes, that makes sense, what I call the Xtreme approach... Less confusion.
However it would be wonderfull to get feedback from average programmer
about this on real application.
Maybe this will be a matter of adding one missing convenient message or two...

> Obviously, given the license, people are free to fork the code or start from scratch or whatever and take it in whatever direction they see fit. I just want to make it clear that we are open to collaboration as long as we can agree on general direction. So, no hard feelings Nicolas :-), although we may want to sort out the naming eventually just for the sake of avoiding confusion in the Squeak and Pharo world (if Xtreams ever get traction there).
>

Hehe, I've got the feeling to surf on your hard work with low investment ;)
Xtream ideas should definitly continue to attract. J'espère que la
mayonnaise va prendre.
Of course the best option is to share. Though I don't think Xtream was
developped with portability as main goal, it's not sure it has so many
external dependencies.
Is there a tool to identify the dependency on classes outside the
package in VW ?
We know in advance a few concerning the system, File & Sockets, DLLCC,
compression...
With good package delimitation, that can be worked out.
The problem with Squeak is that we don't have a good File library (non
blocking). The good point with Xtream is that we don't need that many
primitives.
And last good point is the huge effort you made to coverage tests...
That should ease porting.

Then, there is the question of using Exception handling or not. In
Squeak, this isn't going to be a matter of compatibility (though the
difference might be subtle), rather a matter of efficiency.
However, Incomplete is a corner stone of Xtream implementation if I
understand correctly...
Squeak is not using EndOfStream
http://bugs.squeak.org/view.php?id=6755 and some of the efficiency
reasons are there
http://lists.squeakfoundation.org/pipermail/squeak-dev/2007-November/122434.html

On the other hand, the endOfStreamAction of Squeak_Xtream is pleasant
at first, but hard to handle across wrappers. I don't yet have an
answer whether it scales on a more complete xtream library or not
(thinking of blocking streams...).

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

We divided the Xtreams packages up with this in mind. The Core should be portable, the Terminals are not -- they must be implemented for each platform because they're the ones that use the platform specific objects.

> The problem with Squeak is that we don't have a good File library (non
> blocking). The good point with Xtream is that we don't need that many
> primitives.
> And last good point is the huge effort you made to coverage tests...
> That should ease porting.

It's okay if not all terminals and transformation are supported on every platform. We could create a capabilities grid for the different ports to show what can and can't be done.

Even in VisualWorks Exceptions are expensive. We went to great lengths to make sure the exception mechanism did not fire constantly. We spent several agonizing weeks studying the effects of our implementation on garbage generation and efficiency until we finally arrived at the design we have now. The biggest problems were with substreaming which we eventually solved.

Michael
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Resent, reply to all...

2010/10/9 Michael Lucas-Smith <[hidden email]>:
I'm browsing VW right now, at first sight there are not so many
uncompatibilities in core, should be solved easily
Incomplete>>#raise could be added (#signal in Squeak)
#copyEmpty: could be an extension
#isFixedArgument not existing in Squeak (external heap notion exists though)
Lack of namespace in Squeak shall force use of a prefix (XT ?).

Some terminals are generic and should port easily too:
BlockClosure/Buffer/Collection
For File, assuming they are non blocking, we can arrange a quick hack.

For transforms, the StreamEncoder does not exist in Squeak (better not
use existing Multilingual, a rewrite will be cleaner).

The simplest thing is just to port a few classes then bench... That's
worth a try.

Nicolas

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

"Nicolas Cellier"<[hidden email]> wrote:
> More example than the site http://code.google.com/p/xtreams/ ? If so, I'm interested.

Not more, but different. For example I'm quite fond of the Morse code decoding example:

        ('. ... ..- --.  -... .- .-. -.-. . .-.. --- -. .-  -- -- -..- ' reading
                transforming: [ :in :out || node beep |
                        node := MorseTree.
                        [ beep := in get.
                                beep = Character space
                        ] whileFalse: [
                                node := beep = $.
                                        ifTrue: [ node at: 3 ]
                                        ifFalse: [ node at: 2 ] ].
                        out put: node first ]
        ) rest

Where MorseTree is (http://en.wikipedia.org/wiki/File:Morse_code_tree3.png) encoded as nested three element arrays: #($ ($T ($M ($O) ($G ($Q) ($Z))) ($N ($K ($Y) ($C)) ($D ($X) ($B)))) ($E ($A ($W ($J) ($P)) ($R () ($L))) ($I ($U () ($F)) ($S ($V) ($H)))))

Anyway, I'll send you the PDF directly.

> Yes, that makes sense, what I call the Xtreme approach... Less confusion.
> However it would be wonderfull to get feedback from average programmer
> about this on real application.

Not sure what you meant.

> Maybe this will be a matter of adding one missing convenient message or two...

Yes we can certainly add a package that implements most (maybe all) of the classic API as an extension. But I'd rather keep the Core clean and minimal. This way you have the choice of both.

> Of course the best option is to share. Though I don't think Xtream was
> developped with portability as main goal, it's not sure it has so many
> external dependencies.

Yes, we didn't spend much time thinking about portability so far. We rewrote the whole thing bottom up several times trying to get the design right, so we didn't want to be distracted by other concerns. That doesn't mean that we can't focus on that now assuming we're reasonably happy with current design. I don't think there are particular dependencies that are critical to make Xtreams work in general. As long as we can preserve the efficiency, we can certainly make necessary changes.

That said, I think we'll just have dialect specific versions of some of the packages. Hopefully Xtreams-Core can be fully portable. Xtreams-Terminals are likely going to be dialect specific. Xtreams-Transforms could probably be portable if we move the VW specific encoding stuff elsewhere. Maybe it can just go into Xtreams-Xtras. It was nice to have the encodings with the core transforms but until we have portable encoders we can just maintain those in dialect specific packages. Something like that.

> Is there a tool to identify the dependency on classes outside the package in VW ?

Yes there is a tab in the browser that autocomputes prerequisites and a menu option that gives you depenedents if you want the other direction.

> We know in advance a few concerning the system, File & Sockets, DLLCC, compression...

Yes, I would expect each dialect to have it's own implementation of most of the terminal streams. It might be possible to share some of those, e.g. collection or buffer streams, but given that different dialects have different ways to deal with them efficiently, we may want to have dialect specific implementations anyway. Ultimately, it's fine if different dialects support different set of terminals. There will probably be some that we want everywhere, but maybe it would be great to be able to stream over some Morphs in Squeak or something such. If that's useful, there's no reason not to do it because VW doesn't have Morphic. Anyway, the point is not all the code has to be portable, we can try to share as much as is feasible and have dialect specific implementation of the rest.

So at least initially, I'd focus on getting a portable Core and Transforms, with dialect specific Terminals, and we can think about stuff in Xtras later.

Yes, this one is tricky. Exceptions cost something in VW as well, so you need to be reasonably prudent with them. As Michael mentioned we've made some serious adjustments in Xtreams earlier this year to avoid triggering Incomplete internally if we could get by without it. See

http://www.cincomsmalltalk.com/userblogs/mls/blogView?entry=3445952077
http://www.cincomsmalltalk.com/userblogs/mls/blogView?entry=3448891140

But I think your original argument that in common case the exception should not be triggered frequently is valid. Certainly if you write a micro-benchmark which stacks the odds the other way it will show, but as they say  micro-benchmarks are not applications. At the same time consider the cost of not using an exception. Exception allows me to write my algorithm without worrying about the end of stream if I wrap the whole thing in a handler. And the handler is normally located in the right place to handle the end of stream. Without the exception you'll have to add end checks to every single read call of the algorithm. I'm not talking about runtime cost here, I'm talking code complexity. And I don't see how an "end of stream block" makes this much better. The exception allows me to get out of the main body of the algorithm at any of its points, and if the handler exits, the main block gets unwound cleanly. How do I do that with a block that I set on the stream instance? You'd!
  have to carefully structure the code so that the algorithm body is in a method of its own and then there's a dedicated wrapper method around that so that the block can use a side-ways return to exit the method. That is some pretty hideous boiler-plate code, if you ask me.f

So, I'm not saying that exception costs are negligible, but there are often ways to minimize them in specific scenarios, often via well designed APIs and patterns (and proper internal implementation). As Michael's posts above show, we were able to match and beat the speed of regular streams in the example without sacrificing the API. At the same time Squeak is getting faster, so I'm not at all convinced that the slow-down factor always does and always will trump the other costs. In fact I feel that it's more often than not the other way around, that the slow down can be made sufficiently small so that it does pay off in terms of code complexity. Maybe some things will always want a stream that doesn't trigger exceptions, but that doesn't mean most things are better off that way. That's just my opinion.

Anyway, so far we've been doing quite well in terms of performance without sacrificing exceptions. I'd say let's see if we can keep that track record on Squeak as well. If we find out that it's just not good enough in general case, maybe we'll re-evaluate then. But at this point I'm not convinced that we need to drop the exception to keep Xtreams reasonably usable.

Martin

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

On Oct 8, 2010, at 10:05 PM, [hidden email] wrote:

> I saw this thread on the pharo list and figured that a bit of feedback from Michael and myself might be helpful. Please, forgive the cross-posting, this concerns both dialects and I'd like to make this discussion visible on the VW side as well.
>
> First of all, let me thank everyone for the "xtreamly" positive feedback. It sure feels good to get a thumbs up once in a while. So, thanks!
>
> Since there was a question earlier about availability of my ESUG slides, I'll also add that a rather crude and plain but usable pdf export of those is available on the ESUG slideshare. I'd be happy to forward the pdf to anyone interested as well. It's full of examples and it certainly should be easier to copy those from there than from the video or memory :-).
>
> As far as the future plans for Xtreams go (at least on VW side), we'd primarily like to validate the design with actual use, and fix any outstanding issues of course. We are certainly open towards collaboration with others on further development or porting efforts. I'd be happy to help sort out any porting issues (at least from the VW side).
>
> Regarding the general direction of the project, we are skeptical about the possibility of maintaining backward compatibility with the classic streams and yet being able to make the kind of changes that we're experimenting with in Xtreams. At this point we believe that the only practical approach is to let them live side by side and let people chose whichever fits better their needs. If Xtreams ever do gain significant acceptance there certainly is a very real possibility that we'll end up having to live with both frameworks, maybe forever. But we don't think there's practical way to somehow evolve the classic streams into something like Xtreams without wreaking havoc.

The best strategy.



> That's also partly why we took the liberty of changing even the basic APIs. They don't behave quite the same way (intentionally) in a number of cases and given that the two will likely have to coexist in the same context, it might be actually beneficial to make them distinctly different, so that it's clear which one is used where. Anyway, what I'm getting at is that some of the choices we've already made with Xtreams might be hard for us to compromise on, but we can certainly discuss the options. On the other hand, implementation changes that aid portability and don't compromise the basic goals are certainly a fair game. Ideally we'll be able to agree on an arrangement that would allow the project to co-evolve on multiple-dialects without the need to effectively fork the code-base. That would certainly be worth-while and we're definitely interested in trying to achieve that. I'll be at the Smalltalks 2010 conference in november and would be happy to discuss any details if !
 the
> re are people interested in this.
>
> Obviously, given the license, people are free to fork the code or start from scratch or whatever and take it in whatever direction they see fit. I just want to make it clear that we are open to collaboration as long as we can agree on general direction. So, no hard feelings Nicolas :-), although we may want to sort out the naming eventually just for the sake of avoiding confusion in the Squeak and Pharo world (if Xtreams ever get traction there).

+ Yes
Nicolas why not
                upstreams
                flow



_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Martin & Michael,

Thank you first for coming up with xtreams and second for taking the open source route and doing the effort to promote it.

I've been browsing a bit in vwnc 7.7 and it really shows that you guys have been thinking about this for quite some time and that this is not the first implementation. This is Smalltalk code that confirms my belief why Smalltalk is better.

I am happy with your reaction and I agree with the goal that xstreams should be loadable and usable next to the standard stuff. Selector names are clearly carefully chosen to make this possible, class names are a bit harder for Smalltalks without name spaces ;-)

I think we (for me, that is Pharo users, for others that might be Squeak users) should really take this opportunity (your invitation) and help turn the xstreams codebase into something portable to multiple Smalltalk implementations.

With Nicolas there is already somebody with great concrete experience and actual code to help make this a reality.

If I can, I will at least be an interested user and hopefully also capable of contributing.

Sven  

On 08 Oct 2010, at 22:04, [hidden email] wrote:
 there are people interested in this.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc