Fuel - a fast object deployment tool

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
73 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Peter van Rooijen
On Fri, 10 Dec 2010 21:11:47 +0100, Eliot Miranda <[hidden email]> wrote:

Note that not writing source to the changes file has ancilliary benefits;
change recovery is now not polluted with package loads and the changes file
does not grow as packages are added, only as one's changes are made.
Unloading a package doesn't leave garbage in the changes files.

There are downsides.  Deploying a development image means deploying all the
associated parcel source files as well, and for this a platform-independent
Filename abstraction really helps.

Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels.

I hacked a dreadful implementation of overrides in vw3.0 and I don't think things are much better now.  But in http://www.mail-archive.com/pharo-project@.../msg17714.html I sketched how I think it should be done:

Maintain a global package load order (a stack of loaded packages, removing interior elements on unload).
Maintain a dictionary from method reference to set of package/method pairs for each method that is overridden.
When a package is removed search overrides and compute the overridden methods to be reinstalled, computing the uppermost method depending on the new package order.

So to answer your question, one finds the previous versions directly in the overrides dictionary, and sorts the results according to the current package load order.


Wasn't Levente asking about "regular" replacing of methods?

I thought his question was about that; if the source is not in

the changes file but in a parcel source file, then when I save

a new version of a method and want to look for the old one,

that will not be in the changes file.


When I build up a new image

from parcels, the load script copies all the sources to the

changes file, so I have easy access to the history.


Just wondering...


Peter


best
Eliot


Levente


best
Eliot



Levente




Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Eliot Miranda-2


On Wed, Dec 15, 2010 at 5:23 AM, Peter van Rooijen <[hidden email]> wrote:
On Fri, 10 Dec 2010 21:11:47 +0100, Eliot Miranda <[hidden email]> wrote:

Note that not writing source to the changes file has ancilliary benefits;
change recovery is now not polluted with package loads and the changes file
does not grow as packages are added, only as one's changes are made.
Unloading a package doesn't leave garbage in the changes files.

There are downsides.  Deploying a development image means deploying all the
associated parcel source files as well, and for this a platform-independent
Filename abstraction really helps.

Thanks for the explanation. I wonder how the previous versions of a method can be found using parcels.

I hacked a dreadful implementation of overrides in vw3.0 and I don't think things are much better now.  But in http://www.mail-archive.com/pharo-project@.../msg17714.html I sketched how I think it should be done:

Maintain a global package load order (a stack of loaded packages, removing interior elements on unload).
Maintain a dictionary from method reference to set of package/method pairs for each method that is overridden.
When a package is removed search overrides and compute the overridden methods to be reinstalled, computing the uppermost method depending on the new package order.

So to answer your question, one finds the previous versions directly in the overrides dictionary, and sorts the results according to the current package load order.


Wasn't Levente asking about "regular" replacing of methods?

I thought his question was about that; if the source is not in

the changes file but in a parcel source file, then when I save

a new version of a method and want to look for the old one,

that will not be in the changes file.


No.  Anything you write goes on the changes files including new method definitions, doits, reorganizations, method removals class removals etc.
 


When I build up a new image

from parcels, the load script copies all the sources to the

changes file, so I have easy access to the history.


Why bother?  A published parcel is an immutable versioned artifact.  It doesn't change over time, so, like a release sources file, one can always rely on its contents.

HTH
Eliot


Just wondering...


Peter


best
Eliot


Levente


best
Eliot



Levente





Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Igor Stasenko
On 15 December 2010 19:30, Eliot Miranda <[hidden email]> wrote:

>
>
> On Wed, Dec 15, 2010 at 5:23 AM, Peter van Rooijen <[hidden email]>
> wrote:
>>
>> On Fri, 10 Dec 2010 21:11:47 +0100, Eliot Miranda
>> <[hidden email]> wrote:
>>
>> Note that not writing source to the changes file has ancilliary benefits;
>> change recovery is now not polluted with package loads and the changes
>> file
>> does not grow as packages are added, only as one's changes are made.
>> Unloading a package doesn't leave garbage in the changes files.
>>
>> There are downsides.  Deploying a development image means deploying all
>> the
>> associated parcel source files as well, and for this a
>> platform-independent
>> Filename abstraction really helps.
>>>
>>> Thanks for the explanation. I wonder how the previous versions of a
>>> method can be found using parcels.
>>
>> I hacked a dreadful implementation of overrides in vw3.0 and I don't think
>> things are much better now.  But
>> in http://www.mail-archive.com/pharo-project@.../msg17714.html
>> I sketched how I think it should be done:
>> Maintain a global package load order (a stack of loaded packages, removing
>> interior elements on unload).
>> Maintain a dictionary from method reference to set of package/method pairs
>> for each method that is overridden.
>> When a package is removed search overrides and compute the overridden
>> methods to be reinstalled, computing the uppermost method depending on the
>> new package order.
>> So to answer your question, one finds the previous versions directly in
>> the overrides dictionary, and sorts the results according to the current
>> package load order.
>>
>> Wasn't Levente asking about "regular" replacing of methods?
>>
>> I thought his question was about that; if the source is not in
>>
>> the changes file but in a parcel source file, then when I save
>>
>> a new version of a method and want to look for the old one,
>>
>> that will not be in the changes file.
>
> No.  Anything you write goes on the changes files including new method
> definitions, doits, reorganizations, method removals class removals etc.
>
>>
>> When I build up a new image
>>
>> from parcels, the load script copies all the sources to the
>>
>> changes file, so I have easy access to the history.
>
> Why bother?  A published parcel is an immutable versioned artifact.  It
> doesn't change over time, so, like a release sources file, one can always
> rely on its contents.

+1
Instead of putting all methods and classes from package into .changes
it could be just a simple doit, like:
MC load: 'blahblah.mcz'


It could be somewhat different during merge.

> HTH
> Eliot



--
Best regards,
Igor Stasenko AKA sig.

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Eliot Miranda-2
In reply to this post by tinchodias
Hi Mariano and Martin,

    some feedback.  The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me.  I would just compute this map at the start of serialization.  It'll take hardly any time.  Whereas there's lots of scope for this information, if cached, to become stale over time.

Find attached.  If you're happy with this then before or after remember to evaluate
    SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper

best
Eliot

FLClassPoolAssociationMapper.st (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Eliot Miranda-2


On Wed, Jun 8, 2011 at 11:59 AM, Eliot Miranda <[hidden email]> wrote:
Hi Mariano and Martin,

    some feedback.  The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me.  I would just compute this map at the start of serialization.  It'll take hardly any time.  Whereas there's lots of scope for this information, if cached, to become stale over time.

Find attached.  If you're happy with this then before or after remember to evaluate
    SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper

and the same goes for FLGlobalMapper.  Also remember SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLGlobalMapper
 

best
Eliot


FLGlobalMapper.st (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Sean Malloy
In reply to this post by tinchodias
> Last months I and Tristan have been working on Fuel project, an object
> binary serialization tool. The idea is that objects are much more
> times loaded than stored, therefore it is worth to spend time while
> storing in order to have faster loading and user experience. We
> present an implementation of a pickle format that is based on
> clustering similar objects.

Having not looked at Fuel at all...

Has anyone done any serious work with Paul Baumann's SRP?

http://sourceforge.net/projects/srp/

http://web.archive.org/web/20060201011511/http://wiki.cs.uiuc.edu/CampSmalltalk/About+State+Replication+Protocol+%28SRP%29

I found it to be an extremely innovative serialisation mechanism, then
again it has been a few years since I looked at it.

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Eliot Miranda-2
In reply to this post by tinchodias
Hi Martin and Mariano,

    a couple of questions.  What's the right way to exclude certain objects from the serialization?  Is there a way of excluding certain inst vars from certain objects?

best
Eliot
Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Lukas Renggli
In reply to this post by Sean Malloy
On 9 June 2011 03:23, Sean Malloy <[hidden email]> wrote:

>> Last months I and Tristan have been working on Fuel project, an object
>> binary serialization tool. The idea is that objects are much more
>> times loaded than stored, therefore it is worth to spend time while
>> storing in order to have faster loading and user experience. We
>> present an implementation of a pickle format that is based on
>> clustering similar objects.
>
> Having not looked at Fuel at all...
>
> Has anyone done any serious work with Paul Baumann's SRP?
>
> http://sourceforge.net/projects/srp/

And also with MessagePack (http://msgpack.org/), a highly space/time
efficient binary serialization protocol. There is an implementation on
SqueakSource at http://www.squeaksource.com/MessagePack and a
high-level object-serialization protocol on top of that at
http://www.squeaksource.com/STOMP.

Lukas

--
Lukas Renggli
www.lukas-renggli.ch

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

tinchodias
In reply to this post by Eliot Miranda-2
Hi Eliot,

I am glad to receive your feedback. I definitely agree with that, your changes have been added on last version.

The reason for that cached information was mainly to support a Mariano's project requirement: he needs to frequently do lots of small serializations. But in any case, that cache should be managed externally of Fuel.

In my computer, this bench:
[ FLGlobalMapper newWith: nil ] bench.

went from '402,000 per second' to '35.8 per second'. But in the common use case of a serializer it is really imperceptible.

Thank you very much.
Martin


On Wed, Jun 8, 2011 at 5:20 PM, Eliot Miranda <[hidden email]> wrote:


On Wed, Jun 8, 2011 at 11:59 AM, Eliot Miranda <[hidden email]> wrote:
Hi Mariano and Martin,

    some feedback.  The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me.  I would just compute this map at the start of serialization.  It'll take hardly any time.  Whereas there's lots of scope for this information, if cached, to become stale over time.

Find attached.  If you're happy with this then before or after remember to evaluate
    SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper

and the same goes for FLGlobalMapper.  Also remember SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLGlobalMapper
 

best
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

tinchodias
In reply to this post by Eliot Miranda-2

On Wed, Jun 8, 2011 at 10:35 PM, Eliot Miranda <[hidden email]> wrote:
Hi Martin and Mariano,

    a couple of questions.  What's the right way to exclude certain objects from the serialization?  Is there a way of excluding certain inst vars from certain objects?


Hi,
 
We are close to have this feature, but still is an open issue.

The idea is to to support this "statically" via:

ClassWithTransients >> fuelTransientInstanceVariables
   ^ #(myTransientValue1 myTransientValue1)

or more dinamically via:

FLSerializationSettings newDefault
    onClass: ClassWithTransients
    declareTransients: #(myTransientValue1 myTransientValue1).

Do you have a better idea?

Cheers
Martin

 
best
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Stéphane Ducasse
In reply to this post by Lukas Renggli
do  you know what is the encoding scheme under message pack?
Huffman like?
It would be a nice student projects to get one version for Smalltalk
Stef

On Jun 9, 2011, at 6:51 AM, Lukas Renggli wrote:

> On 9 June 2011 03:23, Sean Malloy <[hidden email]> wrote:
>>> Last months I and Tristan have been working on Fuel project, an object
>>> binary serialization tool. The idea is that objects are much more
>>> times loaded than stored, therefore it is worth to spend time while
>>> storing in order to have faster loading and user experience. We
>>> present an implementation of a pickle format that is based on
>>> clustering similar objects.
>>
>> Having not looked at Fuel at all...
>>
>> Has anyone done any serious work with Paul Baumann's SRP?
>>
>> http://sourceforge.net/projects/srp/
>
> And also with MessagePack (http://msgpack.org/), a highly space/time
> efficient binary serialization protocol. There is an implementation on
> SqueakSource at http://www.squeaksource.com/MessagePack and a
> high-level object-serialization protocol on top of that at
> http://www.squeaksource.com/STOMP.
>
> Lukas
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>


Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Mariano Martinez Peck
In reply to this post by tinchodias


On Thu, Jun 9, 2011 at 9:18 AM, Martin Dias <[hidden email]> wrote:
Hi Eliot,

I am glad to receive your feedback. I definitely agree with that, your changes have been added on last version.

The reason for that cached information was mainly to support a Mariano's project requirement: he needs to frequently do lots of small serializations.

Exactly. I was the guilty. In fact, at the beginning this was exactly as you suggested now. And I ask Martin to change it heheheh.
 
But in any case, that cache should be managed externally of Fuel.


Yes, I can do that.
 
In my computer, this bench:
[ FLGlobalMapper newWith: nil ] bench.

went from '402,000 per second' to '35.8 per second'. But in the common use case of a serializer it is really imperceptible.

Thank you very much.
Martin



On Wed, Jun 8, 2011 at 5:20 PM, Eliot Miranda <[hidden email]> wrote:


On Wed, Jun 8, 2011 at 11:59 AM, Eliot Miranda <[hidden email]> wrote:
Hi Mariano and Martin,

    some feedback.  The ClassPoolBindings cache in FLClassPoolAssociationMapper feels like a mistake to me.  I would just compute this map at the start of serialization.  It'll take hardly any time.  Whereas there's lots of scope for this information, if cached, to become stale over time.

Find attached.  If you're happy with this then before or after remember to evaluate
    SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLClassPoolAssociationMapper

and the same goes for FLGlobalMapper.  Also remember SystemChangeNotifier uniqueInstance noMoreNotificationsFor: FLGlobalMapper
 

best
Eliot





--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

tinchodias
In reply to this post by Lukas Renggli
Hi

> Has anyone done any serious work with Paul Baumann's SRP?
>
> http://sourceforge.net/projects/srp/

And also with MessagePack (http://msgpack.org/), a highly space/time
efficient binary serialization protocol. There is an implementation on
SqueakSource at http://www.squeaksource.com/MessagePack and a
high-level object-serialization protocol on top of that at
http://www.squeaksource.com/STOMP.


I have never used SRP or MessagePack, both seems great. An important difference of Fuel is that it is not focused in cross-dialect or cross-language object exchange.
Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Lukas Renggli
In reply to this post by Stéphane Ducasse
On 9 June 2011 09:40, Stéphane Ducasse <[hidden email]> wrote:
> do  you know what is the encoding scheme under message pack?

It is very simple (like JSON and serializes only booleans, integers,
floats, strings, arrays and dictionaries) and is described here:

- design: http://wiki.msgpack.org/display/MSGPACK/Design+of+Serialization
- format: http://wiki.msgpack.org/display/MSGPACK/Format+specification.

> Huffman like?

Kind of, but static. Common objects are serialized with very few bits.

> It would be a nice student projects to get one version for Smalltalk

I posted the links in my previous mail, there is already an
implementation on SqueakSource
(http://www.squeaksource.com/MessagePack). Also there is a high-level
implementation that can serialize any Smalltalk object
(http://www.squeaksource.com/STOMP). It seems to handle class changes
too.

Lukas

--
Lukas Renggli
www.lukas-renggli.ch

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Eliot Miranda-2
In reply to this post by tinchodias


On Thu, Jun 9, 2011 at 12:36 AM, Martin Dias <[hidden email]> wrote:

On Wed, Jun 8, 2011 at 10:35 PM, Eliot Miranda <[hidden email]> wrote:
Hi Martin and Mariano,

    a couple of questions.  What's the right way to exclude certain objects from the serialization?  Is there a way of excluding certain inst vars from certain objects?


Hi,
 
We are close to have this feature, but still is an open issue.

The idea is to to support this "statically" via:

ClassWithTransients >> fuelTransientInstanceVariables
   ^ #(myTransientValue1 myTransientValue1)

or more dinamically via:

FLSerializationSettings newDefault
    onClass: ClassWithTransients
    declareTransients: #(myTransientValue1 myTransientValue1).

Do you have a better idea?

No.  The above looks fine.
 

Cheers
Martin

 
best
Eliot


Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Stéphane Ducasse
In reply to this post by Lukas Renggli

On Jun 9, 2011, at 7:01 PM, Lukas Renggli wrote:

> On 9 June 2011 09:40, Stéphane Ducasse <[hidden email]> wrote:
>> do  you know what is the encoding scheme under message pack?
>
> It is very simple (like JSON and serializes only booleans, integers,
> floats, strings, arrays and dictionaries) and is described here:
>
> - design: http://wiki.msgpack.org/display/MSGPACK/Design+of+Serialization
> - format: http://wiki.msgpack.org/display/MSGPACK/Format+specification.
>
>> Huffman like?
>
> Kind of, but static. Common objects are serialized with very few bits.
>
>> It would be a nice student projects to get one version for Smalltalk
>
> I posted the links in my previous mail, there is already an
> implementation on SqueakSource
> (http://www.squeaksource.com/MessagePack). Also there is a high-level
> implementation that can serialize any Smalltalk object
> (http://www.squeaksource.com/STOMP). It seems to handle class changes
> too.

Ok I missed that.

>
> Lukas
>
> --
> Lukas Renggli
> www.lukas-renggli.ch
>


Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Stéphane Ducasse
In reply to this post by Eliot Miranda-2
>
> FLSerializationSettings newDefault
>     onClass: ClassWithTransients
>     declareTransients: #(myTransientValue1 myTransientValue1).
>
> Do you have a better idea?

I'm dead tired and not having clear idea but is it better to mention what should be save or what
should not be saved?

Stef
Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Eliot Miranda-2


On Thu, Jun 9, 2011 at 1:26 PM, Stéphane Ducasse <[hidden email]> wrote:
>
> FLSerializationSettings newDefault
>     onClass: ClassWithTransients
>     declareTransients: #(myTransientValue1 myTransientValue1).
>
> Do you have a better idea?

I'm dead tired and not having clear idea but is it better to mention what should be save or what
should not be saved?

IMO, what should not, since this is the exception. Most times one wants to save all inst vars.
 
e.


Stef

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Mariano Martinez Peck
In reply to this post by Eliot Miranda-2


On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <[hidden email]> wrote:
Hi Martin and Mariano,

    a couple of questions.  What's the right way to exclude certain objects from the serialization?  Is there a way of excluding certain inst vars from certain objects?



Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.

Behavior >> fuelIgnoredInstanceVariableNames
    "Indicates which variables have to be ignored during serialization."

    ^#()


MyClass class >> fuelIgnoredInstanceVariableNames
  ^ #('instVar1')


The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ?  We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...

thanks


--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Fuel - a fast object deployment tool

Henrik Sperre Johansen
On 13.06.2011 19:16, Mariano Martinez Peck wrote:


On Thu, Jun 9, 2011 at 3:35 AM, Eliot Miranda <[hidden email]> wrote:
Hi Martin and Mariano,

    a couple of questions.  What's the right way to exclude certain objects from the serialization?  Is there a way of excluding certain inst vars from certain objects?



Eliot and the rest....Martin implemented this feature in Fuel-MartinDias.258. For the moment, we decided to put #fuelIgnoredInstanceVariableNames at class side.

Behavior >> fuelIgnoredInstanceVariableNames
    "Indicates which variables have to be ignored during serialization."

    ^#()


MyClass class >> fuelIgnoredInstanceVariableNames
  ^ #('instVar1')


The impact in speed is nothing, so this is good. Now....we were thinking if it is common to need that 2 different instances of the same class need different instVars to ignore. Is this common ? do you usually need this ?  We checked in SIXX and it is at instance side. Java uses the prefix 'transient' so it is at class side...

thanks


--
Mariano
http://marianopeck.wordpress.com

You can have variables on both instances and classes.
 I believe in SIXX you send #sixxIgnorableInstVarNames to the object you want to serialize.
IE. on class side you define ignorable class instance variables, while on instance side you define ignorable instance variables.

Cheers,
Henry

1234