When will Fuel file format stabilize?
Locked
123
123
Re: When will Fuel file format stabilize?
 
|
On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse <[hidden email]> wrote: Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again.
It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example: I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ? Thanks Because then we get a win win situation. -- Mariano http://marianopeck.wordpress.com |
Re: When will Fuel file format stabilize?
|
|
On Sun, Oct 16, 2011 at 3:01 PM, Mariano Martinez Peck <[hidden email]> wrote:
(of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] )
-- Mariano http://marianopeck.wordpress.com |
Re: When will Fuel file format stabilize?
|
|
Stef, Mariano,
There are obvious advantages to a development phase during which serialized streams are at risk. At some point, I suspect Fuel will have to take the reading of old streams as a nearly sacred obligation. Otherwise, it sounds like a cheap horror movie series: "Fuel II: Your data go in, but they don't come out." I am using humor to make a point, not to ridicule you. I read the paper, and learned a LOT. Please see my post from yesterday referring to Dolphin's #environment. You might be able to simply facade old formats into a coherent backwardly-compatible whole. Without some attention to that, there are going to be projects that will not adopt Fuel. Bill ________________________________________ From: [hidden email] [[hidden email]] On Behalf Of Mariano Martinez Peck [[hidden email]] Sent: Sunday, October 16, 2011 9:03 AM To: [hidden email] Subject: Re: [Pharo-project] When will Fuel file format stabilize? On Sun, Oct 16, 2011 at 3:01 PM, Mariano Martinez Peck <[hidden email]<mailto:[hidden email]>> wrote: On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse <[hidden email]<mailto:[hidden email]>> wrote: Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: If by design a serializer takes the perspective that it will be able to load old formats and ensure evolution all the time then you have a really cool serializer. Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again. Now why in fuel it is not possible to have FuelMaterializer load: aFile load using the default = current format FuelMaterializer load: aFile asOfVersion: 1.4 Of course this can be much slower but when you want to load data of the past you are just happy that this is working. So would not be possible to milestone formats get the default working fast get old version loading slowler but loading. It sounds easy, but at least from what I can see this is very complicated. Do you know a serializer that support this? Lets clarify with an example: (of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] ) I serialize graph X with Fuel 1.4 in Pharo 1.1. Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ? Thanks Because then we get a win win situation. Stef -- Mariano http://marianopeck.wordpress.com -- Mariano http://marianopeck.wordpress.com |
Re: When will Fuel file format stabilize?
|
|
In reply to this post by Mariano Martinez Peck
>
> > Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again. for a serializer I think that it is worth it. I once used a backup system which used a private format for my archives and I lost them since it was discontinued.... > > > Now why in fuel it is not possible to have > FuelMaterializer load: aFile > > load using the default = current format > > > FuelMaterializer load: aFile asOfVersion: 1.4 > Of course this can be much slower but when you want to load data of the past you are just happy that this is > working. > > So would not be possible to > milestone formats > get the default working fast > get old version loading slowler but loading. > > > It sounds easy, but at least from what I can see this is very complicated. > Do you know a serializer that support this? Lets clarify with an example: > > I serialize graph X with Fuel 1.4 in Pharo 1.1. > Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. > So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ? Yes what I want to say is that if evolution is the key then you have to be explicit about milestones and version and in the end this is excellent. But of course you get a cost and effort. But this means that after that there is no more other serializer. I think that it is also a nice way to get meta about the format. Stef > > Thanks > > > Because then we get a win win situation. > Stef > > > > -- > Mariano > http://marianopeck.wordpress.com > |
Re: When will Fuel file format stabilize?
|
|
In reply to this post by Mariano Martinez Peck
> It sounds easy, but at least from what I can see this is very complicated.
> Do you know a serializer that support this? Lets clarify with an example: > > > (of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] ) of course we are smalltalkers no? :) It means that for the slow version you have to get a format representation explicit. > I serialize graph X with Fuel 1.4 in Pharo 1.1. > Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. > So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ? > > Thanks > > > Because then we get a win win situation. > Stef > > > > -- > Mariano > http://marianopeck.wordpress.com > > > > > -- > Mariano > http://marianopeck.wordpress.com > |
Re: When will Fuel file format stabilize?
|
|
In reply to this post by Schwab,Wilhelm K
On Oct 16, 2011, at 4:05 PM, Schwab,Wilhelm K wrote: > Stef, Mariano, > > There are obvious advantages to a development phase during which serialized streams are at risk. At some point, I suspect Fuel will have to take the reading of old streams as a nearly sacred obligation. Otherwise, it sounds like a cheap horror movie series: "Fuel II: Your data go in, but they don't come out." With Fuel II no zombies :) Fuel III the zombies are dead again :) > I am using humor to make a point, not to ridicule you. I read the paper, and learned a LOT. Please see my post from yesterday referring to Dolphin's #environment. You might be able to simply facade old formats into a coherent backwardly-compatible whole. Without some attention to that, there are going to be projects that will not adopt Fuel. > > Bill > > > ________________________________________ > From: [hidden email] [[hidden email]] On Behalf Of Mariano Martinez Peck [[hidden email]] > Sent: Sunday, October 16, 2011 9:03 AM > To: [hidden email] > Subject: Re: [Pharo-project] When will Fuel file format stabilize? > > On Sun, Oct 16, 2011 at 3:01 PM, Mariano Martinez Peck <[hidden email]<mailto:[hidden email]>> wrote: > > > On Sun, Oct 16, 2011 at 2:03 PM, Stéphane Ducasse <[hidden email]<mailto:[hidden email]>> wrote: > Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: > If by design a serializer takes the perspective that it will be able to load old formats and ensure > evolution all the time then you have a really cool serializer. > > Yes, I know. The problem is if it is worth it. The effort of providing such feature could be too much in compare to just load a previous version, materialize, load new version, and serialize again. > > > Now why in fuel it is not possible to have > FuelMaterializer load: aFile > > load using the default = current format > > > FuelMaterializer load: aFile asOfVersion: 1.4 > Of course this can be much slower but when you want to load data of the past you are just happy that this is > working. > > So would not be possible to > milestone formats > get the default working fast > get old version loading slowler but loading. > > > It sounds easy, but at least from what I can see this is very complicated. > Do you know a serializer that support this? Lets clarify with an example: > > > (of course, without putting all along the code IFs like (version > 2) ifTrue: [blabla] ) > > > I serialize graph X with Fuel 1.4 in Pharo 1.1. > Then I am in Pharo 1.3 with Fuel 1.7 and I want to materialize X. Fuel 1.4 doesn't load anymore in Pharo 1.3, so you have to use Fuel 1.7. > So you want to be able to take Fuel 1.7 and materialize X "using version 1.4" without even loading 1.4 (because it doesn't load) ? > > Thanks > > > Because then we get a win win situation. > Stef > > > > -- > Mariano > http://marianopeck.wordpress.com > > > > > -- > Mariano > http://marianopeck.wordpress.com > > |
Re: When will Fuel file format stabilize?
|
|
Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread.
We will keep you informed. Cheers On Sun, Oct 16, 2011 at 5:14 PM, Stéphane Ducasse <[hidden email]> wrote:
-- Mariano http://marianopeck.wordpress.com |
Re: When will Fuel file format stabilize?
|
|
My 2 cents:
i understand that people want to have backwards compatibility, so they could read even 10 years old data from files. now, it could mean a lot of effort to do that. For instance, if your binary model is spawned along 30 classes, then in order to support 2 versions of it, you need another 30 classes.. 3 versions - 90 classes, and so on. So i am thinking that a least painful way could be to create a migration or "upgrade" scripts which people can use to update their data from old format to new one, and put it in a separate package, to not burden a main development line with cruft. Then it will be like a patch system. This is how i imagine it: suppose that you having 2.0 version installed in your system, but want to load 1.6 version data. you load an additional package(s) titled like: migrating 1.6 -> 1.7 migrating 1.7 -> 1.8 migrating 1.8 -> 2.0 and then run migration scripts for your data until you get to the current version. Once you done, you can unload this stuff and be happy. And i think it is also cheaper for developers: it takes an effort to write a new migration procedure every time you changing a format, but you don't have to keep everything in one place and change a model to be compatible with everything. A migration scripts should do their work, while core model and implementation can be kept clean from cruft. -- Best regards, Igor Stasenko. |
|
|
In reply to this post by Mariano Martinez Peck
Mariano Martinez Peck wrote:
> Well...thanks guys. The point is taken. Thanks Philip, Bill and all > who gave feedback and suggestions. We are aware of this and we are > already discussing with Martin possible solutions and analyzing all > the things that has been said in this thread. > We will keep you informed. > > Cheers Just one thing I hadn't seen discussed. Is Fuel possibly a way to exchange objects/data between users/systems ? In which case it would have to deal with different systems having different versions of Fuel. cheers, Ben |
Re: When will Fuel file format stabilize?
|
|
On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <[hidden email]> wrote:
Yes In which case it would have to deal with different systems having different versions of Fuel. Exactly. It doesn't matter whether the different versions of Fuel happens in your own image or in different systems. So yes, it is the same problem. Note this is not only a problem for Fuel, but for any serializer which does not manages full backward compatibility between versions. -- Mariano http://marianopeck.wordpress.com |
Re: When will Fuel file format stabilize?
|
|
On Thu, Oct 20, 2011 at 1:00 PM, Mariano Martinez Peck <[hidden email]> wrote:
But it doesn't mean it is the best option. Maybe a XML or JSON serializer is better there...
-- Mariano http://marianopeck.wordpress.com |
Re: When will Fuel file format stabilize?
|
|
On 20 October 2011 13:01, Mariano Martinez Peck <[hidden email]> wrote:
> > > On Thu, Oct 20, 2011 at 1:00 PM, Mariano Martinez Peck > <[hidden email]> wrote: >> >> >> On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <[hidden email]> wrote: >>> >>> Mariano Martinez Peck wrote: >>>> >>>> Well...thanks guys. The point is taken. Thanks Philip, Bill and all who >>>> gave feedback and suggestions. We are aware of this and we are already >>>> discussing with Martin possible solutions and analyzing all the things that >>>> has been said in this thread. >>>> We will keep you informed. >>>> >>>> Cheers >>> >>> Just one thing I hadn't seen discussed. Is Fuel possibly a way to >>> exchange objects/data between users/systems ? >> >> Yes >> > > But it doesn't mean it is the best option. Maybe a XML or JSON serializer is > better there... > If i want speed, i can live with non-portable binary format. If i want portability, compatibility and other blah blah then yeah.. one can always use textual representation of object graph, but don't cry about being slow anymore. > >>> >>> In which case it would have to deal with different systems having >>> different versions of Fuel. >>> >> >> Exactly. It doesn't matter whether the different versions of Fuel happens >> in your own image or in different systems. So yes, it is the same problem. >> Note this is not only a problem for Fuel, but for any serializer which >> does not manages full backward compatibility between versions. >> >> >> -- >> Mariano >> http://marianopeck.wordpress.com >> > > > > -- > Mariano > http://marianopeck.wordpress.com > > -- Best regards, Igor Stasenko. |
Re: When will Fuel file format stabilize?
|
|
In reply to this post by Mariano Martinez Peck
Mariano,
I gave you a pretty tough time :) It appears that you understand that I did so with the best of intentions. Thanks for listening. One problem that I do *not* think you need to solve is that of old software talking to new software. Someone wanting forward compatibility is probably going to use XML or something similar to get a timeless protocol that will be of necessity very simple. A good serializer is hard to write, and forward-compatibility is probably too much to ask, particularly when efficiency is a stated goal. A backwardly-compatible Fuel will be most welcome and useful. Bill ________________________________________ From: [hidden email] [[hidden email]] On Behalf Of Mariano Martinez Peck [[hidden email]] Sent: Thursday, October 20, 2011 7:00 AM To: [hidden email] Subject: Re: [Pharo-project] When will Fuel file format stabilize? On Thu, Oct 20, 2011 at 12:54 PM, Ben Coman <[hidden email]<mailto:[hidden email]>> wrote: Mariano Martinez Peck wrote: Well...thanks guys. The point is taken. Thanks Philip, Bill and all who gave feedback and suggestions. We are aware of this and we are already discussing with Martin possible solutions and analyzing all the things that has been said in this thread. We will keep you informed. Cheers Just one thing I hadn't seen discussed. Is Fuel possibly a way to exchange objects/data between users/systems ? Yes In which case it would have to deal with different systems having different versions of Fuel. Exactly. It doesn't matter whether the different versions of Fuel happens in your own image or in different systems. So yes, it is the same problem. Note this is not only a problem for Fuel, but for any serializer which does not manages full backward compatibility between versions. -- Mariano http://marianopeck.wordpress.com |
Re: When will Fuel file format stabilize?
|
|
Hi I followed with interest the discussion but without many interventions. I want to explain an idea I have for supporting backward-compatibility without neither stabilizing the format nor getting the current code dirty. Maybe it's a bit crazy but I think it can be a practical solution. I would like to know your opinions.
We can offer: - Maintain a *compatibility package* for each released version, that can co-exist in the same image. For example Fuel1_5, Fuel1_6, etc. Obviously, the current version always has the nice clean name (Fuel).
- Provide a facade that works like "Fuel load: aFile asOfVersion: '1.4'" who interacts with the compatibility packages.
To generate each *compatibility package*, we rename several elements of the original Fuel released package, this way:
* Extension methods: Append at beggining of each extension method selector a prefix. For example: #fuelAccept: --> #fuel1_6_fuelAccept:
* Classes: Append to each one some version identifier. For example: FLSerializer --> FLSerializer_1_6
* Methods: Fix each reference to a class or extension method selector THAT BELONGS to the package. For example: "FLSerializer new" --> "FLSerializer_1_6 new". I think it could be not that hard to make such renames automatically with a small tool. Then, we can simultaneously maintain each *compatibility package* (Fuel1_5, Fuel1_6, etc.) when Pharo evolves: Our Jenkins will run tests for all the compatibility Fuel packages. For example... Does Fuel1_7 stop working in Pharo1.5? then (if it's possible) we fix it, save the monticello version and mark it in our ConfigurationOfFuel as "this is the stable monticello version of fuel 1.7 in pharo 1.5".
Now that I see how many lines I took to explain it, I think it's a bit insane. I hope to read some opinions!
Bests, MartínOn Thu, Oct 20, 2011 at 6:00 PM, Schwab,Wilhelm K <[hidden email]> wrote: Mariano, |
Re: When will Fuel file format stabilize?
|
|
In reply to this post by Stéphane Ducasse
On 10/16/2011 02:03 PM, Stéphane Ducasse wrote:
> Mariano I think that the point that Philippe (our national butcher) is trying to make is the following: > If by design a serializer takes the perspective that it will be able to load old formats and ensure > evolution all the time then you have a really cool serializer. > > Now why in fuel it is not possible to have > FuelMaterializer load: aFile > > load using the default = current format > > > FuelMaterializer load: aFile asOfVersion: 1.4 > Of course this can be much slower but when you want to load data of the past you are just happy that this is > working. Not exactly but almost. Put the version in the format and automatically load it using the right class (it has already been written, just don't delete it). This also allows to find out whether the format is too new. Not being able to serialize in an older format would be fine as long as you can still load them. Cheers Philippe |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |