Smalltalk › Gemtalk › GLASS

Class Migration

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

12 messages Options

GLASS mailing list

Class Migration

Ciao,

i need to manage the 'migrations' of class and relative instances.

I read the Prog-guide3-1 Class Creation, Versions, and Instance Migration .

Now my question is how to manage the migration when i development in Pharo,

and load the new structure into GLASS with the relative package using GemTools or Tode.

I do some test with GemTools migrate option and filetree packages load.

Question:

someone implemented a support into Pharo to handle these problematic

and porting the 'migration' into Gemstone?

I think of something that when I change classes in Pharo,

record - define the statements that then allow gemstone to applying migration.

Thanks for any considerations,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

Hello Dario,

When your database is small, automatic migrations will be fine. But we like to control when and how the migrations take place, because we like to take the site down so that we do not have active users on the system while migrations take place. I don't know how it's possible anyway to be sure how your system will behave when migrations have run partially (you then have the situation with some objects migrated and others not: how will the system behave?)

We often override the method #migrateFrom:instVarMap: to initialize new instVars or to transform some instVars that have been renamed or changed structure. What is important is that the you clean up these methods after migration because it may cause harm if you migrate the same class again. You can also write the migrateFrom:instVarMap: in a way that it is idempotent, for example:

migrateFrom: oldObject instVarMap: aMap

super migrateFrom: oldObject instVarMap: aMap.

someInstVar ifNil: [ someInstVar := OrderedCollection new ]

We write this code in Pharo and put it in an extension package. This code obviously only runs in GemStone.

To invoke migrations, we have written a class that does the work, similar to GsDeployer, which is what I think you can start with. If you want to use some of the code we've written, you can just say.

There are several other issues to deal with when the system becomes complex, some are:

- Fixing data problems before doing the migration

- Renaming classes

- Controlling the order of migrating some classes before others

- Ensuring that all instances are migrated when you're done

Let me know if this helps or if you need more information.

Cheers

Otto

On Sat, Jan 30, 2016 at 5:24 PM, Trussardi Dario Romano via Glass <[hidden email]> wrote:

Ciao,

i need to manage the 'migrations' of class and relative instances.

I read the Prog-guide3-1 Class Creation, Versions, and Instance Migration .

Now my question is how to manage the migration when i development in Pharo,

and load the new structure into GLASS with the relative package using GemTools or Tode.

I do some test with GemTools migrate option and filetree packages load.

Question:

someone implemented a support into Pharo to handle these problematic

and porting the 'migration' into Gemstone?

I think of something that when I change classes in Pharo,

record - define the statements that then allow gemstone to applying migration.

Thanks for any considerations,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

Ciao Otto,

thanks for your considerations.

Hello Dario,

When your database is small, automatic migrations will be fine. But we like to control when and how the migrations take place, because we like to take the site down so that we do not have active users on the system while migrations take place. I don't know how it's possible anyway to be sure how your system will behave when migrations have run partially (you then have the situation with some objects migrated and others not: how will the system behave?)

We often override the method #migrateFrom:instVarMap: to initialize new instVars or to transform some instVars that have been renamed or changed structure. What is important is that the you clean up these methods after migration because it may cause harm if you migrate the same class again. You can also write the migrateFrom:instVarMap: in a way that it is idempotent, for example:

What you intend?

This is not the default implementation of: migrateFrom: oldObject instVarMap: aMap

migrateFrom: oldObject instVarMap: aMap
super migrateFrom: oldObject instVarMap: aMap.
someInstVar ifNil: [ someInstVar := OrderedCollection new ]

We write this code in Pharo and put it in an extension package. This code obviously only runs in GemStone.

To invoke migrations, we have written a class that does the work, similar to GsDeployer, which is what I think you can start with. If you want to use some of the code we've written, you can just say.

O yes i'm very interested.

I read and test the GsDeployer support on GLASS 3.1.0.6. it's very interesting based.

Now i need to understand and work for implement a support to integrate it with Pharo development.

I think your experience is very good and if you can share your works, i can start with some value added.

Thanks,

Dario

There are several other issues to deal with when the system becomes complex, some are:
- Fixing data problems before doing the migration
- Renaming classes
- Controlling the order of migrating some classes before others
- Ensuring that all instances are migrated when you're done

Let me know if this helps or if you need more information.

Cheers
Otto

On Sat, Jan 30, 2016 at 5:24 PM, Trussardi Dario Romano via Glass <[hidden email]> wrote:
Ciao,

i need to manage the 'migrations' of class and relative instances.

I read the Prog-guide3-1 Class Creation, Versions, and Instance Migration .

Now my question is how to manage the migration when i development in Pharo,

and load the new structure into GLASS with the relative package using GemTools or Tode.

I do some test with GemTools migrate option and filetree packages load.

Question:

someone implemented a support into Pharo to handle these problematic

and porting the 'migration' into Gemstone?

I think of something that when I change classes in Pharo,

record - define the statements that then allow gemstone to applying migration.

Thanks for any considerations,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

Sorry, did not attach the package in the previous mail.

On Mon, Feb 1, 2016 at 2:52 PM, Otto Behrens <[hidden email]> wrote:

We often override the method #migrateFrom:instVarMap: to initialize new instVars or to transform some instVars that have been renamed or changed structure. What is important is that the you clean up these methods after migration because it may cause harm if you migrate the same class again. You can also write the migrateFrom:instVarMap: in a way that it is idempotent, for example:

What you intend?

This is not the default implementation of: migrateFrom: oldObject instVarMap: aMap

When you migrate objects where you changed the instVars and you want to do something special with each object (like initializing the new instVar as in the example I gave you below), in this case, you override migrateFrom:instVarMap:

migrateFrom: oldObject instVarMap: aMap
super migrateFrom: oldObject instVarMap: aMap.
someInstVar ifNil: [ someInstVar := OrderedCollection new ]

O yes i'm very interested.

I attach a package that I extracted from our code (into a new package). You can load this package and call

Migrator new packageNames: #(); migrateAll

Now i need to understand and work for implement a support to integrate it with Pharo development.

You can load this same package in Pharo. But, there are some classes like System that you need defined. GemStone has a "Grease" package that you can load in Pharo. I do not know what is its name. We just have our own package where we define classes such as System and Metaclass3. And this package we only load in Pharo as this Migrator code will never run in Pharo.

I think your experience is very good and if you can share your works, i can start with some value added.

Hope this helps. Let me know where are some gaps.

Thanks,

Dario

There are several other issues to deal with when the system becomes complex, some are:
- Fixing data problems before doing the migration
- Renaming classes
- Controlling the order of migrating some classes before others
- Ensuring that all instances are migrated when you're done

Let me know if this helps or if you need more information.

Cheers
Otto

On Sat, Jan 30, 2016 at 5:24 PM, Trussardi Dario Romano via Glass <[hidden email]> wrote:
Ciao,

i need to manage the 'migrations' of class and relative instances.

I read the Prog-guide3-1 Class Creation, Versions, and Instance Migration .

Now my question is how to manage the migration when i development in Pharo,

and load the new structure into GLASS with the relative package using GemTools or Tode.

I do some test with GemTools migrate option and filetree packages load.

Question:

someone implemented a support into Pharo to handle these problematic

and porting the 'migration' into Gemstone?

I think of something that when I change classes in Pharo,

record - define the statements that then allow gemstone to applying migration.

Thanks for any considerations,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Wonka-Migrations-Core-otto.12.mcz (14K) Download Attachment

GLASS mailing list

Re: Class Migration

In reply to this post by GLASS mailing list

On 01/31/2016 10:24 PM, Otto Behrens via Glass wrote:
> But we like to control when and how the migrations take place, because
> we like to take the site down so that we do not have active users on the
> system while migrations take place. I don't know how it's possible
> anyway to be sure how your system will behave when migrations have run
> partially (you then have the situation with some objects migrated and
> others not: how will the system behave?)

If you have the flexibility to take the site down while migrating, that
is easier. For applications that cannot incur the downtime, migration is
usually a two-phase process.

In a two-phase application update, the first phase modifies the
structure of the objects, but leaves the behavior the same. This allows
the system to behave predictably when some objects are migrated and
others are not. Once all objects have their structure migrated, a second
software update with no structural changes brings in the behavior that
needed the structural change. The second phase can be done as a single
commit, so the cutover to the new application version is essentially all
at once.

This takes more careful coding and is more work, but it does enable 24/7
uptime applications to evolve.

Regards,

-Martin
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

Thanks, that's interesting.

> If you have the flexibility to take the site down while migrating, that
> is easier. For applications that cannot incur the downtime, migration is
> usually a two-phase process.

Fortunately, we've always had a window to take the site down.

> In a two-phase application update, the first phase modifies the
> structure of the objects, but leaves the behavior the same. This allows
> the system to behave predictably when some objects are migrated and
> others are not.

Your code cannot be totally ignorant of new structures, so perhaps
you'll have to write some code that can handle both old and new
structures? But you can do that in one commit though.

> Once all objects have their structure migrated, a second
> software update with no structural changes brings in the behavior that
> needed the structural change. The second phase can be done as a single
> commit, so the cutover to the new application version is essentially all
> at once.

A seaside application using stuff like "commitWhenAlmostOutOfMemory"
would certainly not work then.

> This takes more careful coding and is more work, but it does enable 24/7
> uptime applications to evolve.

I can imagine! A bit more careful planning too.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

On 02/01/2016 08:36 PM, Otto Behrens wrote:

> Thanks, that's interesting.
>
>> If you have the flexibility to take the site down while migrating, that
>> is easier. For applications that cannot incur the downtime, migration is
>> usually a two-phase process.
>
> Fortunately, we've always had a window to take the site down.
>
>> In a two-phase application update, the first phase modifies the
>> structure of the objects, but leaves the behavior the same. This allows
>> the system to behave predictably when some objects are migrated and
>> others are not.
>
> Your code cannot be totally ignorant of new structures, so perhaps
> you'll have to write some code that can handle both old and new
> structures? But you can do that in one commit though.

The old class version and the new class version are different classes,
and so can have different code. The important thing is to make the
external behavior the same in the first phase, even though the structure
is different.

>
>> Once all objects have their structure migrated, a second
>> software update with no structural changes brings in the behavior that
>> needed the structural change. The second phase can be done as a single
>> commit, so the cutover to the new application version is essentially all
>> at once.
>
> A seaside application using stuff like "commitWhenAlmostOutOfMemory"
> would certainly not work then.

You update the code with a single commit. Each production gem then sees
that change at its individual next transaction boundary. Because it's
all transactional, no gem can see changes committed by a gem that's
using the new code until it is also using the new code.

You can have a gem that's using old code commit (once) after some other
gem(s) have committed using new code, but the normal prevention of
commit conflicts is usually enough to prevent this from being a problem.

>
>> This takes more careful coding and is more work, but it does enable 24/7
>> uptime applications to evolve.
>
> I can imagine! A bit more careful planning too.

Indeed. You have to think through your changes, and what the
transitional states will be.

Regards,

-Martin

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

In reply to this post by GLASS mailing list

On 01/31/2016 10:24 PM, Otto Behrens via Glass wrote:

> Hello Dario,
>
> When your database is small, automatic migrations will be fine. But we
> like to control when and how the migrations take place, because we
> like to take the site down so that we do not have active users on the
> system while migrations take place. I don't know how it's possible
> anyway to be sure how your system will behave when migrations have run
> partially (you then have the situation with some objects migrated and
> others not: how will the system behave?)
>
> We often override the method #migrateFrom:instVarMap: to initialize
> new instVars or to transform some instVars that have been renamed or
> changed structure. What is important is that the you clean up these
> methods after migration because it may cause harm if you migrate the
> same class again. You can also write the migrateFrom:instVarMap: in a
> way that it is idempotent, for example:
>
> migrateFrom: oldObject instVarMap: aMap
> super migrateFrom: oldObject instVarMap: aMap.
> someInstVar ifNil: [ someInstVar := OrderedCollection new ]
>
> We write this code in Pharo and put it in an extension package. This
> code obviously only runs in GemStone.
>
> To invoke migrations, we have written a class that does the work,
> similar to GsDeployer, which is what I think you can start with. If
> you want to use some of the code we've written, you can just say.

Would it be possible for you to shre your variant of GsDeployer? It
sounds like it has some useful functionality ...

Dale
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

> Would it be possible for you to shre your variant of GsDeployer? It sounds
> like it has some useful functionality ...

I did in a previous post on this thread. I'll attach it again anyway.
It has some nice features, I think. If you want, I can point some out.
Let me know. The package version number is random.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Wonka-Migrations-Core-otto.4283.mcz (13K) Download Attachment

GLASS mailing list

Re: Class Migration

Haha, I saw it after I sent the post ... thanks ... as always I have
multiple balls in the air, but I will keep your work in mind for
integration when I have a chance ...

Dale

On 02/02/2016 10:08 AM, Otto Behrens wrote:
>> Would it be possible for you to shre your variant of GsDeployer? It sounds
>> like it has some useful functionality ...
> I did in a previous post on this thread. I'll attach it again anyway.
> It has some nice features, I think. If you want, I can point some out.
> Let me know. The package version number is random.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

I would benefit also from better migration tools. But I won't be able to dig into it until I face the problem.

So I will keep this in mind :)

Thanks Otto,

On Tue, Feb 2, 2016 at 3:10 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Haha, I saw it after I sent the post ... thanks ... as always I have multiple balls in the air, but I will keep your work in mind for integration when I have a chance ...

Dale

On 02/02/2016 10:08 AM, Otto Behrens wrote:

Would it be possible for you to shre your variant of GsDeployer? It sounds
like it has some useful functionality ...

I did in a previous post on this thread. I'll attach it again anyway.
It has some nice features, I think. If you want, I can point some out.
Let me know. The package version number is random.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

GLASS mailing list

Re: Class Migration

In reply to this post by GLASS mailing list

Ciao,

thanks Otto for your work.

It's very interesting.

Now i have a question:

I defined the class Animal and the subClass AnimalPlus

I have some Animal class version ( 585 the current ) ( 897 & 209 old version ) with relative Metaclasses

and AnimalPlus ( 769 ) class define the:

classNamesToRenameFrom

^ #( Animal )

When i do the :

Migrator new packageNames: #( 'DTRMigrateTesting' ); migrateAll

the log report:

Warning replacing migration to Animal ( 585) in preference of renaming from Animal (897) to AnimalPlus (769)

Warning replacing migration to Animal ( 585) in preference of renaming from Animal (209) to AnimalPlus (769)

I think to solve this warning removing the old Animal class version ( 897 ) and ( 209 ) and the relative metaclasses

but when i do the relative Animal ( 897) removeFromSystem i get false.

Any considerations ?

Another relative question:

because when do: removeAndCleanUpEmptyInstances

you remove self removeAllMethodsOfInvisibleClass: fromClass.

but don't remove the old class and relative metaclass ?

Thanks,

Dario

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass