Continuous integration server for GLASS

At ESUG, several of us talked about the desire to have a continuous integration server for GLASS ... the basic idea was that the GLASS CI server would produce prebuilt/tested extents instead of images:

  extent0.glass.dbf
  extent0.seaside30.dbf
  extent0.pier2.dbf.
  etc.

I've talked with folks back here about having VMware host a GLASS CI server and it looks like we should be able to do it ... Monty has been working with Jenkins to put together an internal CI server for MagLev so he's familiar with the nitty gritty details ...

I recall that I was in favor of fixed metacello configurations being used to drive the contents, but that 'you guys' preferred a somewhat looser process.

'You guys' are the customers and the CI server should produce artifacts that 'you guys' will find useful, so I suppose it is time to start hashing out the details.


Dale

I'd be glad to hear what your needs are. I'll see if there is a way we can setup Jenkins to produce the artifacts you need and copy them to a place you can obtain them.

I'd prefer sticking with GemStone/S 3.x as I don't have any CI setup for GS/S 2.4. I hope that would work for you.

-- Monty

----- Original Message -----
From: "Dale Henrichs" <[hidden email]>
To: "GemStone Seaside beta discussion" <[hidden email]>
Sent: Tuesday, September 6, 2011 12:11:31 PM
Subject: [GS/SS Beta] Continuous integration server for GLASS

At ESUG, several of us talked about the desire to have a continuous integration server for GLASS ... the basic idea was that the GLASS CI server would produce prebuilt/tested extents instead of images:

  extent0.glass.dbf
  extent0.seaside30.dbf
  extent0.pier2.dbf.
  etc.

I've talked with folks back here about having VMware host a GLASS CI server and it looks like we should be able to do it ... Monty has been working with Jenkins to put together an internal CI server for MagLev so he's familiar with the nitty gritty details ...

I recall that I was in favor of fixed metacello configurations being used to drive the contents, but that 'you guys' preferred a somewhat looser process.

'You guys' are the customers and the CI server should produce artifacts that 'you guys' will find useful, so I suppose it is time to start hashing out the details.


Dale

Am 07.09.2011 um 01:47 schrieb Monty Williams:

> I'd be glad to hear what your needs are. I'll see if there is a way we can setup Jenkins to produce the artifacts you need and copy them to a place you can obtain them.
>
> I'd prefer sticking with GemStone/S 3.x as I don't have any CI setup for GS/S 2.4. I hope that would work for you.
>
>
Monty,

please consider to have a look at the tools I made. I would be glad if we could join some forces. I'm really interested to improve the existing tools so they might fit your needs. The tools are:

stone creator: creates a new stone by creating a directory skeleton and environment where a stone can be started. The ci server uses this for every build. The stone name is [jenkins-job-name][build number]
Repository: https://gitorious.org/gemstone-tools/stone-creator

gsbuilder: Is a small shell script that works like Lukas' builder script but for gemstone. It uses stone creator to integrate gemstone into jenkins.
Repository: https://gitorious.org/gemstone-tools/gsbuilder

I can help improving or do the necessary adjustments for different shells if needed. I would also be glad if some of the scripts could be just integrated in the gemstone distribution so they can evolve and it would be easy for people to use.

As you know I'm working on a chef recipes to build something similar to the virtual appliance. My vision would be that it is really easy to deploy a gemstone test server, CI server and production server with chef.

Norbert

Hi Dale,

On 07/09/11 4:38 AM, Norbert Hartl wrote:
>
> As you know I'm working on a chef recipes to build something similar
> to the virtual appliance. My vision would be that it is really easy
> to deploy a gemstone test server, CI server and production server
> with chef.

How does a CI built GemStone image/extent fit into the production
process? Does the production data have to be migrated into each deployment?

Am 07.09.2011 um 16:27 schrieb Yanni Chiu:

> On 07/09/11 4:38 AM, Norbert Hartl wrote:
>>
>> As you know I'm working on a chef recipes to build something similar
>> to the virtual appliance. My vision would be that it is really easy
>> to deploy a gemstone test server, CI server and production server
>> with chef.
>
> How does a CI built GemStone image/extent fit into the production process? Does the production data have to be migrated into each deployment?


What I meant above is the deployment of the virtual machines. With chef I want to create a test machine, a CI machine and a production machine. And with this I mean the installation and configuration of those machines.

Apart from machines towards deployment: In my case the CI server is part of the deplyoment scenario. I think smalltalk and especially gemstone are great because you have a few alternatives to choose when doing deployment. My workflow for the deployment is the following

- develop in pharo, commit monticello package to server
- jenkins is triggered if a new package is saved and builds the app for gemstone and pharo, running tests, lint and test coverage
- the last succesful build is stored (default hudson option)
- on upgrade I serialize my data with Sixx, rsync the last succesful built extent to the production machine (just skipping the test server for now)
- then I import the serialized data and everything is upgraded.

That workflow should always work and covers some of the emergency cases. Apart from that I do log into the production image and load packages manually/use metacello if:

- I need to migrate classes.
- I have minor fixes and I don't want to have a downtime

that is how I do these things.

Norbert

Norbert,

Yanni is referring to the fact that for many GemStone installations it isn't practical to serialize/deserialize as part of the upgrade process. I suppose that that is my mindset as well, which is why I am more interested in validating a particular configuration than producing an extent ....

I suppose the reality is that for certain apps it can be easier to serialize/deserialize than migrate.

Being able to produce up-to-date appliances is definitely another possible artifact for the CI server to produce ...

Dale

----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Wednesday, September 7, 2011 7:53:21 AM
| Subject: Re: [GS/SS Beta] Continuous integration server for GLASS
|
|
| Am 07.09.2011 um 16:27 schrieb Yanni Chiu:
|
| > On 07/09/11 4:38 AM, Norbert Hartl wrote:
| >>
| >> As you know I'm working on a chef recipes to build something
| >> similar
| >> to the virtual appliance. My vision would be that it is really
| >> easy
| >> to deploy a gemstone test server, CI server and production server
| >> with chef.
| >
| > How does a CI built GemStone image/extent fit into the production
| > process? Does the production data have to be migrated into each
| > deployment?
|
|
| What I meant above is the deployment of the virtual machines. With
| chef I want to create a test machine, a CI machine and a production
| machine. And with this I mean the installation and configuration of
| those machines.
|
| Apart from machines towards deployment: In my case the CI server is
| part of the deplyoment scenario. I think smalltalk and especially
| gemstone are great because you have a few alternatives to choose
| when doing deployment. My workflow for the deployment is the
| following
|
| - develop in pharo, commit monticello package to server
| - jenkins is triggered if a new package is saved and builds the app
| for gemstone and pharo, running tests, lint and test coverage
| - the last succesful build is stored (default hudson option)
| - on upgrade I serialize my data with Sixx, rsync the last succesful
| built extent to the production machine (just skipping the test
| server for now)
| - then I import the serialized data and everything is upgraded.
|
| That workflow should always work and covers some of the emergency
| cases. Apart from that I do log into the production image and load
| packages manually/use metacello if:
|
| - I need to migrate classes.
| - I have minor fixes and I don't want to have a downtime
|
| that is how I do these things.
|
| Norbert
|
|
|

Am 07.09.2011 um 17:34 schrieb Dale Henrichs:

> Norbert,
>
> Yanni is referring to the fact that for many GemStone installations it isn't practical to serialize/deserialize as part of the upgrade process. I suppose that that is my mindset as well, which is why I am more interested in validating a particular configuration than producing an extent ....

It is my mind set as well. But both scenarios are feasible so I need both. I have class shape changes that are easy to handle by live migration. I just don't want to face a situation like having my app in seaside2.8 and like to upgrade to 3.0 without having a serialize/deserialize approach at hand. That is all.
>
> I suppose the reality is that for certain apps it can be easier to serialize/deserialize than migrate.
>
> Being able to produce up-to-date appliances is definitely another possible artifact for the CI server to produce ...

That is what I'm working at right now. I just discovered that in Gemstone3.0beta4 the extent0.seaside.dbf is missing seaside somehow. I hope I didn't do anything wrong but I couldn't start a web server gem because seaside seems to be missing from the extent. Apart from that I have a basic draft appliance. Well, it needs a lot more energy but I hope within two or three weeks I have something usable.

If you like to watch or participate, have a look at https://github.com/noha/virtual-gemstone

Norbert

Norbert,

Agreed that both scenarios are useful.

I probably didn't have Seaside functional in Gemstone3.0beta4. Seaside runs in GemStone3.0, but there are a handful of issues that need to be resolved. We are in the process of planning the contents (not necessarily schedule:) for 3.0.1 which will hopefully contain the set of Seaside fixes that we need ...

Dale


----- Original Message -----
| From: "Norbert Hartl" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Wednesday, September 7, 2011 8:49:44 AM
| Subject: Re: [GS/SS Beta] Continuous integration server for GLASS
|
|
| Am 07.09.2011 um 17:34 schrieb Dale Henrichs:
|
| > Norbert,
| >
| > Yanni is referring to the fact that for many GemStone installations
| > it isn't practical to serialize/deserialize as part of the upgrade
| > process. I suppose that that is my mindset as well, which is why I
| > am more interested in validating a particular configuration than
| > producing an extent ....
|
| It is my mind set as well. But both scenarios are feasible so I need
| both. I have class shape changes that are easy to handle by live
| migration. I just don't want to face a situation like having my app
| in seaside2.8 and like to upgrade to 3.0 without having a
| serialize/deserialize approach at hand. That is all.
| >
| > I suppose the reality is that for certain apps it can be easier to
| > serialize/deserialize than migrate.
| >
| > Being able to produce up-to-date appliances is definitely another
| > possible artifact for the CI server to produce ...
|
| That is what I'm working at right now. I just discovered that in
| Gemstone3.0beta4 the extent0.seaside.dbf is missing seaside somehow.
| I hope I didn't do anything wrong but I couldn't start a web server
| gem because seaside seems to be missing from the extent. Apart from
| that I have a basic draft appliance. Well, it needs a lot more
| energy but I hope within two or three weeks I have something usable.
|
| If you like to watch or participate, have a look at
| https://github.com/noha/virtual-gemstone
|
| Norbert
|
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "Norbert Hartl" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Wednesday, September 7, 2011 7:53:21 AM
| > | Subject: Re: [GS/SS Beta] Continuous integration server for GLASS
| > |
| > |
| > | Am 07.09.2011 um 16:27 schrieb Yanni Chiu:
| > |
| > | > On 07/09/11 4:38 AM, Norbert Hartl wrote:
| > | >>
| > | >> As you know I'm working on a chef recipes to build something
| > | >> similar
| > | >> to the virtual appliance. My vision would be that it is really
| > | >> easy
| > | >> to deploy a gemstone test server, CI server and production
| > | >> server
| > | >> with chef.
| > | >
| > | > How does a CI built GemStone image/extent fit into the
| > | > production
| > | > process? Does the production data have to be migrated into each
| > | > deployment?
| > |
| > |
| > | What I meant above is the deployment of the virtual machines.
| > | With
| > | chef I want to create a test machine, a CI machine and a
| > | production
| > | machine. And with this I mean the installation and configuration
| > | of
| > | those machines.
| > |
| > | Apart from machines towards deployment: In my case the CI server
| > | is
| > | part of the deplyoment scenario. I think smalltalk and especially
| > | gemstone are great because you have a few alternatives to choose
| > | when doing deployment. My workflow for the deployment is the
| > | following
| > |
| > | - develop in pharo, commit monticello package to server
| > | - jenkins is triggered if a new package is saved and builds the
| > | app
| > | for gemstone and pharo, running tests, lint and test coverage
| > | - the last succesful build is stored (default hudson option)
| > | - on upgrade I serialize my data with Sixx, rsync the last
| > | succesful
| > | built extent to the production machine (just skipping the test
| > | server for now)
| > | - then I import the serialized data and everything is upgraded.
| > |
| > | That workflow should always work and covers some of the emergency
| > | cases. Apart from that I do log into the production image and
| > | load
| > | packages manually/use metacello if:
| > |
| > | - I need to migrate classes.
| > | - I have minor fixes and I don't want to have a downtime
| > |
| > | that is how I do these things.
| > |
| > | Norbert
| > |
| > |
| > |
|
|

Nick and company,

Agreed that we can't imagine running tests on all possible configurations of the configurations...

What combinations of products would be of interest to y'all?

Dale
----- Original Message -----
| From: "Nick Ager" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Wednesday, September 7, 2011 2:24:56 AM
| Subject: Re: [GS/SS Beta] Continuous integration server for GLASS
|
| Hi Dale,
|
|
| On 6 September 2011 20:11, Dale Henrichs < [hidden email] >
| wrote:
|
|
| At ESUG, several of us talked about the desire to have a continuous
| integration server for GLASS ... the basic idea was that the GLASS
| CI server would produce prebuilt/tested extents instead of images:
|
| extent0.glass.dbf
| extent0.seaside30.dbf
| extent0.pier2.dbf.
| etc.
|
| I've talked with folks back here about having VMware host a GLASS CI
| server and it looks like we should be able to do it ... Monty has
| been working with Jenkins to put together an internal CI server for
| MagLev so he's familiar with the nitty gritty details ...
|
| I recall that I was in favor of fixed metacello configurations being
| used to drive the contents, but that 'you guys' preferred a somewhat
| looser process.
|
|
|
| I remember the conversation where we discussed using metacello
| configurations to build all the possible outcomes of the metacello
| configurations, based on the groups defined within the
| configurations. This seemed like a great idea, though potentially a
| lot of work, especially when we considered supporting branched
| configurations.
| I guess the more pragmatic approach is to manually set up a number of
| standard builds e.g. Seaside (in various versions) Pier etc and let
| us add any additional configurations we're using. Hopefully this
| will allow us to feel confident to make GLASS releases, rather than
| having to rely on you to do it all.
|
|
| Nick

Hi Norbert,

Thanks! I'll gladly look at the tools you made. I don't have time or patience to reinvent the wheel.

A agree that having Chef recipes would be useful.

-- Monty

----- Original Message -----
From: "Norbert Hartl" <[hidden email]>
To: "GemStone Seaside beta discussion" <[hidden email]>
Sent: Wednesday, September 7, 2011 1:38:27 AM
Subject: Re: [GS/SS Beta] Continuous integration server for GLASS


Am 07.09.2011 um 01:47 schrieb Monty Williams:

> I'd be glad to hear what your needs are. I'll see if there is a way we can setup Jenkins to produce the artifacts you need and copy them to a place you can obtain them.
>
> I'd prefer sticking with GemStone/S 3.x as I don't have any CI setup for GS/S 2.4. I hope that would work for you.
>
>
Monty,

please consider to have a look at the tools I made. I would be glad if we could join some forces. I'm really interested to improve the existing tools so they might fit your needs. The tools are:

stone creator: creates a new stone by creating a directory skeleton and environment where a stone can be started. The ci server uses this for every build. The stone name is [jenkins-job-name][build number]
Repository: https://gitorious.org/gemstone-tools/stone-creator

gsbuilder: Is a small shell script that works like Lukas' builder script but for gemstone. It uses stone creator to integrate gemstone into jenkins.
Repository: https://gitorious.org/gemstone-tools/gsbuilder

I can help improving or do the necessary adjustments for different shells if needed. I would also be glad if some of the scripts could be just integrated in the gemstone distribution so they can evolve and it would be easy for people to use.

As you know I'm working on a chef recipes to build something similar to the virtual appliance. My vision would be that it is really easy to deploy a gemstone test server, CI server and production server with chef.

Norbert

Id be interested to see what tools you have for updating the appliance itself, before doing anything with gemstone.  being able to "build" a vm from a starting point or set of parameters would be very useful.

Hi Dale,

Nick,

This is a good list ... seaside + pier covers a pretty wide range of the GemStone packages so it sounds like a good place to start ...

Any other comments or ideas?

Dale

----- Original Message -----
| From: "Nick Ager" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Friday, September 9, 2011 7:28:36 AM
| Subject: Re: [GS/SS Beta] Continuous integration server for GLASS
|
| Hi Dale,
|
|
|
|
| Agreed that we can't imagine running tests on all possible
| configurations of the configurations...
|
| What combinations of products would be of interest to y'all?
|
|
|
| At a minimum it would be great if the Gemstone CI server could build:
| 1) latest released version of Seaside on the latest released version
| of GLASS
| 2) latest released version of Seaside on the latest GLASS packages
| 3) latest Seaside packages on the latest released version of GLASS
| 4) latest Seaside packages on the latest GLASS packages.
|
|
|
|
| 1) will only require building when there is a new release, but will
| provide a stable base-line for understanding changes
|
|
| From my perspective it would be good to also build Pier with the set
| of default add-on on top of the above four configurations making
| eight in total. Pier also provides a test of Seaside, ensuring that
| something hasn't broken below it.
|
|
| My 2c
|
|
| Nick

On 09/09/2011 12:58 PM, Dale Henrichs wrote:
> Any other comments or ideas?
>
> Dale

Do you expect to include Zinc in the future versions of GLASS?

Paul,

Yes. I've included Zinc in the Seaside 3.0.6 configuration and the Pharo and GemStone Zinc adaptors are included.

On Pharo, Zinc can cause issues on image startup, so it was decided to make Zinc optional (i.e., not loaded by default).

On GemStone, there are still a couple of overrides and they cause some of the tests to fail, so more work needs to be done on Zinc ... there are also about 10+ unresolved symbols in the code ... Presumably Zinc is working for most of the use cases in GemStone and Pharo, but with overrides, tests failing and unresolved symbols I don't feel comfortable, yet - I don't like ignoring failing tests no matter how benign.

If you want to load Zinc into Seaside 3.0.6 you need to execute the following expression:

  (ConfigurationOfSeaside30 project version: '3.0.6') load:'Zinc-Seaside'

and the Zinc and the correct adaptors will be loaded in both Pharo and GemStone. BTW, Seaside 3.0.6 should be released very soon now.

With all of the said, Zinc is on my sorta short list of "things to do real soon now" and the existing issues are more of a matter of cleanup than fundamental problems...

Dale

----- Original Message -----
| From: "Paul DeBruicker" <[hidden email]>
| To: [hidden email]
| Sent: Friday, September 9, 2011 1:34:12 PM
| Subject: Re: [GS/SS Beta] Continuous integration server for GLASS
|
| On 09/09/2011 12:58 PM, Dale Henrichs wrote:
| > Any other comments or ideas?
| >
| > Dale
|
| Do you expect to include Zinc in the future versions of GLASS?
|