Automated VM build server
Locked
 
|
In our nearest plans is make an automated (Hudson based) build setup
for Cog and Squeak VMs. All files, which used for building an artifacts should be included in source code repository. So, most probably i will add the 'hudson-build' dir at same level as 'platforms', where scripts and configs will be stored. Some changes which i going to do: - all generated code will be removed from version controlled tree. Instead of it i will add an information, which will be enough to reproduce the generated code by taking 3 separate config file(s) a) image version (or its URL(s) for download) b) script for loading VMMaker (of concrete version) + all addons like FFI , FT2 etc etc into image c) script for generating the source code using VMMaker image .. z) tell me if i miss anything here With removing generated source code from repo we now can make sure that any changes in VMMaker (including code generation itself) can be tested and can be reproduced by build server at any snapshot version. Also, we could test & see if VMMaker loads well and can generate code using different base images. So, we will instantly know if any changes in compiler, or other layers of system is breaking VMMaker. Because having a years old image where VMMaker works 100%, but at same time have 100% chances that it couldn't be loaded (or even if loaded, but can't run correctly) in latest image is not what i think good practice for keeping your house clean. So, by getting rid any intermediate artifacts from build chain we will be able to make sure that: - VMMaker loads w/o problems in latest/multiple image(s) - VMMaker works well and can generate code That's means that for any newcomer, loading and building VM will be much less tedious process, and for 99% of cases it can be fully automated error free process. And let me remind our motto: we're not trying to produce perfect system/setup from first lucky shot. We slowly but steady crawling towards improved state of art. Any comments, ideas and suggestions are welcome. -- Best regards, Igor Stasenko AKA sig. |
Re: Automated VM build server
|
|
This is really cool Igor.
A next step would be to generate packages for major Linux distro (Debian / Ubuntu, Arch, Fedora, ...). Laurent
On Fri, Dec 17, 2010 at 9:17 PM, Igor Stasenko <[hidden email]> wrote: In our nearest plans is make an automated (Hudson based) build setup |
Re: Automated VM build server
|
|
In reply to this post by Igor Stasenko
On Fri, Dec 17, 2010 at 9:17 PM, Igor Stasenko <[hidden email]> wrote: In our nearest plans is make an automated (Hudson based) build setup this is cool igor!!!
but how can you set a concrete version if the buid is automatic ? how do you know which version of VMMaker and ALL its dependencies? are you going to use ConfigurationOfVMMaker ? if true, how is going to maintain it? + all addons like |
Re: Automated VM build server
|
|
In reply to this post by Igor Stasenko
Gorgeous!
Alexandre On 17 Dec 2010, at 17:17, Igor Stasenko wrote: > In our nearest plans is make an automated (Hudson based) build setup > for Cog and Squeak VMs. > > All files, which used for building an artifacts should be included in > source code repository. > So, most probably i will add the 'hudson-build' dir at same level as > 'platforms', where scripts and configs will be stored. > > Some changes which i going to do: > - all generated code will be removed from version controlled tree. > Instead of it i will add an information, which will be enough > to reproduce the generated code by taking 3 separate config file(s) > > a) image version (or its URL(s) for download) > b) script for loading VMMaker (of concrete version) + all addons like > FFI , FT2 etc etc into image > c) script for generating the source code using VMMaker image > .. > z) tell me if i miss anything here > > With removing generated source code from repo we now can make sure > that any changes in VMMaker (including code generation itself) > can be tested and can be reproduced by build server at any snapshot version. > Also, we could test & see if VMMaker loads well and can generate code > using different base images. > So, we will instantly know if any changes in compiler, or other layers > of system is breaking VMMaker. > > Because having a years old image where VMMaker works 100%, but at same > time have 100% chances that it > couldn't be loaded (or even if loaded, but can't run correctly) in > latest image is not what i think good practice for > keeping your house clean. > > > So, by getting rid any intermediate artifacts from build chain we > will be able to make sure that: > - VMMaker loads w/o problems in latest/multiple image(s) > - VMMaker works well and can generate code > > That's means that for any newcomer, loading and building VM will be > much less tedious process, and for 99% > of cases it can be fully automated error free process. > > And let me remind our motto: we're not trying to produce perfect > system/setup from first lucky shot. > We slowly but steady crawling towards improved state of art. > > Any comments, ideas and suggestions are welcome. > > -- > Best regards, > Igor Stasenko AKA sig. > -- _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;: Alexandre Bergel http://www.bergel.eu ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;. |
Re: Automated VM build server
|
|
In reply to this post by laurent laffont
Sig,
Packages would be nice, but I think a more important goal is to get the vm to describe error conditions. If it did that reliably, I would have been completely content to unzip a binary and write a shell script, or just use a one-click image. Of the things that are under reported, the most notable is when a shared library does not load, where did the vm look for it? One gives it a name via #moduleName, but the vm modifies it and then tries to load using the result. If it fails, it needs to say where it looked. What was the error code? Bill ________________________________________ From: [hidden email] [[hidden email]] On Behalf Of laurent laffont [[hidden email]] Sent: Friday, December 17, 2010 3:33 PM To: [hidden email] Subject: Re: [Pharo-project] Automated VM build server This is really cool Igor. A next step would be to generate packages for major Linux distro (Debian / Ubuntu, Arch, Fedora, ...). Laurent On Fri, Dec 17, 2010 at 9:17 PM, Igor Stasenko <[hidden email]<mailto:[hidden email]>> wrote: In our nearest plans is make an automated (Hudson based) build setup for Cog and Squeak VMs. All files, which used for building an artifacts should be included in source code repository. So, most probably i will add the 'hudson-build' dir at same level as 'platforms', where scripts and configs will be stored. Some changes which i going to do: - all generated code will be removed from version controlled tree. Instead of it i will add an information, which will be enough to reproduce the generated code by taking 3 separate config file(s) a) image version (or its URL(s) for download) b) script for loading VMMaker (of concrete version) + all addons like FFI , FT2 etc etc into image c) script for generating the source code using VMMaker image .. z) tell me if i miss anything here With removing generated source code from repo we now can make sure that any changes in VMMaker (including code generation itself) can be tested and can be reproduced by build server at any snapshot version. Also, we could test & see if VMMaker loads well and can generate code using different base images. So, we will instantly know if any changes in compiler, or other layers of system is breaking VMMaker. Because having a years old image where VMMaker works 100%, but at same time have 100% chances that it couldn't be loaded (or even if loaded, but can't run correctly) in latest image is not what i think good practice for keeping your house clean. So, by getting rid any intermediate artifacts from build chain we will be able to make sure that: - VMMaker loads w/o problems in latest/multiple image(s) - VMMaker works well and can generate code That's means that for any newcomer, loading and building VM will be much less tedious process, and for 99% of cases it can be fully automated error free process. And let me remind our motto: we're not trying to produce perfect system/setup from first lucky shot. We slowly but steady crawling towards improved state of art. Any comments, ideas and suggestions are welcome. -- Best regards, Igor Stasenko AKA sig. |
|
|
In reply to this post by Mariano Martinez Peck
On 17 December 2010 22:37, Mariano Martinez Peck <[hidden email]> wrote:
> > > On Fri, Dec 17, 2010 at 9:17 PM, Igor Stasenko <[hidden email]> wrote: >> >> In our nearest plans is make an automated (Hudson based) build setup >> for Cog and Squeak VMs. >> >> > > this is cool igor!!! > >> >> a) image version (or its URL(s) for download) >> b) script for loading VMMaker (of concrete version) > > but how can you set a concrete version if the buid is automatic ? how do you > know which version of VMMaker and ALL its dependencies? we will collect necessary info bit by bit. Obviously, this knowledge is already sits over there, in image, otherwise how else we manage to build VMs till those days? I just want to automate it, which will leave less space for human mistake. > are you going to use ConfigurationOfVMMaker ? if true, how is going to > maintain it? > i am not quite sure if metacello will be capable to keep all this info. Some things may require using changesets or other kind of non-packageable messy code. We will see. Things are scary from distance :) In any case i don't think that a config, which fully describing a conglomerate used to generate VM sources and plugins is any bit worse than image which contains same info. Except that you can't diff two smalltalk images to see what difference between them. -- Best regards, Igor Stasenko AKA sig. |
|
|
In reply to this post by Schwab,Wilhelm K
On 17 December 2010 23:11, Schwab,Wilhelm K <[hidden email]> wrote:
> Sig, > > Packages would be nice, but I think a more important goal is to get the vm to describe error conditions. If it did that reliably, I would have been completely content to unzip a binary and write a shell script, or just use a one-click image. Of the things that are under reported, the most notable is when a shared library does not load, where did the vm look for it? One gives it a name via #moduleName, but the vm modifies it and then tries to load using the result. If it fails, it needs to say where it looked. What was the error code? > This problem lies in different plane: improving the VM code for error reporting. But first we need to build VM. Once we have it, we can play with various options, configurations, settings and so on. And then like biologists, we can afford simply put a dozens of various micro-cultures into incubator (vm configs into build server) and come back next day and see which culture having higher chances to survive (faster, less red tests and error reports) :) > Bill > -- Best regards, Igor Stasenko AKA sig. |
Re: Automated VM build server
|
|
In reply to this post by Schwab,Wilhelm K
On Fri, 17 Dec 2010, Schwab,Wilhelm K wrote:
> Sig, > > Packages would be nice, but I think a more important goal is to get the vm to describe error conditions. If it did that reliably, I would have been completely content to unzip a binary and write a shell script, or just use a one-click image. Of the things that are under reported, the most notable is when a shared library does not load, where did the vm look for it? One gives it a name via #moduleName, but the vm modifies it and then tries to load using the result. If it fails, it needs to say where it looked. What was the error code? ioLoadModule writes this information to stderr. For example: ioLoadModule(/lib/squeak/3.9-7/UUIDPlugin): libuuid.so.1: cannot open shared object file: No such file or directory So I think your problem is already solved. IIRC #moduleName can also contain the full path, so if you want to avoid the VM to guess the path you can define it explicitly. Levente > > Bill > > > ________________________________________ > From: [hidden email] [[hidden email]] On Behalf Of laurent laffont [[hidden email]] > Sent: Friday, December 17, 2010 3:33 PM > To: [hidden email] > Subject: Re: [Pharo-project] Automated VM build server > > This is really cool Igor. > > A next step would be to generate packages for major Linux distro (Debian / Ubuntu, Arch, Fedora, ...). > > Laurent > > On Fri, Dec 17, 2010 at 9:17 PM, Igor Stasenko <[hidden email]<mailto:[hidden email]>> wrote: > In our nearest plans is make an automated (Hudson based) build setup > for Cog and Squeak VMs. > > All files, which used for building an artifacts should be included in > source code repository. > So, most probably i will add the 'hudson-build' dir at same level as > 'platforms', where scripts and configs will be stored. > > Some changes which i going to do: > - all generated code will be removed from version controlled tree. > Instead of it i will add an information, which will be enough > to reproduce the generated code by taking 3 separate config file(s) > > a) image version (or its URL(s) for download) > b) script for loading VMMaker (of concrete version) + all addons like > FFI , FT2 etc etc into image > c) script for generating the source code using VMMaker image > .. > z) tell me if i miss anything here > > With removing generated source code from repo we now can make sure > that any changes in VMMaker (including code generation itself) > can be tested and can be reproduced by build server at any snapshot version. > Also, we could test & see if VMMaker loads well and can generate code > using different base images. > So, we will instantly know if any changes in compiler, or other layers > of system is breaking VMMaker. > > Because having a years old image where VMMaker works 100%, but at same > time have 100% chances that it > couldn't be loaded (or even if loaded, but can't run correctly) in > latest image is not what i think good practice for > keeping your house clean. > > > So, by getting rid any intermediate artifacts from build chain we > will be able to make sure that: > - VMMaker loads w/o problems in latest/multiple image(s) > - VMMaker works well and can generate code > > That's means that for any newcomer, loading and building VM will be > much less tedious process, and for 99% > of cases it can be fully automated error free process. > > And let me remind our motto: we're not trying to produce perfect > system/setup from first lucky shot. > We slowly but steady crawling towards improved state of art. > > Any comments, ideas and suggestions are welcome. > > -- > Best regards, > Igor Stasenko AKA sig. > > > > |
Re: Automated VM build server
|
|
In reply to this post by Igor Stasenko
On 17 Dec 2010, at 21:17, Igor Stasenko wrote: > In our nearest plans is make an automated (Hudson based) build setup for Cog and Squeak VMs. [...] Really, really great news! We must also make sure that the resulting VMs are just as fast as those built manually by wizards ;-) Sven |
Re: Automated VM build server
|
|
In reply to this post by Igor Stasenko
>>
>> this is cool igor!!! yes after all this fight with XCode yesterday ;) >>> a) image version (or its URL(s) for download) >>> b) script for loading VMMaker (of concrete version) >> >> but how can you set a concrete version if the buid is automatic ? how do you >> know which version of VMMaker and ALL its dependencies? > > we will collect necessary info bit by bit. > Obviously, this knowledge is already sits over there, in image, > otherwise how else > we manage to build VMs till those days? > I just want to automate it, which will leave less space for human mistake. agility agility agility agility agility. :) |
Re: Automated VM build server
|
|
In reply to this post by Igor Stasenko
On Fri, Dec 17, 2010 at 11:13 PM, Igor Stasenko <[hidden email]> wrote:
Igor, what I wanted to express (maybe I did it wrong), is that we need to first solve a previous problem with the VM. I think we first need to be able to TRACE! And for TRACE I mean, that for a specific VM state, I can trace: - which METACELLO version to use in order to generate sources. And with this metacello version, we can guess ALL the versions of all requiered packages - which SVN version If the idea is to always load all the latest code, then you can just load the baseline of ConfigurationOfVMMaker and that's all. With svn is even easier. cheers mariano -- |
|
In reply to this post by Levente Uzonyi-2
2010/12/18 Levente Uzonyi <[hidden email]>
[...] so if you want to avoid the VM to guess the path you can define it explicitly. The SqueakVM path/filename guessing never works out for me on Ubuntu. So I always have to explicitly set the module/library name like this moduleName "Was: ^ 'sqlite'" ^ '/usr/lib/libsqlite3.so.0' |
|
|
In reply to this post by Mariano Martinez Peck
this is cool Igor.
can I get your scripts? (I'm working on a set to build the vms for mac, and your scripts would be helpful) I think we need to move the vm making stuff into a more formal and repeatable process... so I think we need to bring the vmmaker into a ConfigurationOfVMMaker... if we can achieve that, we can automatize the building in a reliable way. So... if there are things inside cs's and other formats, we first need to package them into monticello files (that's why I packaged ClipboardExtended into VMMaker - my first contribution to vm :) ) that's my 2c. Cheers, Esteban El 18/12/2010, a las 7:31a.m., Mariano Martinez Peck escribió:
|
|
|
In reply to this post by Mariano Martinez Peck
On 18 December 2010 11:31, Mariano Martinez Peck <[hidden email]> wrote:
> > > On Fri, Dec 17, 2010 at 11:13 PM, Igor Stasenko <[hidden email]> wrote: >> >> On 17 December 2010 22:37, Mariano Martinez Peck <[hidden email]> >> wrote: >> > >> > >> > On Fri, Dec 17, 2010 at 9:17 PM, Igor Stasenko <[hidden email]> >> > wrote: >> >> >> >> In our nearest plans is make an automated (Hudson based) build setup >> >> for Cog and Squeak VMs. >> >> >> >> >> > >> > this is cool igor!!! >> > >> >> >> >> a) image version (or its URL(s) for download) >> >> b) script for loading VMMaker (of concrete version) >> > >> > but how can you set a concrete version if the buid is automatic ? how do >> > you >> > know which version of VMMaker and ALL its dependencies? >> >> we will collect necessary info bit by bit. >> Obviously, this knowledge is already sits over there, in image, >> otherwise how else >> we manage to build VMs till those days? >> I just want to automate it, which will leave less space for human mistake. >> >> > are you going to use ConfigurationOfVMMaker ? if true, how is going to >> > maintain it? >> > >> >> i am not quite sure if metacello will be capable to keep all this info. >> Some things may require using changesets or other kind of >> non-packageable messy code. >> We will see. Things are scary from distance :) >> In any case i don't think that a config, which fully describing a >> conglomerate used to generate VM sources and plugins >> is any bit worse than image which contains same info. >> Except that you can't diff two smalltalk images to see what difference >> between them. >> > > Igor, what I wanted to express (maybe I did it wrong), is that we need to > first solve a previous problem with the VM. > I think we first need to be able to TRACE! And for TRACE I mean, that for a > specific VM state, I can trace: > - which METACELLO version to use in order to generate sources. And with this > metacello version, we can guess ALL the versions of all requiered packages > - which SVN version > Yes, this is what i want to do. All metainformation about what is used for generating sources and building VM. Including a VMMaker config (like set of plugins to include/exclude). > If the idea is to always load all the latest code, then you can just load > the baseline of ConfigurationOfVMMaker and that's all. With svn is even > easier. > No. the idea is to take any snapshot stored in version-managed repository and be able to accurately reproduce it, without using external info/files. > cheers > > mariano > >> >> -- >> Best regards, >> Igor Stasenko AKA sig. >> > > -- Best regards, Igor Stasenko AKA sig. |
Re: Automated VM build server
|
|
In reply to this post by EstebanLM
On Dec 18, 2010, at 1:23 PM, Esteban Lorenzano wrote: > this is cool Igor. > can I get your scripts? (I'm working on a set to build the vms for mac, and your scripts would be helpful) Everything will be on gitorious (or it is already there). > I think we need to move the vm making stuff into a more formal and repeatable process... oh yes! > so I think we need to bring the vmmaker into a ConfigurationOfVMMaker... is not already the case? I thought so at least jean-baptiste did one. > if we can achieve that, we will :) > we can automatize the building in a reliable way. We plan to have hudson servers for that too :) > So... if there are things inside cs's and other formats, we first need to package them into monticello files (that's why I packaged ClipboardExtended into VMMaker - my first contribution to vm :) ) > that's my 2c. > > Cheers, > Esteban > > El 18/12/2010, a las 7:31a.m., Mariano Martinez Peck escribió: > >> >> >> On Fri, Dec 17, 2010 at 11:13 PM, Igor Stasenko <[hidden email]> wrote: >> On 17 December 2010 22:37, Mariano Martinez Peck <[hidden email]> wrote: >> > >> > >> > On Fri, Dec 17, 2010 at 9:17 PM, Igor Stasenko <[hidden email]> wrote: >> >> >> >> In our nearest plans is make an automated (Hudson based) build setup >> >> for Cog and Squeak VMs. >> >> >> >> >> > >> > this is cool igor!!! >> > >> >> >> >> a) image version (or its URL(s) for download) >> >> b) script for loading VMMaker (of concrete version) >> > >> > but how can you set a concrete version if the buid is automatic ? how do you >> > know which version of VMMaker and ALL its dependencies? >> >> we will collect necessary info bit by bit. >> Obviously, this knowledge is already sits over there, in image, >> otherwise how else >> we manage to build VMs till those days? >> I just want to automate it, which will leave less space for human mistake. >> >> > are you going to use ConfigurationOfVMMaker ? if true, how is going to >> > maintain it? >> > >> >> i am not quite sure if metacello will be capable to keep all this info. >> Some things may require using changesets or other kind of >> non-packageable messy code. >> We will see. Things are scary from distance :) >> In any case i don't think that a config, which fully describing a >> conglomerate used to generate VM sources and plugins >> is any bit worse than image which contains same info. >> Except that you can't diff two smalltalk images to see what difference >> between them. >> >> >> Igor, what I wanted to express (maybe I did it wrong), is that we need to first solve a previous problem with the VM. >> I think we first need to be able to TRACE! And for TRACE I mean, that for a specific VM state, I can trace: >> - which METACELLO version to use in order to generate sources. And with this metacello version, we can guess ALL the versions of all requiered packages >> - which SVN version >> >> If the idea is to always load all the latest code, then you can just load the baseline of ConfigurationOfVMMaker and that's all. With svn is even easier. >> >> cheers >> >> mariano >> >> -- >> Best regards, >> Igor Stasenko AKA sig. >> >> > |
|
|
In reply to this post by EstebanLM
On 18 December 2010 13:23, Esteban Lorenzano <[hidden email]> wrote:
> this is cool Igor. > can I get your scripts? (I'm working on a set to build the vms for mac, and > your scripts would be helpful) I will put everything to my clone repo git://gitorious.org/~abrabapupa/cogvm/sig-cog.git as soon as will be able to fully build Cog VM with freetype plugin. I built Cog with xcode, but i didn't checked if it can be built using command line. > I think we need to move the vm making stuff into a more formal and > repeatable process... so I think we need to bring the vmmaker into a > ConfigurationOfVMMaker... if we can achieve that, we can automatize the > building in a reliable way. So... if there are things inside cs's and other > formats, we first need to package them into monticello files (that's why I > packaged ClipboardExtended into VMMaker - my first contribution to vm :) ) > that's my 2c. Yes. > Cheers, > Esteban -- Best regards, Igor Stasenko AKA sig. |
|
|
El 18/12/2010, a las 3:28p.m., Igor Stasenko escribió: > On 18 December 2010 13:23, Esteban Lorenzano <[hidden email]> wrote: >> this is cool Igor. >> can I get your scripts? (I'm working on a set to build the vms for mac, and >> your scripts would be helpful) > > I will put everything to my clone repo > git://gitorious.org/~abrabapupa/cogvm/sig-cog.git > as soon as will be able to fully build Cog VM with freetype plugin. > > I built Cog with xcode, but i didn't checked if it can be built using > command line. yep, with "xcodebuild"... it is not a great tool, but it is enough :) cheers, Esteban |
|
|
On 18 December 2010 22:32, Esteban Lorenzano <[hidden email]> wrote:
> > El 18/12/2010, a las 3:28p.m., Igor Stasenko escribió: > >> On 18 December 2010 13:23, Esteban Lorenzano <[hidden email]> wrote: >>> this is cool Igor. >>> can I get your scripts? (I'm working on a set to build the vms for mac, and >>> your scripts would be helpful) >> >> I will put everything to my clone repo >> git://gitorious.org/~abrabapupa/cogvm/sig-cog.git >> as soon as will be able to fully build Cog VM with freetype plugin. >> >> I built Cog with xcode, but i didn't checked if it can be built using >> command line. > > yep, with "xcodebuild"... it is not a great tool, but it is enough :) > okay, i managed to build the whole VM & pushed changes to my repo. see macbuild/FT2Plugin macbuild/makevm (changed) also look at codegen-scripts/VMMaker-Version.st there i gathered info of all stuff which related to code generation. I also removed src dir :) > cheers, > Esteban > -- Best regards, Igor Stasenko AKA sig. |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |