Automating the runtime packager
Locked
Automating the runtime packager
 
|
I spent a bit of time looking into how possible it would be to automate image packaging using the RuntimePackager (RTP). It looks like the packaging code is highly coupled with the UI (including methods that do work and raise dialogs that have to be dismissed by hand). I don't even see a good way to load a parameters file using code, as the parameter loading code is embedded in the same method that prompts for the parameters file.
Has anyone had any success doing this? If not, is there an AR for decoupling the packaging logic from the UI so that this can be scripted/automated? Thanks, Randy -- Randy Coulman [hidden email] |
RE: Automating the runtime packager
|
|
I seem to recall there was some AR that was addressed recently relating
to this, but this is what I ended up doing as part of our automated builds, adjust as needed (#pundlesToInclude is a simple collection of pundle names to keep and #buildReferenceDirectory is where the rtp file lives), DeploymentManager>>packageRuntimeImage | rtp wrap editor tree | RuntimePackager.RuntimeBuilderUI open. rtp := RuntimePackager.RuntimeBuilderUI activeApplication. wrap := (Refactory.Wrappers.ConditionalMethodWrapper on: #requestFileName:default:version:ifFail: inClass: Dialog class alternative: [:a :b | (self buildReferenceDirectory asFilename construct: 'params.rtp') asString]) condition: [:a :b | false]; yourself. wrap install. [rtp loadParams] ensure: [wrap uninstall]. rtp keptItems edit. editor := rtp keptItems editor. tree := editor systemPackagesTree treeHolder tree. tree copy do: [:ea | tree expandFully: (tree indexOf: ea)]. self pundlesToInclude do: [:nm | | subject | subject := (editor systemPackagesTree selectionAspect list detect: [:ea | ea value name = nm]) value. editor systemPackagesTree select: subject. editor keepSelectedPundle]. editor accept. wrap := (Refactory.Wrappers.ConditionalMethodWrapper on: #confirm: inClass: Dialog class alternative: [:a :b | true]) condition: [:a :b | false]; yourself. wrap install. [rtp packageRuntimeImage] ensure: [wrap uninstall] -Boris -- +1.604.689.0322 DeepCove Labs Ltd. 4th floor 595 Howe Street Vancouver, Canada V6C 2T5 http://tinyurl.com/r7uw4 [hidden email] CONFIDENTIALITY NOTICE This email is intended only for the persons named in the message header. Unless otherwise indicated, it contains information that is private and confidential. If you have received it in error, please notify the sender and delete the entire message including any attachments. Thank you. > -----Original Message----- > From: Randy Coulman [mailto:[hidden email]] > Sent: Friday, December 07, 2007 3:46 PM > To: visualWorks Mailing List > Subject: Automating the runtime packager > > I spent a bit of time looking into how possible it would be to automate > image packaging using the RuntimePackager (RTP). It looks like the > packaging code is highly coupled with the UI (including methods that do > work and raise dialogs that have to be dismissed by hand). I don't even > see a good way to load a parameters file using code, as the parameter > loading code is embedded in the same method that prompts for the > parameters file. > > Has anyone had any success doing this? If not, is there an AR for > decoupling the packaging logic from the UI so that this can be > scripted/automated? > > Thanks, > Randy > -- > Randy Coulman > [hidden email] |
Re: Automating the runtime packager
|
|
Boris, you are truly evil :-). I never even thought of using MethodWrappers.
It looks like it might work. Thanks for the help. Randy On Dec 7, 2007 3:51 PM, Boris Popov <[hidden email]> wrote: I seem to recall there was some AR that was addressed recently relating -- Randy Coulman [hidden email] |
RE: Automating the runtime packager
|
|
Hehe, wrappers aren't evil, trying to override RTP's methods would be!
Cheers, -Boris -- +1.604.689.0322 DeepCove Labs Ltd. 4th floor 595 Howe Street Vancouver, Canada V6C 2T5 http://tinyurl.com/r7uw4 [hidden email] CONFIDENTIALITY NOTICE This email is intended only for the persons named in the message header. Unless otherwise indicated, it contains information that is private and confidential. If you have received it in error, please notify the sender and delete the entire message including any attachments. Thank you. > -----Original Message----- > From: Randy Coulman [mailto:[hidden email]] > Sent: Friday, December 07, 2007 3:58 PM > To: Boris Popov > Cc: visualWorks Mailing List > Subject: Re: Automating the runtime packager > > Boris, you are truly evil :-). I never even thought of using > MethodWrappers. > > It looks like it might work. Thanks for the help. > > Randy > > > On Dec 7, 2007 3:51 PM, Boris Popov < [hidden email] > <mailto:[hidden email]> > wrote: > > > I seem to recall there was some AR that was addressed recently > relating > to this, but this is what I ended up doing as part of our > builds, adjust as needed (#pundlesToInclude is a simple collection > of > pundle names to keep and #buildReferenceDirectory is where the rtp > file > lives), > > DeploymentManager>>packageRuntimeImage > > | rtp wrap editor tree | > RuntimePackager.RuntimeBuilderUI open. > rtp := RuntimePackager.RuntimeBuilderUI activeApplication. > wrap := (Refactory.Wrappers.ConditionalMethodWrapper on: > #requestFileName:default:version:ifFail: inClass: Dialog class > alternative: [:a :b | (self buildReferenceDirectory asFilename > construct: 'params.rtp') asString]) condition: [:a :b | false]; > yourself. > wrap install. > [rtp loadParams] ensure: [wrap uninstall]. > rtp keptItems edit. > editor := rtp keptItems editor. > tree := editor systemPackagesTree treeHolder tree. > tree copy do: [:ea | tree expandFully: (tree indexOf: ea)]. > self pundlesToInclude > do: > [:nm | > | subject | > subject := (editor systemPackagesTree > list detect: [:ea | ea value name = nm]) value. > editor systemPackagesTree select: subject. > editor keepSelectedPundle]. > editor accept. > wrap := (Refactory.Wrappers.ConditionalMethodWrapper on: #confirm: > inClass: Dialog class alternative: [:a :b | true]) condition: [:a :b > | > false]; yourself. > wrap install. > [rtp packageRuntimeImage] ensure: [wrap uninstall] > > -Boris > > -- > +1.604.689.0322 > DeepCove Labs Ltd. > 4th floor 595 Howe Street > Vancouver, Canada V6C 2T5 > http://tinyurl.com/r7uw4 <http://tinyurl.com/r7uw4> > > [hidden email] > > CONFIDENTIALITY NOTICE > > This email is intended only for the persons named in the message > header. Unless otherwise indicated, it contains information that > private and confidential. If you have received it in error, please > notify the sender and delete the entire message including any > attachments. > > Thank you. > > > > -----Original Message----- > > From: Randy Coulman [mailto:[hidden email]] > > Sent: Friday, December 07, 2007 3:46 PM > > To: visualWorks Mailing List > > Subject: Automating the runtime packager > > > > I spent a bit of time looking into how possible it would be to > automate > > image packaging using the RuntimePackager (RTP). It looks > the > > packaging code is highly coupled with the UI (including methods > that > do > > work and raise dialogs that have to be dismissed by hand). I > don't > even > > see a good way to load a parameters file using code, as the > parameter > > loading code is embedded in the same method that prompts for the > > parameters file. > > > > Has anyone had any success doing this? If not, is there an AR for > > decoupling the packaging logic from the UI so that this can be > > scripted/automated? > > > > Thanks, > > Randy > > -- > > Randy Coulman > > [hidden email] > > > > > > -- > Randy Coulman > [hidden email] <mailto:[hidden email]> |
Re: Automating the runtime packager
|
|
In reply to this post by Randy Coulman
Randy Coulman wrote:
> I spent a bit of time looking into how possible it would be to > automate image packaging using the RuntimePackager (RTP). It looks > like the packaging code is highly coupled with the UI (including > methods that do work and raise dialogs that have to be dismissed by > hand). I don't even see a good way to load a parameters file using > code, as the parameter loading code is embedded in the same method > that prompts for the parameters file. > > Has anyone had any success doing this? If not, is there an AR for > decoupling the packaging logic from the UI so that this can be > scripted/automated? This is not trivial. I spent almost half a year modifying RTP to suit my needs. One of the most helpful steps was to create a CodeComponentCollector utility class: collector := CodeComponentCollector new includePundleWildpattern: 'CGN Product Base *' withPrerequisites: true; includePundleWildpattern: 'CGN Installer' withPrerequisites: true; excludeClassesMarked: #excludeFromRuntime; excludeMethodsWithPragma: #strip: argument: #demo; yourself. These collectors work similar to sets and understand #merge:, #intersect:, #without:, etc. After having created the appropriate collector for my product, I can ask it for #classes, #extensionMethods, #bundles, #packages, #specificallyExcludedClasses, #specificallyExcludedMethods, ..., etc in order to configure the basic stripping logic of RTP programmatically. Maybe this can serve as an inspiration. If I had some more spare time, I would review the code and publish it. It's basically only a single class. I'll have a look at it next week. Andre |
Re: Automating the runtime packager
|
|
On Dec 7, 2007 4:17 PM, Andre Schnoor <[hidden email]> wrote:
This sounds interesting. I didn't mention in my original message that we ship our stuff as a minimal base images plus parcels that we load at startup, so I'm not as worried about this part of the process. We start from the base.im that's in the preview/package directory, load a couple of small parcels, tweak a few settings, and then package up the image. But even those few steps have lots of GUI coupling in them. Thanks, Randy -- Randy Coulman [hidden email] |
Re: Automating the runtime packager
|
|
In reply to this post by Boris Popov, DeepCove Labs (SNN)
If you load WebToolkit, and look at the code it uses to save a headless
image, you'll see one approach to scripting RTP ----- Original Message ----- From: "Boris Popov" <[hidden email]> To: "Randy Coulman" <[hidden email]> Cc: "visualWorks Mailing List" <[hidden email]> Sent: Friday, December 07, 2007 7:00 PM Subject: RE: Automating the runtime packager > Hehe, wrappers aren't evil, trying to override RTP's methods would be! > > Cheers, > > -Boris > > -- > +1.604.689.0322 > DeepCove Labs Ltd. > 4th floor 595 Howe Street > Vancouver, Canada V6C 2T5 > http://tinyurl.com/r7uw4 > > [hidden email] > > CONFIDENTIALITY NOTICE > > This email is intended only for the persons named in the message > header. Unless otherwise indicated, it contains information that is > private and confidential. If you have received it in error, please > notify the sender and delete the entire message including any > attachments. > > Thank you. > >> -----Original Message----- >> From: Randy Coulman [mailto:[hidden email]] >> Sent: Friday, December 07, 2007 3:58 PM >> To: Boris Popov >> Cc: visualWorks Mailing List >> Subject: Re: Automating the runtime packager >> >> Boris, you are truly evil :-). I never even thought of using >> MethodWrappers. >> >> It looks like it might work. Thanks for the help. >> >> Randy >> >> >> On Dec 7, 2007 3:51 PM, Boris Popov < [hidden email] >> <mailto:[hidden email]> > wrote: >> >> >> I seem to recall there was some AR that was addressed recently >> relating >> to this, but this is what I ended up doing as part of our > automated >> builds, adjust as needed (#pundlesToInclude is a simple > collection >> of >> pundle names to keep and #buildReferenceDirectory is where the > rtp >> file >> lives), >> >> DeploymentManager>>packageRuntimeImage >> >> | rtp wrap editor tree | >> RuntimePackager.RuntimeBuilderUI open. >> rtp := RuntimePackager.RuntimeBuilderUI activeApplication. >> wrap := (Refactory.Wrappers.ConditionalMethodWrapper on: >> #requestFileName:default:version:ifFail: inClass: Dialog class >> alternative: [:a :b | (self buildReferenceDirectory asFilename >> construct: 'params.rtp') asString]) condition: [:a :b | false]; >> yourself. >> wrap install. >> [rtp loadParams] ensure: [wrap uninstall]. >> rtp keptItems edit. >> editor := rtp keptItems editor. >> tree := editor systemPackagesTree treeHolder tree. >> tree copy do: [:ea | tree expandFully: (tree indexOf: ea)]. >> self pundlesToInclude >> do: >> [:nm | >> | subject | >> subject := (editor systemPackagesTree > selectionAspect >> list detect: [:ea | ea value name = nm]) value. >> editor systemPackagesTree select: subject. >> editor keepSelectedPundle]. >> editor accept. >> wrap := (Refactory.Wrappers.ConditionalMethodWrapper on: > #confirm: >> inClass: Dialog class alternative: [:a :b | true]) condition: > [:a :b >> | >> false]; yourself. >> wrap install. >> [rtp packageRuntimeImage] ensure: [wrap uninstall] >> >> -Boris >> >> -- >> +1.604.689.0322 >> DeepCove Labs Ltd. >> 4th floor 595 Howe Street >> Vancouver, Canada V6C 2T5 >> http://tinyurl.com/r7uw4 <http://tinyurl.com/r7uw4> >> >> [hidden email] >> >> CONFIDENTIALITY NOTICE >> >> This email is intended only for the persons named in the message >> header. Unless otherwise indicated, it contains information that > is >> private and confidential. If you have received it in error, > please >> notify the sender and delete the entire message including any >> attachments. >> >> Thank you. >> >> >> > -----Original Message----- >> > From: Randy Coulman [mailto:[hidden email]] >> > Sent: Friday, December 07, 2007 3:46 PM >> > To: visualWorks Mailing List >> > Subject: Automating the runtime packager >> > >> > I spent a bit of time looking into how possible it would be to >> automate >> > image packaging using the RuntimePackager (RTP). It looks > like >> the >> > packaging code is highly coupled with the UI (including > methods >> that >> do >> > work and raise dialogs that have to be dismissed by hand). I >> don't >> even >> > see a good way to load a parameters file using code, as the >> parameter >> > loading code is embedded in the same method that prompts for > the >> > parameters file. >> > >> > Has anyone had any success doing this? If not, is there an AR > for >> > decoupling the packaging logic from the UI so that this can be >> > scripted/automated? >> > >> > Thanks, >> > Randy >> > -- >> > Randy Coulman >> > [hidden email] >> >> >> >> >> >> -- >> Randy Coulman >> [hidden email] <mailto:[hidden email]> > > |
Re: Automating the runtime packager
|
|
In reply to this post by Randy Coulman
On Dec 7, 2007 4:17 PM, Andre Schnoor <[hidden email]> wrote: I'm using the CodeComponentCollector basically to create and save one big "product.pcl" and load that into a small base image later on in the build procedure. Consequently, there's a subclass of CodeComponentCollector: ParcelBuilder ;-) This parcel could of course also be shipped with the product directly. I found it very convenient and robust to maintain method pragmas (e.g. <strip: #demo>) and class methods (e.g. #excludeFromProduct) to specify at development time already what's in a product and what's not. Especially when there are multipe variants of a product (demo, express, pro), this saves a lot of hassle. Andre |
Re: Automating the runtime packager
|
|
In reply to this post by Randy Coulman
I have created an automated build tool for "building up" runtime
images. It consists of around 30 method extensions to the runtime builder UI class. No modification to existing VisualWorks code is needed. Although it uses an UI class, it does not display the actual UI during the build process. It is fully automated and requires no user intervention. My code waits until a new version of a specified top bundle is published. It then creates a (.st) script to load this bundle. A Cincom base visual.im is forked of to execute the generated script. The script loads the top bundle and starts the RTP to create a runtime image. The process then starts over again. Our process is based on doing no reduction ("stripping") of the loaded code.I structure the code so that examples and unit tests are outside the top bundle used for deployment. This makes the process a lot simpler and less error prone. I will try to make the methods public if anyone is interested. Runar Jordahl |
Re: Automating the runtime packager
|
|
In reply to this post by Andre Schnoor
Dear Andre et al,
I also have found it less easy than it could be to automate RTP. I found it easiest not to fight its exploitation of 'last window open' behaviour and settled for programmatically driving the RTP UI. This was doable with various trivial gotchas: - The RTP only takes _some_ memory parameters from its params file; others are taken from the current object memory policy set at the time the file is loaded (see use of the currentMemoryPolicy in RuntimePackagerMemoryParams>>decodeFrom:fileVersion:). This unintuitive behaviour requires you to set the image to your preferred deployment memory policy values for these cases before you call RuntimeBuilderUI activeApplication loadParams. - The RuntimePackager does not invite programmatic (re)setting of the kept and stripped items; you must add protocol. - The packager silently nils its ErrorLogPath if you provide a logical filename although that is clearly an acceptable polymorph. This seems really silly behaviour and I strongly suspect it is an accident deriving from some code written by someone only thinking about strings (a 'size = 0' check is the culprit). You can bypass the setter and assign one to the variable directly. And I suspect there are other issues I've forgotten at the moment. The result was always cluncky. Andre, your stuff sounds better and if published to the OR might be welcomed. (We all agree that build-up is better than strip but in some large old systems the former is not an option.) Yours faithfully Niall Ross > Randy Coulman wrote: > >> I spent a bit of time looking into how possible it would be to >> automate image packaging using the RuntimePackager (RTP). It looks >> like the packaging code is highly coupled with the UI (including >> methods that do work and raise dialogs that have to be dismissed by >> hand). I don't even see a good way to load a parameters file using >> code, as the parameter loading code is embedded in the same method >> that prompts for the parameters file. >> >> Has anyone had any success doing this? If not, is there an AR for >> decoupling the packaging logic from the UI so that this can be >> scripted/automated? > > > This is not trivial. I spent almost half a year modifying RTP to suit > my needs. One of the most helpful steps was to create a > CodeComponentCollector utility class: > > collector := CodeComponentCollector new > includePundleWildpattern: 'CGN Product Base *' > withPrerequisites: true; > includePundleWildpattern: 'CGN Installer' withPrerequisites: true; > excludeClassesMarked: #excludeFromRuntime; > excludeMethodsWithPragma: #strip: argument: #demo; > yourself. > > These collectors work similar to sets and understand #merge:, > #intersect:, #without:, etc. After having created the appropriate > collector for my product, I can ask it for > > #classes, #extensionMethods, #bundles, #packages, > #specificallyExcludedClasses, #specificallyExcludedMethods, ..., etc > > in order to configure the basic stripping logic of RTP programmatically. > > Maybe this can serve as an inspiration. If I had some more spare time, > I would review the code and publish it. It's basically only a single > class. I'll have a look at it next week. > > Andre > > > > |
Re: Automating the runtime packager
|
|
Thanks Niall!
That's a great list of issues for us to fix. I'll note that WebToolkit includes something that scripts RTP to create a runtime image, although it doesn't do any stripping. And yes, it's, um, "less easy than it could be" to automate. There are a good many ARs on various aspects of this, and other things about packaging and deployment. They haven't moved forward as much as I would have liked for 7.6, although there are a number of improvements that did get in, and over the past couple of releases we've done a number of things to make it easier to build up images without using runtime packager. So, for example, one of the significant things that changes in 7.6 is that we've pulled out the RTP-specific error logging code and made a basic version of error logging available in the base image. With that, we can also pull the "old debugger" out of the base image, so if you start up base.im, there's no debugger at all, and errors will just get logged. This reduces the initial size of the base, and also reduces the need to use RTP in order to get things (like error logging) that are tangential to its main function. One thing I find particularly irritating is that you can't use RTP to strip something down to the point where it has no GUI code whatsoever. It requires you to keep on set of GUI looks, and the reason it does that is primarily so that it can display its little progress dialog. At 05:47 PM 12/10/2007, Niall Ross wrote: Dear Andre et al, --
Alan Knight [|], Cincom Smalltalk Development
|
Re: Automating the runtime packager
|
|
In reply to this post by Niall Ross
Niall Ross wrote: > - The RTP only takes _some_ memory parameters from its params file; > others are taken from the current object memory policy set at the time > the file is loaded (see use of the currentMemoryPolicy in > RuntimePackagerMemoryParams>>decodeFrom:fileVersion:). This > unintuitive behaviour requires you to set the image to your preferred > deployment memory policy values for these cases before you call > RuntimeBuilderUI activeApplication loadParams. Ouch. I haven't checkd that yet. Thanks Niall, for pointing this out. This may explain some unexpected behavior of my deployed apps. > We all agree that build-up is better than strip but in some large old > systems the former is not an option. He he. Building up always implies (at least a certain amount of) stripping down. One may not want to keep the code that controls the "build up" procedure itself with the final product and hence it needs to commit suicide in some elegant way. Unless one's product is intended to be open and visible like a bunch of scripts, keeping software engineering tools in a final product image is a bad idea. Provisions that can interpret powerful command line options, make snapshots, evaluate arbitrary expressions, connect to configuration databases and load (unsigned, trojan) parcels, should not be accessible to everyone out there. Stripping these in the correct sequence can be tricky. IMHO, this is a situation where the known lack of subject-object separation strikes especially hard. I still wish there was a provision for creating runtime images "synthetically" from scratch ;-) Andre |
Re: Automating the runtime packager
|
|
On 11/12/2007, at 10:11 AM, Andre Schnoor wrote: > I still wish there was a provision for creating runtime images > "synthetically" from scratch ;-) This is one of the goals of the file-based image system I've been working on. To my mind a declarative approach is to be preferred over a programatic one. Obviously I'm not going to solve this problem tomorrow given my slow progress. Antony Blakey -------------------------- CTO, Linkuistics Pty Ltd Ph: 0438 840 787 Only two things are infinite, the universe and human stupidity, and I'm not sure about the former. -- Albert Einstein |
RE: Automating the runtime packager
|
|
In reply to this post by Andre Schnoor
Andre
If you begin with a base image that has no dev tools other than a parcel loader or compiler you can do a build up using a command line script. One other thing. If you were real ambitious you could probably mutate the imageWriter in Preview->64-bit to write out an arbitrary image. Terry =========================================================== Terry Raymond Crafted Smalltalk 80 Lazywood Ln. Tiverton, RI 02878 (401) 624-4517 [hidden email] <http://www.craftedsmalltalk.com> =========================================================== > -----Original Message----- > From: [hidden email] [mailto:[hidden email]] > Sent: Monday, December 10, 2007 6:41 PM > To: Niall Ross > Cc: Randy Coulman; visualWorks Mailing List > Subject: Re: Automating the runtime packager > > > Niall Ross wrote: > > - The RTP only takes _some_ memory parameters from its params file; > > others are taken from the current object memory policy set at the time > > the file is loaded (see use of the currentMemoryPolicy in > > RuntimePackagerMemoryParams>>decodeFrom:fileVersion:). This > > unintuitive behaviour requires you to set the image to your preferred > > deployment memory policy values for these cases before you call > > RuntimeBuilderUI activeApplication loadParams. > > Ouch. I haven't checkd that yet. Thanks Niall, for pointing this out. > This may explain some unexpected behavior of my deployed apps. > > > We all agree that build-up is better than strip but in some large old > > systems the former is not an option. > > He he. Building up always implies (at least a certain amount of) > stripping down. One may not want to keep the code that controls the > "build up" procedure itself with the final product and hence it needs to > commit suicide in some elegant way. > > Unless one's product is intended to be open and visible like a bunch of > scripts, keeping software engineering tools in a final product image is > a bad idea. Provisions that can interpret powerful command line options, > make snapshots, evaluate arbitrary expressions, connect to configuration > databases and load (unsigned, trojan) parcels, should not be accessible > to everyone out there. Stripping these in the correct sequence can be > tricky. IMHO, this is a situation where the known lack of subject-object > separation strikes especially hard. > > I still wish there was a provision for creating runtime images > "synthetically" from scratch ;-) > > Andre |
RE: Automating the runtime packager
|
|
In reply to this post by Randy Coulman
Hi Randy, Please find attached a couple of methods I’ve
hacked to suppress UI interaction when packaging. There’s also a BootstrapSystem
class that you may find useful to kick the whole thing off via the command
line. HTH, Stewart -----Original Message----- I spent a bit of time looking into how possible it would be to automate
image packaging using the RuntimePackager (RTP). It looks like the
packaging code is highly coupled with the UI (including methods that do work
and raise dialogs that have to be dismissed by hand). I don't even see a
good way to load a parameters file using code, as the parameter loading code is
embedded in the same method that prompts for the parameters file.  BootstrapSystem.st (3K) Download Attachment RuntimeBuilderUI-loadParametersFrom.st (3K) Download Attachment RuntimeBuilderUI-packageRuntimeImageQuietly.st (925 bytes) Download Attachment |
«
Return to VisualWorks
|
1 view|%1 views
| Free forum by Nabble | Edit this page |