Automatically building Application Images via Jenkins/CLI

You will find attached the system we use for automatically building smalltalk images for our applications from Jenkins as well as manually from the command line (if our developers want an ad-hoc image for testing).  Unfortunately google groups prevents posting windows batch scripts so they are posted with a .txt extension added to get them accepted.

I'm posting this to help others that may also want to automatically build their applications images and thus to get community co-operation in improving this system so we can all benefit.  What I have here is by no means perfect.  I'll list the known issues towards the end of the post.

Our organization had an issue in that our applications were only being reliably built by one of our developers using a magic master image setup 'just right'.  We wanted to fix this so that image creation was repeatable and automated and thus anyone could do it and do it quickly.  Attached is the result of this effort.  I'll note that the scripts to create the image that are attached are only part of the solution.  This system relies on you having config maps containing everything you need to create your application.  If you don't have these already, I cannot really help too much.  It took us a bit of effort to get our config maps containing all the relevant dependencies, but that is a per-requisite of this solution.  Basically, you need to be able to build your applications manually from a clean 'new' image to then automate the process.

The attached solution assumes that Windows is the primary build environment and that UNIX is the XD environment.  In our case we have 3 windows images and 8 UNIX images to build for each major release and build all of them from the windows version of VA Smalltalk.  If Linux is your primary build environment, you would only have to adapt the batch scripts to shell scripts and change the UNIX.txt file (which defines the XD image properties) to be a different platform.

This build system also assumes VAST 9.1 (as that is what we use right now - we will be upgrading to 9.2 on the next release cycle for our application) although the only thing that makes that assumption is the paths used in the batch files.  I believe it will work in any version where the PostStartUp capability in image startup scripts exists (the doco at https://www.instantiations.com/docs/91/sg/wwhelp/wwhimpl/common/html/wwhelp.htm#href=stug515.html&single=true says this is from 8.6.2).  You just have to change the paths to nodialog.exe and newimage\abt.icx to match your installation.

Pre-requisities

Our solution assumes you have:

1. A config map that has all the features needed by your application along with all the packaging instructions classes for your application.  If you need to build XD images, you need this config map to include all the XD packaging support.  In our case this map also includes some of the VAST Goodies and 3rd party applications we use although that is a personal choice to cut down the time it takes to create the applications images.
   
2. If you need to build an XD image, a config map containing the features needed by the XD image for your application
3. A config map for your application.  The attached solution assumes that you name the versions of this config map with a version number prefixed by a capital V somewhere in the version name.

High Level Flow

The solution involves to basic steps:

1. Build an image that is used as the basis image from which the application image is built.  If you build images manually, this takes the place of the Master image you always load to create a release image.  In our case, we have a single 'build' image from which we build the 11 applications images.  The 'build' image is the one that the config map containing all the features if put into.  We only re-build this image if we change the config map used by this image - which for us is only really at major release boundaries.
   
2. Using this build image, then load the application config map and run the packager for the application

Note that the way things are written right now, the packaging instructions for the application must exist in the 'build' image.  If you try this out and it cannot find the packaging instruction class, they I suspect you have it in the application config map rather than the 'build' config map.

Components

There are very few components to our solution.  You will need to supply your own abt.ini with the reference to your library and any parameters you need.  Otherwise in the attached zip file you will find

• abt.cnf: The initial image startup script used to create the build image.  This takes the following command line options
• -user=yourusername that will own the image.  Must be a user in the library.
  
• -map=MainConfigMap that points to the features you need in the main build image
  
• Optionally -xdmap=XDConfigMap if you need an XD image listing the config map for the features to put into the XD Image
  
• UNIX.txt: A text file used by the XD image creator that defines the properties of the XD image.  Note that while this file has a place to define InstalledFeatures, this list is ignored by the XD subsystem.  Instead, you will find a bit of code in abt.cnf that does 'xdImage installFeatures:' that will load features not in your config map
  
• build.cnf: The image startup script used by the builder image to create your final application images.  This takes the following command line options
• -map=ApplicationConfigMap which is your application config map
  
• -pkg=PkgInstructionName which is the name of the packaging instruction to use
• Optionally -ver=<version> which is the version number of the application config map to load.  If no version is supplied, the latest edition of the config map (versioned or not) is built.  Else the latest versioned config map which has the string V<version> in it is built.
• create_builder.cmd: A sample windows batch file to make it quick to create new builder images.  You will have to edit this for your situation.
  
• build_image.cmd: A windows batch file to use the build image to create application images.  You might have to edit this for your situation.
  

We use a 3 segment version number for our config maps.  V<major>.<minor>.<patch>.  If you use a different system you will need to change the edition detect: logic in build.cnf

The build.cnf looks at the type of packaging instruction supplied on the command line.  If it is an XD packaging instruction, the config map and version is loaded into the development image and XD image and the XD packager is invoked.

Running

To create a build.icx image used to build application images you can put the attached files into a directory and edit create_builder script or run

COPY /Y "C:\Program Files (x86)\Instantiations\VA Smalltalk\9.1x86\newimage\abt.icx"
"C:\Program Files (x86)\Instantiations\VA Smalltalk\9.1x86\nodialog.exe" -iabt.icx -ini:abt.ini -loutput.txt -user=yourusername "-map=MainConfigMap" "-xdmap=XDConfigMap"

TYPE output.txt

To create an application image, you can then use this build.icx and run build_image.cmd which basically does

"C:\Program Files (x86)\Instantiations\VA Smalltalk\9.1x86\nodialog.exe" -ibuild.icx -ini:abt.ini -loutput.txt "-map=ApplicationConfigMap" "-pkg=PackagingInstructionName"-ver=X.X

TYPE output.txt

Outputs

You will (hopefully) get all the outputs configured in your packaging instruction.  In addition, we generate an HTML snippet containing the description of the config map edition being built.  We use this as part of our release note generator for our releases.

Jenkins tips and tricks

Jenkins cannot consume the -lCON output generated by abt,exe/nodialog.exe.  It assumes that the output it will grab is generated directly by the command that is being executed.  To get around this, I tell abt.exe to write to a file (output.txt) and once the operation is completed I output this file to jenkins.  This means that the console output in Jenkins is not updated while an image is being built - only at the end.

The second tip I can give is that it is not possible (well I couldn't find a way) to run the packager in a purely headless mode.  This means it requires a desktop to work.  Jenkins only runs a slave build on a desktop if the agent is run from a desktop - and not as a windows service.  While the build will still run if your run it as a build job on a windows service jenkins slave, you will not be able to resolve any issues with the build job (as you will have no UI).

We have one build job to create the build.icx which is then archived as an artifact by Jenkins.  The application image build jobs then Copy this artifact as part of the application image build jobs.  The output images are then fed to the build jobs we use to package our total application (eg with the VA smalltalk runtime DLLs and all our support images and stuff).

Issues

1. I've not found a way to guarantee that the process 'exits' on failure.  Any walkback inducing error will launch the interactive debugger.  I've tried to intercept most prompts and errors such that Jenkins is told of the error, but definitely not all.  If nodialog.exe does not 'exit' then Jenkins waits forever for it to finish.  The only way to 'fix' this is to login to the desktop running the jenkins slave and answer whatever prompt is being displayed in the VA Smalltalk UI.
   
2. I've not found a way to intercept the XD transcript so that it appears in the jenkins job output.  There appears to be lots of code inside the XD system that assumes the XD transcript is a CwShell style window.  So right now, you cannot see progress of the XD portion of the build.  If anyone has a tip or trick to fix this, I'd be interested in it.
3. We've not yet added in the running of the SUnit tests as part of this.  Right now we run this sort of testing prior to versioning the config map for release.  We'll be looking at adding a build job stage to run the tests in the near future.  This first 'cut'
   

Feel free to post questions about this solution and I'll try to help, but if you have improvements, I'd be most interested.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/641e2778-2b57-453a-b1ae-4f8d84093faa%40googlegroups.com.