Early Pull Request Validation Process

Hi all,

As some/a lot of you have been asking how to contribute, I'm here to answer some questions :).

First of all, integrating fixes in Pharo has been "slowed down" because we missed the Monkey for Pull Request (PR) integration. Some of you have seen a travis job going around and checking. That was our initial solution, but it has a main issue for us: it only works for unix because our build process is long, uses lots of small files due to file tree and osx travis support is REALLY SLOW with I/O access. This is why we chose to use good old Jenkins.

    https://ci.inria.fr/pharo-ci-jenkins2/ 

# Why it took so much time

Just some explanation of why it took so much time (you can skip this if you're not interested).

I started working on integrating Jenkins with Pull Requests several weeks ago (note that I'm part time on this, also working in other projects and the deadlines of the past weeks like IWST were killing my time). We have now a new and working Jenkins server for Pharo7. This jenkins server is hosted in Inria, its url being 

- We chose to use Jenkins 2 because I managed to make the PR integration plugin work only in Jenkins 2.

- This required the creation of a new jenkins server + slaves. The pharo and pharo-contribution servers run with Jenkins 1.6 and I did not want to migrate existing servers to not break existing jobs.

- At the same moment, the inria CI infrastructure was having some (lots?) of hiccups because of storage limitations. I spent several weeks to create the running slaves for each platform and connect jenkins to them because of that.

- Finally, I spent some time to test the new Jenkins MultiJobs. These kind of jobs allow to split a single job in sequential phases. And each phase can have multiple jobs that run concurrently. This allows us to bootstrap once (on unix) and test in all platforms.

An example of the new multijob can be found here:

    https://ci.inria.fr/pharo-ci-jenkins2/job/Pharo-PullRequest-Pipeline/

# Current Design

Right at this moment, we are validating PRs using Jenkins. The process is as follows:

1) Github notifies about changes in PRs to Jenkins using webhooks. So far, we have configured it to notify about PR changes (including comments) and normal pushes (to capture merges into the #development branch).

2) Upon the reception of a GH notification, Jenkins will launch two jobs/validations. The basic structure of both validations is the same so far. Bootstrap once (unix) and test in many platforms. There are the following two test configurations however.

  2.1) Full Validation: tests 32 and 64 bits, for linux, osx and windows.

       This configuration is not fully working right now because we found a VM bug in 64 bits that prevents us from creating a full pharo image. Bootstrap is Ok, but there is a segfault when loading a big Metacello configuration.

  

  2.2) Basic Validation: tests only 32 bits, for now in linux and osx (I'm planning to do windows today).

       This validation is to give us nevertheless a feedback of the build even if the full is failing so far.

  

  2.3) Others? We could easily add new validations to run some faster tests i.e., just loading a patch on the previous image.

  

3) Jenkins takes control and notifies Github

So far, Full validation is failing, and I plan to keep it running until it gets green. I don't care if it's red right now, the idea is that we should fix it, and we should remember it is broken :).

On the other hand, the Basic Validation is working and running tests. Once the tests runs are finished, Github PR state will be either a success (if all tests went green) or a failure (if it failed or some tests failed). You will see actually a message saying:

Validation Failure. Please check test results.

 7782 tests run, 0 skipped, 6 failed.

 

+ a link to jenkins if you want to inspect results.

You can see an example of it in action in here:

  https://github.com/pharo-project/pharo/pull/119

  

# Next points

Right now we are missing:

  - Fogbugz integration

  - Publish new images in the file server when a push/merge happens in the #development branch

    (I should sit today with Esteban to do some ssh key configuration in a server I have no access to)

  - fixing the failing tests => TO HAVE A GREEN BUILD> :D

  - Enhance tests?

  

  - your enhancement idea :)

  

The current validation already gives us an idea of what works and what does not work, what breaks the bootstrap, and if we have failing tests or not.

I'd like however that before integrating a fix, there is at least 2 people that look at it and put a +1. Because Monkey validation and the ability to merge does not mean much actually :).

If you have some questions, enhancements, there is this thread, there is the #bootstrap channel in discord also.

Guille (in behalf of everybody that helped me with this)

--

Guille Polito Research Engineer French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13