CI Build order and strange "latest" Pharo 7 effect

On Thu, Oct 12, 2017 at 9:23 PM, Torsten Bergmann <[hidden email]> wrote:

Hi Guille,

I understand that we want run the pull requests for contributions in parallel. But do we want to run also  the main build for
the daily official relase in parallel to have such side effects? I doubt that.

As I wrote it is not only an issue of the file server. The zero conf does not give you the latest
although available (which is bad if you want to do the next contribution) and we inform about
later build earlier than previous builds, etc.

Yes, I agree that this is a real issue. The reason is the following: every build, all artifacts are uploaded to the file server + a copy named latest.zip, which is used by zeroconf scripts. latest.zip is overwritten on every build. This means that the last build to finish is the last that writes to latest.zip.

 

Sure it would be "nice" if our future Pharo 12 would already be available before we release Pharo 7 - but it
still is not correct and confuses people ;)

So is there anything that can be done to improve/solve this?

I'll check the options and come back to everybody here.

 

Thanks
T.


Am 12.10.2017 um 16:12 schrieb Guillermo Polito:

On Thu, Oct 12, 2017 at 4:01 PM, Norbert Hartl <[hidden email] <mailto:[hidden email]>> wrote:


    > Am 12.10.2017 um 14:23 schrieb Torsten Bergmann <[hidden email]
    <mailto:[hidden email]>>:

    >
    > Hi,
    >
    > today there was an interesting effect visible for the Pharo
    build on CI for Pharo 7:
    >
    > Esteban merged three contributions for Pharo 7 into #development
    (see [1]). All three merges on git triggered the CI.
    > For each a separate build for Pharo 7 started on CI independently.
    >
    > While usually one would expect the builds in regular build order
    183, 184, 185 the CI job for building 185 job was faster
    > (as you can see on the attached screenshot).
    >
    > So the 185 series of images was copied first to the file server
    [3] because the job was already finished.
    > Afterwards 183 build completed on CI and these images were
    copied to the file server with a newer timestamp.
    >
    > So you have the order 185, 183 with 183 having the files with
    the newest timestamp (but not being the most recent state of Pharo).
    >
    > This leads to several unfortunate effects:
    >  1. When you sort on file server [3] descending by date the
    build 183 images are newer than the build 185 images
    >  2. When you use PharoLauncher to download "latest-32" you get
    an 183 image instead of the "real latest" 185 image
    >  3. When you ask Zeroconf to download the latest using
    http://get.pharo.org/70+vm you also get 183 instead of the already
    >     available 185
    >
    > So as the CI build of an older "update" (here 183) finished
    later than the most recent version (here 185)
    > you get 183 as "latest" instead of 185.
    >
    > Any ideas how we could avoid such a side effect? Maybe a chain
    on the builds so 183 has to finished before 184 and 185 could be
    > built?
    >
    I don't understand why it is working like this. Usually only build
    for a job is active, others will be queued. So the problem here is
    that
    the jenkins spawns more builds in parallel for the same job.


Yes, the thing is that the same job is being used to build the development branch and the pull requests.
If you think about it, this is the same setup as in travis.

The thing is that this does not bother me too much, considering the tradeoffs.

- I want pull request to run in parallel.
- I want to have less things to maintain :)

Now,
 - we could split the job in two (integration vs pr validation). But this requires setting up a separate jenkins because otherwise they share the same slave configuration :/. And there is no way in the actual state of the plugins we use to limit the number of parallel builds...
 - we could try to make a build wait until a build with lower build number is building...
 - we could also forbid Esteban to push on the merge button while a build is running
But doing any of those is a lot of work...

- Moreover, I think the problem is that we are using the file server as something it is not. The file server is... a file server.
Ideally, I'd like to attach meta-data to a release/build artifact, and be able to browse/search/sort using that metadata. Something alike artifactory or bintray.

- Also, I don't know how the pharo launcher is getting the latest image, but if instead of using the file timestamp it uses the build number, then this may be fixed, isn't it?


    Norbert

    > Thx
    > T.
    >
    >
    >
    > [1] https://github.com/pharo-project/pharo/commits/development
    <https://github.com/pharo-project/pharo/commits/development>
    > [2]
    https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development
    <https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development>
    > [3] http://files.pharo.org/image/70/

    <http://files.pharo.org/image/70/>
    > <buildOrder.png>




--

       

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - _http://www.cnrs.fr_


*Web:* _http://guillep.github.io_

*Phone: *<a href="tel:%2B33%2006%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13

--

Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13

Well, apparently there is an option to disable concurrent builds, and we should be able to make it work only for the development branch.

https://jenkins.io/doc/book/pipeline/syntax/#options

https://stackoverflow.com/questions/41492688/how-to-limit-jenkins-concurrent-multibranch-pipeline-builds

https://issues.jenkins-ci.org/browse/JENKINS-46593

I've opened the following issue:

https://pharo.fogbugz.com/f/cases/20541/Diable-concurrent-builds-for-the-development-branch

and the following PR:

https://github.com/pharo-project/pharo/pull/363

Could you review/check? I'm in the middle of a meeting right now :)

On Fri, Oct 13, 2017 at 9:40 AM, Guillermo Polito <[hidden email]> wrote:

On Thu, Oct 12, 2017 at 9:23 PM, Torsten Bergmann <[hidden email]> wrote:

Hi Guille,

I understand that we want run the pull requests for contributions in parallel. But do we want to run also  the main build for
the daily official relase in parallel to have such side effects? I doubt that.

As I wrote it is not only an issue of the file server. The zero conf does not give you the latest
although available (which is bad if you want to do the next contribution) and we inform about
later build earlier than previous builds, etc.

Yes, I agree that this is a real issue. The reason is the following: every build, all artifacts are uploaded to the file server + a copy named latest.zip, which is used by zeroconf scripts. latest.zip is overwritten on every build. This means that the last build to finish is the last that writes to latest.zip.

 

Sure it would be "nice" if our future Pharo 12 would already be available before we release Pharo 7 - but it
still is not correct and confuses people ;)

So is there anything that can be done to improve/solve this?

I'll check the options and come back to everybody here.

 

Thanks
T.


Am 12.10.2017 um 16:12 schrieb Guillermo Polito:

On Thu, Oct 12, 2017 at 4:01 PM, Norbert Hartl <[hidden email] <mailto:[hidden email]>> wrote:


    > Am 12.10.2017 um 14:23 schrieb Torsten Bergmann <[hidden email]
    <mailto:[hidden email]>>:

    >
    > Hi,
    >
    > today there was an interesting effect visible for the Pharo
    build on CI for Pharo 7:
    >
    > Esteban merged three contributions for Pharo 7 into #development
    (see [1]). All three merges on git triggered the CI.
    > For each a separate build for Pharo 7 started on CI independently.
    >
    > While usually one would expect the builds in regular build order
    183, 184, 185 the CI job for building 185 job was faster
    > (as you can see on the attached screenshot).
    >
    > So the 185 series of images was copied first to the file server
    [3] because the job was already finished.
    > Afterwards 183 build completed on CI and these images were
    copied to the file server with a newer timestamp.
    >
    > So you have the order 185, 183 with 183 having the files with
    the newest timestamp (but not being the most recent state of Pharo).
    >
    > This leads to several unfortunate effects:
    >  1. When you sort on file server [3] descending by date the
    build 183 images are newer than the build 185 images
    >  2. When you use PharoLauncher to download "latest-32" you get
    an 183 image instead of the "real latest" 185 image
    >  3. When you ask Zeroconf to download the latest using
    http://get.pharo.org/70+vm you also get 183 instead of the already
    >     available 185
    >
    > So as the CI build of an older "update" (here 183) finished
    later than the most recent version (here 185)
    > you get 183 as "latest" instead of 185.
    >
    > Any ideas how we could avoid such a side effect? Maybe a chain
    on the builds so 183 has to finished before 184 and 185 could be
    > built?
    >
    I don't understand why it is working like this. Usually only build
    for a job is active, others will be queued. So the problem here is
    that
    the jenkins spawns more builds in parallel for the same job.


Yes, the thing is that the same job is being used to build the development branch and the pull requests.
If you think about it, this is the same setup as in travis.

The thing is that this does not bother me too much, considering the tradeoffs.

- I want pull request to run in parallel.
- I want to have less things to maintain :)

Now,
 - we could split the job in two (integration vs pr validation). But this requires setting up a separate jenkins because otherwise they share the same slave configuration :/. And there is no way in the actual state of the plugins we use to limit the number of parallel builds...
 - we could try to make a build wait until a build with lower build number is building...
 - we could also forbid Esteban to push on the merge button while a build is running
But doing any of those is a lot of work...

- Moreover, I think the problem is that we are using the file server as something it is not. The file server is... a file server.
Ideally, I'd like to attach meta-data to a release/build artifact, and be able to browse/search/sort using that metadata. Something alike artifactory or bintray.

- Also, I don't know how the pharo launcher is getting the latest image, but if instead of using the file timestamp it uses the build number, then this may be fixed, isn't it?


    Norbert

    > Thx
    > T.
    >
    >
    >
    > [1] https://github.com/pharo-project/pharo/commits/development
    <https://github.com/pharo-project/pharo/commits/development>
    > [2]
    https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development
    <https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development>
    > [3] http://files.pharo.org/image/70/

    <http://files.pharo.org/image/70/>
    > <buildOrder.png>




--

       

Guille Polito

Research Engineer

Centre de Recherche en Informatique, Signal et Automatique de Lille

CRIStAL - UMR 9189

French National Center for Scientific Research - _http://www.cnrs.fr_


*Web:* _http://guillep.github.io_

*Phone: *<a href="tel:%2B33%2006%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13

--

Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13

--

Guille Polito Research Engineer Centre de Recherche en Informatique, Signal et Automatique de Lille CRIStAL - UMR 9189 French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13