Squeak release process - CI server

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Squeak release process - CI server

Chris Cunnington
Super!

The CI server is at

http://www.squeakci.org

I just emailed you a username and password. If there is a problem with
logging in, let me know. If you have questions about how Jenkins works,
you're better off posting here on the list, as people here have more
knowledge than I do about using Jenkins.

Chris

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
On 20 July 2012 17:52, Chris Cunnington <[hidden email]> wrote:

> Super!
>
> The CI server is at
>
> http://www.squeakci.org
>
> I just emailed you a username and password. If there is a problem with
> logging in, let me know. If you have questions about how Jenkins works,
> you're better off posting here on the list, as people here have more
> knowledge than I do about using Jenkins.

OK, so it looks like builtastic.sh is the thing that Jenkins runs. It
fires up a Squeak image and passes in tests.st. Right now, tests.st
runs only the HelpSystem-Tests suite, so we ought to just add more
packages to its list, and those tests will also be run. That's easy
enough, but I DON'T see how I can push an updated tests.st into the
build's workspace. What am I missing?

> Chris
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Sean P. DeNigris
Administrator
Frank Shearar-3 wrote
I DON'T see how I can push an updated tests.st into the
build's workspace.
I don't think you can do that via the Jenkins web UI. Putting the scripts on github like Lukas did makes contributing convenient. Then you can have a really simple Jenkins job that just downloads the scripts and runs one.

HTH,
Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
On 26 July 2012 13:16, Sean P. DeNigris <[hidden email]> wrote:

>
> Frank Shearar-3 wrote
>>
>> I DON'T see how I can push an updated tests.st into the
>> build's workspace.
>>
>
> I don't think you can do that via the Jenkins web UI. Putting the scripts on
> github like Lukas did makes contributing convenient. Then you can have a
> really simple Jenkins job that just downloads the scripts and runs one.

That's an excellent idea: that puts the build scripts under version control.

But I guess I'm asking how those files got there in the first place.
And the answer is probably that someone dropped them there via FTP.
Er, which is actually what Chris Cunnington said. Sigh!

frank

> HTH,
> Sean
>
>
>
> --
> View this message in context: http://forum.world.st/Squeak-release-process-CI-server-tp4640962p4641675.html
> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Chris Cunnington
In reply to this post by Chris Cunnington
HDTestReport runPackages: #(
    'HelpSystem-Tests'
).
"Save and Quit"
WorldState addDeferredUIMessage: [
    SmalltalkImage current snapshot: true andQuit: true ].

or something more succinct:

HDTestReport runPackages: ((WADevelopment allPackages
    select: [ :each | each name includesSubString: '-Tests-' ])
    collect: [ :each | each name ]).


<the simple answer>
tests.st is in the FTP directory. It contains the content above. Add test names as strings in the array like 'HelpSystem-Tests'. Or, iterate over some packages for '-Tests-'
</the simple answer>

<longer explanation/probably a rant>
I don't think GitHub has anything to do with it. The Pharo Jenkins server doesn't use it. That's just where Lukas decided to make his scripts available.

https://github.com/renggli/builder/tree/master/scripts
https://github.com/renggli/builder/blob/master/build.sh

Lukas has a system with two parts: scripts; and, the build.sh shell script. There are loads of scripts lying around on the Pharo Jenkins server. They are attached and combined dynamically by a ~160 line shell script called build.sh. So you'd write something like this:

build.sh -i Omnibrowser -i Greco -i Roman -i Etc

That collects a bunch of scripts. All of them look like the content above: string names in an array. build.sh merges these scripts elegantly into one single stream. There is no tangible difference between that and having a file called tests.st where you cut and paste in the test names as strings.

Is Lukas's system elegant? Yes. Do I want to wade through a ~160 shell script to faithfully reproduce that? Let me put it this way: why would anybody write a shell script over ~100 lines? Even if it was an init script? It seems to me build.sh could have been written in another scripting language and been  both shorter and easier to comprehend for the next guy who came along. It's not 1983.
</longer explanation/probably a rant>

Chris


"I don't think you can do that via the Jenkins web UI. Putting the scripts on github like Lukas did makes contributing convenient. Then you can have a really simple Jenkins job that just downloads the scripts and runs one."

This is interesting, but I don't think it works that way. If I understand what you're saying, build.sh would pull down scripts from Github. I don't think that's even possible. Why does Github use SSL?
To password protect scripts it gives away for free? I don't think so. Only humans can download content from Github. Try putting a picture gif on Github and then link to it from the webpage you're building. Github uses SSL to enforce an interface. They don't want people linking to its resources. Pharo Jenkins has all the scripts locally in a directory somewhere. I don't think you can clone individual files on demand programmaticlly. I'm willing to be wrong, but I feel confident that things are as I just described.

 




Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
On 26 July 2012 15:17, Chris Cunnington <[hidden email]> wrote:

> HDTestReport runPackages: #(
>     'HelpSystem-Tests'
> ).
> "Save and Quit"
> WorldState addDeferredUIMessage: [
>     SmalltalkImage current snapshot: true andQuit: true ].
>
> or something more succinct:
>
> HDTestReport runPackages: ((WADevelopment allPackages
>     select: [ :each | each name includesSubString: '-Tests-' ])
>     collect: [ :each | each name ]).
>
>
> <the simple answer>
> tests.st is in the FTP directory. It contains the content above. Add test
> names as strings in the array like 'HelpSystem-Tests'. Or, iterate over some
> packages for '-Tests-'
> </the simple answer>
>
> <longer explanation/probably a rant>
> I don't think GitHub has anything to do with it. The Pharo Jenkins server
> doesn't use it. That's just where Lukas decided to make his scripts
> available.
>
> https://github.com/renggli/builder/tree/master/scripts
> https://github.com/renggli/builder/blob/master/build.sh
>
> Lukas has a system with two parts: scripts; and, the build.sh shell script.
> There are loads of scripts lying around on the Pharo Jenkins server. They
> are attached and combined dynamically by a ~160 line shell script called
> build.sh. So you'd write something like this:
>
> build.sh -i Omnibrowser -i Greco -i Roman -i Etc
>
> That collects a bunch of scripts. All of them look like the content above:
> string names in an array. build.sh merges these scripts elegantly into one
> single stream. There is no tangible difference between that and having a
> file called tests.st where you cut and paste in the test names as strings.
>
> Is Lukas's system elegant? Yes. Do I want to wade through a ~160 shell
> script to faithfully reproduce that? Let me put it this way: why would
> anybody write a shell script over ~100 lines? Even if it was an init script?
> It seems to me build.sh could have been written in another scripting
> language and been  both shorter and easier to comprehend for the next guy
> who came along. It's not 1983.
> </longer explanation/probably a rant>

I'd understood Sean to be saying "putting your build system under
version control is a good idea", and it is: it makes for repeatable
builds. Now in an ideal world we'd have some way of saying "take this
base image, run it on that virtual machine, and here's a script
telling you what tests to run". We do the building part very simply,
just by taking what's in the Workspace and sticking it in a git repo.
Jenkins says "check out the repo, and run this file", and suddenly we
need zero Jenkins configuration to change the VM or base image, and we
have tracking of what ran against what, and who made it that way. We
can tackle dependency management another day, maybe by letting Dale
Henrichs solve the problem for us! (See his work with the Travis CI
stuff.)

Otherwise, I'd agree with you that Lukas' build script comes with a
large dose of intimidation: shell scripts suck, and their suckage
increases non-linearly in their length.

frank

> Chris
>
>
>
> "I don't think you can do that via the Jenkins web UI. Putting the scripts
> on github like Lukas did makes contributing convenient. Then you can have a
> really simple Jenkins job that just downloads the scripts and runs one."
>
> This is interesting, but I don't think it works that way. If I understand
> what you're saying, build.sh would pull down scripts from Github. I don't
> think that's even possible. Why does Github use SSL?
> To password protect scripts it gives away for free? I don't think so. Only
> humans can download content from Github. Try putting a picture gif on Github
> and then link to it from the webpage you're building. Github uses SSL to
> enforce an interface. They don't want people linking to its resources. Pharo
> Jenkins has all the scripts locally in a directory somewhere. I don't think
> you can clone individual files on demand programmaticlly. I'm willing to be
> wrong, but I feel confident that things are as I just described.
>
>
>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
In reply to this post by Chris Cunnington
My previous mail forgot one important thing:

On 26 July 2012 15:17, Chris Cunnington <[hidden email]> wrote:

> HDTestReport runPackages: #(
>     'HelpSystem-Tests'
> ).
> "Save and Quit"
> WorldState addDeferredUIMessage: [
>     SmalltalkImage current snapshot: true andQuit: true ].
>
> or something more succinct:
>
> HDTestReport runPackages: ((WADevelopment allPackages
>     select: [ :each | each name includesSubString: '-Tests-' ])
>     collect: [ :each | each name ]).
>
>
> <the simple answer>
> tests.st is in the FTP directory. It contains the content above. Add test
> names as strings in the array like 'HelpSystem-Tests'. Or, iterate over some
> packages for '-Tests-'
> </the simple answer>

I get the idea of "this is autoconfiguring", but I don't like it. The
only way you can see what tests ran is by looking at the output. Well,
OK, that might be well and good, but it means you have to look at
Jenkins and poke around. It means you can't easily exclude a package
of tests that simply can't have its tests run today, but we don't want
to remove the package from the base image. (Also, I like
#select:thenCollect:.)

frank

> <longer explanation/probably a rant>
> I don't think GitHub has anything to do with it. The Pharo Jenkins server
> doesn't use it. That's just where Lukas decided to make his scripts
> available.
>
> https://github.com/renggli/builder/tree/master/scripts
> https://github.com/renggli/builder/blob/master/build.sh
>
> Lukas has a system with two parts: scripts; and, the build.sh shell script.
> There are loads of scripts lying around on the Pharo Jenkins server. They
> are attached and combined dynamically by a ~160 line shell script called
> build.sh. So you'd write something like this:
>
> build.sh -i Omnibrowser -i Greco -i Roman -i Etc
>
> That collects a bunch of scripts. All of them look like the content above:
> string names in an array. build.sh merges these scripts elegantly into one
> single stream. There is no tangible difference between that and having a
> file called tests.st where you cut and paste in the test names as strings.
>
> Is Lukas's system elegant? Yes. Do I want to wade through a ~160 shell
> script to faithfully reproduce that? Let me put it this way: why would
> anybody write a shell script over ~100 lines? Even if it was an init script?
> It seems to me build.sh could have been written in another scripting
> language and been  both shorter and easier to comprehend for the next guy
> who came along. It's not 1983.
> </longer explanation/probably a rant>
>
> Chris
>
>
>
> "I don't think you can do that via the Jenkins web UI. Putting the scripts
> on github like Lukas did makes contributing convenient. Then you can have a
> really simple Jenkins job that just downloads the scripts and runs one."
>
> This is interesting, but I don't think it works that way. If I understand
> what you're saying, build.sh would pull down scripts from Github. I don't
> think that's even possible. Why does Github use SSL?
> To password protect scripts it gives away for free? I don't think so. Only
> humans can download content from Github. Try putting a picture gif on Github
> and then link to it from the webpage you're building. Github uses SSL to
> enforce an interface. They don't want people linking to its resources. Pharo
> Jenkins has all the scripts locally in a directory somewhere. I don't think
> you can clone individual files on demand programmaticlly. I'm willing to be
> wrong, but I feel confident that things are as I just described.
>
>
>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Sean P. DeNigris
Administrator
In reply to this post by Chris Cunnington
Chris Cunnington wrote
<longer explanation/probably a rant>
I don't think GitHub has anything to do with it. The Pharo Jenkins
server doesn't use it.
Yes, that did seem a little ranty ;-) Pharo uses gitorious, for reasons you can read about on vm-dev, which I don't think apply here. I suggested github because I find it has better collaboration features

Chris Cunnington wrote
This is interesting, but I don't think it works that way... I'm willing to be wrong, but I feel confident that
things are as I just described.
That's how it works. You can check the scripts-download project [1] which seems to use download.sh [2] to download the files via wget. If you want to cherry-pick individual files, see [3].

The passion is great, but I could have saved you a lot of typing if you had just asked ;-)

[1] https://ci.lille.inria.fr/pharo/job/Scripts-download/
[2] https://ci.lille.inria.fr/pharo/job/Scripts-download/ws/download.sh/*view*/
[3]http://stackoverflow.com/questions/9159894/download-specific-files-from-github-in-command-line-not-clone-the-entire-repo
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Sean P. DeNigris
Administrator
In reply to this post by Frank Shearar-3
Frank Shearar-3 wrote
I'd agree with you that Lukas' build script comes with a
large dose of intimidation: shell scripts suck, and their suckage
increases non-linearly in their length.
I'm not sure how we got to this point in the discussion. I definitely don't think we should adopt Lukas's system. I also agree that shell scripts suck (that's why I'm developing a Jenkins Smalltalk framework that uses the image for 99% of the UI and scripting). What I'm saying is - if we're gonna write shell scripts, keeping them version controlled and universally-contributable-to instead of hidden away on the server might be beneficial.

Cheers,
Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Bert Freudenberg
On 26.07.2012, at 07:50, Sean P. DeNigris wrote:

> if we're gonna write
> shell scripts, keeping them version controlled and
> universally-contributable-to instead of hidden away on the server might be
> beneficial.


+1

(even if we end up having most of the build stuff in Smalltalk, and the scripts need only be tweaked every other year, not having them under version control is just a bad idea)

- Bert -



Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Chris Cunnington
In reply to this post by Chris Cunnington
This topic is starting to feel kind of loaded. I suspect that's because
it's an idea that's been floating around for a long time and has now
just landed as something real.

I've pulled a tiny piece of functionality from a very large rig. I feel
responsible for that small part. And, I think that is enough for Frank
to work on a single image. The aim is so Frank can generate a page like
this:

https://ci.lille.inria.fr/pharo/view/Cog/job/StackVM-Mac-Esteban-Tests/5/testReport/

If this kernel of functionality works and other people want to evolve it
into something higher. That's great. If and when that happens, and
people want to start zipping, scripting, and Gighubbing, I'll be by the
pool.

We can move from this server, which I will shut down in the next few
months, and build a better version of Jenkins on the server we're moving
to.

Thanks,
Chris


(Although I have to admit it makes me nuts that Sean always seems to
know more that I do, I feel I must say: "Well played, sir!".)

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
On 26 July 2012 18:47, Chris Cunnington <[hidden email]> wrote:

> This topic is starting to feel kind of loaded. I suspect that's because it's
> an idea that's been floating around for a long time and has now just landed
> as something real.
>
> I've pulled a tiny piece of functionality from a very large rig. I feel
> responsible for that small part. And, I think that is enough for Frank to
> work on a single image. The aim is so Frank can generate a page like this:
>
> https://ci.lille.inria.fr/pharo/view/Cog/job/StackVM-Mac-Esteban-Tests/5/testReport/
>
> If this kernel of functionality works and other people want to evolve it
> into something higher. That's great. If and when that happens, and people
> want to start zipping, scripting, and Gighubbing, I'll be by the pool.
>
> We can move from this server, which I will shut down in the next few months,
> and build a better version of Jenkins on the server we're moving to.
>
> Thanks,
> Chris
>
>
> (Although I have to admit it makes me nuts that Sean always seems to know
> more that I do, I feel I must say: "Well played, sir!".)

Oh hey, I don't mean to suggest (nor, I hope, is anyone else) that we
need great big shiny edifices built today. For now, today, this is
enough to get me going.

Someone built me a CI server and a skeleton job I can flesh out. I'm
happy! Thanks, Chris!

frank

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
In reply to this post by Bert Freudenberg
On 26 July 2012 17:30, Bert Freudenberg <[hidden email]> wrote:

> On 26.07.2012, at 07:50, Sean P. DeNigris wrote:
>
>> if we're gonna write
>> shell scripts, keeping them version controlled and
>> universally-contributable-to instead of hidden away on the server might be
>> beneficial.
>
>
> +1
>
> (even if we end up having most of the build stuff in Smalltalk, and the scripts need only be tweaked every other year, not having them under version control is just a bad idea)

https://github.com/frankshearar/squeak-ci contains what ought to be a
sensible environment. I wasn't sure what version of Cog was/is in the
Jenkins Workspace, so I just took the latest release - r2562.

frank

> - Bert -
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Sean P. DeNigris
Administrator
Frank Shearar-3 wrote
https://github.com/frankshearar/squeak-ci contains what ought to be a
sensible environment
Cool! Nice start. One helpful thing is that Jenkins sets some environment variables for you, so you can use e.g. $WORKSPACE in your scripts instead of hardcoding the path, making it easy to move things around on the server.
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Sean P. DeNigris
Administrator
Sean P. DeNigris wrote
you can use e.g. $WORKSPACE in your scripts instead of hardcoding the path, making it easy to move things around on the server.
I made a pull request on github with the changes...

Cheers,
Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
On 27 July 2012 13:25, Sean P. DeNigris <[hidden email]> wrote:
>
> Sean P. DeNigris wrote
>>
>> you can use e.g. $WORKSPACE in your scripts instead of hardcoding the
>> path, making it easy to move things around on the server.
>>
>
> I made a pull request on github with the changes...

That's what happens when I'm too slow to commit! I'm making some other
changes as well: saving snapshots as a new image, for starters.

Thanks!

frank

> Cheers,
> Sean
>
>
>
> --
> View this message in context: http://forum.world.st/Squeak-release-process-CI-server-tp4640962p4641803.html
> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Sean P. DeNigris
Administrator
Frank Shearar-3 wrote
That's what happens when I'm too slow to commit! I'm making some other
changes as well: saving snapshots as a new image, for starters.
One cool thing about git/github is that there's no penalty for committing early and often. Proposed changes can even be discussed right in the pull request. Ordinarily, I would've created an issue on github, and then a branch with the issue name, e.g. issue_14, and committed to the branch to keep master clean, but I was lazy ;-) Although you can see in the pull request that I missed one hard-coded path and made another commit, which appeared in the pull request with the first one.

Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
On 27 July 2012 13:36, Sean P. DeNigris <[hidden email]> wrote:

>
> Frank Shearar-3 wrote
>>
>> That's what happens when I'm too slow to commit! I'm making some other
>> changes as well: saving snapshots as a new image, for starters.
>>
>
> One cool thing about git/github is that there's no penalty for committing
> early and often. Proposed changes can even be discussed right in the pull
> request. Ordinarily, I would've created an issue on github, and then a
> branch with the issue name, e.g. issue_14, and committed to the branch to
> keep master clean, but I was lazy ;-) Although you can see in the pull
> request that I missed one hard-coded path and made another commit, which
> appeared in the pull request with the first one.

Sure: github's my bread & butter in my day job. I meant that while I
was doing the WORKSPACE changes I got sidetracked into the
snapshotting thing, and was going to commit THEN, but you beat me to
the punch!

frank

> Sean
>
>
>
> --
> View this message in context: http://forum.world.st/Squeak-release-process-CI-server-tp4640962p4641808.html
> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Sean P. DeNigris
Administrator
Frank Shearar-3 wrote
github's my bread & butter in my day job
Fantastic! We're going to need some git/github expertise when Dale releases the new Metacello... any minute now...
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Squeak release process - CI server

Frank Shearar-3
In reply to this post by Frank Shearar-3
On 27 July 2012 13:27, Frank Shearar <[hidden email]> wrote:

> On 27 July 2012 13:25, Sean P. DeNigris <[hidden email]> wrote:
>>
>> Sean P. DeNigris wrote
>>>
>>> you can use e.g. $WORKSPACE in your scripts instead of hardcoding the
>>> path, making it easy to move things around on the server.
>>>
>>
>> I made a pull request on github with the changes...
>
> That's what happens when I'm too slow to commit! I'm making some other
> changes as well: saving snapshots as a new image, for starters.
>
> Thanks!

OK, so we have a git repository containing the bits necessary for
running CI tests. I don't know if we have two missing ingredients: git
installed on squeak-ci, and the jenkins plugin for git. I _may_ be
able to install the latter (I haven't tried), given that I have
Jenkins credentials. Who might I presume upon to install a recent git?
(It ought to be as simple as an apt-get.) Chris Cunnington?

frank


> frank
>
>> Cheers,
>> Sean
>>
>>
>>
>> --
>> View this message in context: http://forum.world.st/Squeak-release-process-CI-server-tp4640962p4641803.html
>> Sent from the Squeak - Dev mailing list archive at Nabble.com.
>>

12