Re: Build.squeak.org and squeaksource.com in danger (was Re: [Box-Admins] Disk space usage on box3)

> That's very cool, but it's not the zips we're talking about. We're
> talking about the artifacts produced by Jenkins, which are Squeak
> images. I wrote the release process to produce a series of artifacts
> with different names. Because nothing gets replaced/overwritten, disk
> usage is unbound.

The rate of untamed growth is too fast.

Automating something (builds) shouldn't introduce a new manual
process.  There should be something to keep the size reasonable so we
don't have to remember do it manually every month..

On 8 December 2013 20:57, Chris Muller <[hidden email]> wrote:
>> That's very cool, but it's not the zips we're talking about. We're
>> talking about the artifacts produced by Jenkins, which are Squeak
>> images. I wrote the release process to produce a series of artifacts
>> with different names. Because nothing gets replaced/overwritten, disk
>> usage is unbound.
>
> The rate of untamed growth is too fast.

I have no idea what you mean by that. I don't consider ~30MB of new
data to be "untamed growth", nor do I consider the accumulation of a
mere 6 GB over, what, 600 builds? to be "too fast".

Er. I explicitly decided to produce versioned artifacts that a human
would assess for quality. I stand by that decision. You _should_ be
able to easily go and say "Well, version N is crap, but N - 2 is just
fine. Let's just go with that."

What's crazy is that we don't have graphs that we can use to easily
track disk usage, nor alerting mechanisms other than Ken to warn of
impending doom. Munin and Icinga would do fine, with Apache/nginx to
serve up status pages. If only someone with sufficient time had the
energy to put in the graft necessary to set up these standard sysadmin
tools.

frank

On Sun, Dec 08, 2013 at 09:02:28PM +0000, Frank Shearar wrote:
>
> Er. I explicitly decided to produce versioned artifacts that a human
> would assess for quality. I stand by that decision. You _should_ be
> able to easily go and say "Well, version N is crap, but N - 2 is just
> fine. Let's just go with that."


I want to ask specifically with respect to the SqueakTrunk job, which is
set up to do this:

  Take a base image (currently 4.5-12565), update it, archive the result.
  Run the entire suite of in-image tests.

For that job, do we want to archive all build artifacts, or is it instead
sufficient to discard all but the last successful/stable artifact to save
disk space?

I would like to make the following changes to the SqueakTrunk job
configuration:

1) Archive just the TrunkImage.zip, not all of TrunkImage.*

2) Enable the "Discard all but the last successful/stable artifact to save
disk space" option.

3) Enable the "Abort the build if it's stuck" option, with a timeout
of 30 minutes.

Is it OK to do this for the SqueakTrunk job?

Dave

On Sun, Dec 8, 2013 at 3:02 PM, Frank Shearar <[hidden email]> wrote:
Ken should not have to keep reminding us about space.  We don't need
to keep 600 builds, and I respect Ken's concern about proactive
management.

Of course, so may we discuss something somewhere _in-between_ 600 and
none?  Are the last 10 builds enough?  50?

> What's crazy is that we don't have graphs that we can use to easily
> track disk usage, nor alerting mechanisms other than Ken to warn of
> impending doom. Munin and Icinga would do fine, with Apache/nginx to
> serve up status pages. If only someone with sufficient time had the
> energy to put in the graft necessary to set up these standard sysadmin
> tools.

Space monitoring and alerts are required for any 24/7 system
irregardless of the level of sustainability of processes running on
the machine.  They're both important, but since we don't have alerts,
sustainability is all the more important.

You're now sharing box3 with SqueakSource.  The best approach is for
everyone with access to the box to keep their resource impact
reasonable enough that it doesn't affect service or require alert
notices from others.

Thanks again.

On 8 December 2013 22:18, David T. Lewis <[hidden email]> wrote:
I did this over the weekend. But I just noticed that I didn't
configure Jenkins to actually use the new zipfile. It does now.

> 2) Enable the "Discard all but the last successful/stable artifact to save
> disk space" option.

I specifically don't want to archive only the last successful build
because that makes it harder for us to debug problems. Imagine that
build 101 works, 102 doesn't, and 103 does. It turns out that 103
"succeeds" because of a completely bogus reason, like a bug in the CI
scripts. We need to see the last known good state, 101. But alas, it's
gone.

This is not what caused the disk usage problem.

> 3) Enable the "Abort the build if it's stuck" option, with a timeout
> of 30 minutes.

That's fine, but wouldn't have helped any time the build hung in the
past, because those hung processes were orphaned anyway. We have used
this feature, in fact, We don't anymore because you end up having two
separate places to define timeouts. Eventually it _will_ be useful,
once we teach Squeak to fail, rather than hang.

> Is it OK to do this for the SqueakTrunk job?

frank

> Dave
>
>

On 8 December 2013 22:20, Chris Muller <[hidden email]> wrote:
No, Ken absolutely should _not_ have to keep reminding us about
anything. But you are not suggesting a solution. The solution is this:
more people should be doing more to keep an eye on the box.

I am more than happy to hear suggestions on how to do that. One
suggestion is this: someone periodically takes the latest
ReleaseSqueakTrunk artefact, tries it out, blesses it by pushing it to
ftp.squeak.org, and then tells us. Some kind soul throws away the old
potential release candidates.

We don't have a 24/7 system, because those require operational staff.

Seriously, the big problem here is not that ReleaseSqueakTrunk did
what I told it to. The big problem here is that no one except Ken
cares enough about box3 to monitor it.

frank

> You're now sharing box3 with SqueakSource.  The best approach is for
> everyone with access to the box to keep their resource impact
> reasonable enough that it doesn't affect service or require alert
> notices from others.
>
> Thanks again.

I want to respond to some comments made in this thread.

First I want to admit that my posting on Friday was a bit shrill.  I was
getting frustrated that this was my third or fourth post on this
increasingly problematic issue and little if any action had been taken.
  Further there seemed a real possibility that over the next few days,
possibly over the weekend when I had little time to provide assistance,
that the file system for the server which hosts both build.squeak.org
and squeaksource.com would fill up.  I have seen greater than 1% per 24
hour increases on that server in the past.

Thanks to Frank the immediate issue has been addressed and hopefully we
have a couple of weeks of breathing time now to consider how best to
avoid the issue in the future.

There has been some discussion regarding my admittedly somewhat extreme
comments regarding squeaksource.com.  One thing that has been mentioned
is the idea that 'disk space is cheap'.  I think that is easy to say and
true in general, but I'm not sure it is true in this specific case.  I
will admit to possibly over-estimating the 'cost' but...  Keep in mind
that we have no direct control over the configuration of either
box3.squeak.org and box4.squeak.org.  These were contributed to us by
Gandi.net at the request of the Software Freedom Conservancy.  Neither I
nor anyone else in our community has any access to modify the server
configuration and do things like add disk space.  At best we have to go
through Software Freedom Conservancy for this.  They don't have a lot of
time to spare to such issues themselves, further I don't think we should
make assumptions that Gandi.net is going to be willing to donate more
resources.  I'm not sure it is even easy to throw money at the issue
given the fact that we are using donated resources.  But then, I may
just be unreasonably pessimistic about this.

Someone kindly thanked me and gave the impression that I was the only
one that 'cared' enough to monitor the servers for such issues.  Thanks
but don't assign me too much altruism or think that I'm so interested.
The minor amount of daily server checking I do is largely habit for me
and is an easy way for me to trigger a few endorphins and feel like I
have in some way contributed for the day.

To be honest my interest in Squeak and the community has been waning for
some time and is quite low at this point.  Don't assume I'm going to
continue to do what little I do indefinitely.  Someone else must step up
to take responsibility for the Squeak servers.

Ken