[squeak-dev] Community Server Report

Cross your fingers, with luck we should be good now.

For over a month we've had trouble with the community server locking up
and requiring a reboot.  On Friday Hetzner did some testing on the
server and, as I understand, traded out the RAM at least once (I believe
this has been done before as well).  From then on the server was up in
some sense, but not working properly.

Unfortunately we don't have weekend support so there was a delay.
Additionally there was some confusion that resulted in further delays.
On Tuesday Marcus Denker, our contact with Hetzner, convinced them to
hook up a 'virtual console' to the system which allowed us to see the
console output.  Cees de Groot and Goran Krampe were able to run some
tests that indicated that the RAM was bad.  This convinced Hetzner to
change the RAM again.  After this the server booted up and ran fine for
a few hours.

Then it went down again as it has for weeks now.  After the latest
trouble Hetzner was convinced to do more and took the hard drive from
the server and put it in another server with identical hardware.  And we
are up again.

Hopefully this takes care of us for a while.  But I think there are some
things we can learn from this event and I believe some changes need to
be made to how we manage the server.  I'm not sure what to do right now.
 But I'm thinking about it and sometime soon I would like to start a
discussion of the future of the server.

Ken

@Ken, Goran, Cees, Marcus, ...
Thanks to all of you for taking care of and fixing the server.

It had been a rough weekend without the mailing lists ;-)

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

On Thu, Oct 23, 2008 at 08:38:59AM +0200, Lukas Renggli wrote:
> @Ken, Goran, Cees, Marcus, ...
> Thanks to all of you for taking care of and fixing the server.
>
> It had been a rough weekend without the mailing lists ;-)

Indeed, thank you very much for your work. Not just for fixing
things, but also for all of the time that the servers hum along
uneventfully and we all take it for granted and forget to say
"thanks".

Dave

--
http://people.squeakfoundation.org/person/mikevdg
http://gulik.pbwiki.com/

Michael van der Gulik schrieb: +1
>
> May I recommend that we host the mailing list on another machine than our
> main server? A cheap VPS server would manage this fine. Most services we can
> afford to lose for a while, but it was a bit unnerving not being able to see
> what was happening.

Well it was possible to get informed about the "rescue mission" on irc.

So maybe a message like:
in case of the mailimg list beeing down,
try to get uptodate information on irc.freenode.net: #squeak

can be added to the welcome message of the mailinglist.

Rudolph
--
>
> Gulik.
>
>
>
>
> ------------------------------------------------------------------------
>
>

Hi!

rudolf-Rednose wrote:
> Michael van der Gulik schrieb:
>> May I recommend that we host the mailing list on another machine than our
>> main server? A cheap VPS server would manage this fine. Most services we can
>> afford to lose for a while, but it was a bit unnerving not being able to see
>> what was happening.
>
> Well it was possible to get informed about the "rescue mission" on irc.

One thing is that we are most probably moving DNS to zoneedit or similar
so that we can easily point www.squeak.org to another ip during an event
like this. That should easily fix the "we don't know what is going
on"-problem.

Now we couldn't easily do that unless involving Dan etc.

regards, Göran

On Mon, 2008-10-27 at 11:15 +0100, Göran Krampe wrote: Just a little correction.  We are adding zoneedit.com as additional
secondary DNS servers, not moving it.  The community server will still
be the primary DNS server.  But if it is down for an extended time the
zoneedit configuration can be manually modified to point to a different
server temporarily.

> Now we couldn't easily do that unless involving Dan etc.

I've emailed Dan.  We just have to be patient.

Ken

> regards, Göran
>
>
>