trunk process resilience

On 7 November 2013 21:07, Chris Muller <[hidden email]> wrote:
I think before any person pitches in with any suggestion, that person
should go read up on handling state in a distributed system. (Because
having a second copy in a kind've active-active replication thing is
exactly a distributed system.) It is _not_easy_. (And "Never go to sea
with two chronometers; take one or three".) Here's a good starting
point: http://aphyr.com/posts/281-call-me-maybe-carly-rae-jepsen-and-the-perils-of-network-partitions

frank

It's nice to read someone thinking about the issues faced by networked
programs.  In fact, the little experiment he does (under the heading
"A simple distributed system"), is exactly what one of Magma's HA
test-cases performs -- multiple clients perform rapid-fire commits as
fast as they can, counting upward, while the servers in the HA cluster
undergo various role changes due to HA events like arbitrarily killing
one of the servers with quitPrimitive.  It's quite a piece [1].

Thankfully, none of that applies to what's being proposed here, the
operations needed to achieve a mutual backup are idempotent -- simply
a package copy from remote to local using the existing
MCRepository>>#copyAllFrom:.  So, it uses existing error-handling too,
what could go wrong?

Under normal usage, the same person would not commit two different
UUID versions of a package, but with the same exact name, to each
repository.  But, even if they did, it's no different than when that
happens today between projects which, themselves, are simply different
repositories.

I've seen how fragile and unsustainable our source.squeak.org server
is.  I want to inform the community what I've done and solicit
pragmatic discussion on how we can get more out of it.

Thanks.

[1] -- http://wiki.squeak.org/squeak/6101

On Thu, Nov 7, 2013 at 3:33 PM, Frank Shearar <[hidden email]> wrote:

On Thu, Nov 07, 2013 at 03:07:59PM -0600, Chris Muller wrote:
I like the idea of building some resilience into the SqueakSource servers.
I also like the idea of using Magma to support this, because I know that
Magma has been used to address similar issues on much larger scale systems.

I do have some concerns of a non-technical nature:

1) From an operational point of view, we need to keep our systems as simple
as possible. There are very few people supporting the servers, and their
availability comes and goes over time, so we need to keep things simple
enough that any box-admins person can always figure out how to get things
running even if the expert is not available.

2) We need to be careful not to add more failure modes than we remove. This
is a painfully common mistake, in which people add high availability features
to an existing system with the result that new failure modes are introduced
that turn out to be worse than the failure modes that they were attempting
to mitigate.

As an example, I would point to the recent downtime on SmalltalkHub
(see the excellent recap provided by Philippe Marschall at
https://github.com/blog/1346-network-problems-last-friday). The system
had availability problems for an extended period of time, and the cause
was a (human error induced) failure in some redundant networking gear.
The high availability networking introduced additional failure modes, and
the combination of human error and system complexity reduced the resilience
of the system as a whole.

This is meant only as a cautionary note. I really *do* like the idea of
building in some redundancy, and I think that the work you (Chris) have
done with box4.squeak.org might be a good way to do it.

>
> So, I guess I'm proposing that we have some elements in the image "aware"
> of a second trunk.  But before wrangling out exactly what form that
> awareness would take, what do you think so far?
>

We should keep any changes in the image to a minimum, but the general idea
sounds good to me.

Dave

Thanks for the great discussion Dave.

> I like the idea of building some resilience into the SqueakSource servers.
> I also like the idea of using Magma to support this, because I know that
> Magma has been used to address similar issues on much larger scale systems.

We don't need to use Magma at all to accomplish the redundancy I'm proposing.

The work I did to use a Magma backend for SqueakSource on box4 is
solely for reliable persistence and to support the history function in
the IDE.  Nothing more.  Its HA function is not being used at all, in
fact Magma is being used by the webserver in "local" (direct-connect,
single-user) mode.

> I do have some concerns of a non-technical nature:
>
> 1) From an operational point of view, we need to keep our systems as simple
> as possible. There are very few people supporting the servers, and their
> availability comes and goes over time, so we need to keep things simple
> enough that any box-admins person can always figure out how to get things
> running even if the expert is not available.

Agreed.

> 2) We need to be careful not to add more failure modes than we remove. This
> is a painfully common mistake, in which people add high availability features
> to an existing system with the result that new failure modes are introduced
> that turn out to be worse than the failure modes that they were attempting
> to mitigate.

Agreed.

> As an example, I would point to the recent downtime on SmalltalkHub
> (see the excellent recap provided by Philippe Marschall at
> https://github.com/blog/1346-network-problems-last-friday). The system
> had availability problems for an extended period of time, and the cause
> was a (human error induced) failure in some redundant networking gear.
> The high availability networking introduced additional failure modes, and
> the combination of human error and system complexity reduced the resilience
> of the system as a whole.

You said SmalltalkHub but the link was about GitHub (an interesting
story, nonetheless).  Is Philippe Marschall working at GitHub?

I'll submit a proposal to the Inbox which will clarify exactly what
I'm proposing.

Thanks.