Making trunk repo faster

Accessing the repository gets slower and slower due to the ever increasing number of versions there.

To make it faster we could remove old package versions. Say, everything older than a year (but keep at least one version of each package). We could move those to an "attic" repository.

Sounds good? Other ideas?

- Bert -

I know that I have projects in production that are using code that is older than 1 year ... it would not be easy for me to scramble and identify all of the old packages, configurations and/or build scripts that would have to be modified if you were to start removing packages from the repository ... the net would be worse than "slow repository" or "down for the weekend repository", but of course the only time I would need the old code is in an emergency...

A repository like love is supposed to be forever:)

A more palatable solution would be to have the active projects move to a different (smaller) project with fewer versions, the older infrequently used projects wouldn't be triggered and the new more active projects wouldn't carry the burden of ancient history around...

My $0.02...

Dale

----- Original Message -----
| From: "Bert Freudenberg" <[hidden email]>
| To: "The general-purpose Squeak developers list" <[hidden email]>
| Sent: Wednesday, May 23, 2012 10:28:29 AM
| Subject: [squeak-dev] Making trunk repo faster
|
| Accessing the repository gets slower and slower due to the ever
| increasing number of versions there.
|
| To make it faster we could remove old package versions. Say,
| everything older than a year (but keep at least one version of each
| package). We could move those to an "attic" repository.
|
| Sounds good? Other ideas?
|
| - Bert -
|
|
|

Hi Dale,

that's interesting. I wouldn't have expected anyone to use our trunk repository as a production host.

So you are linking to specific filenames in http://source.squeak.org/trunk/ ? Do you have an example somewhere?

- Bert -

On 23.05.2012, at 19:37, Dale Henrichs wrote:

--
best,

On 23 May 2012 19:37, Dale Henrichs <[hidden email]> wrote:
> I know that I have projects in production that are using code that is older than 1 year ... it would not be easy for me to scramble and identify all of the old packages, configurations and/or build scripts that would have to be modified if you were to start removing packages from the repository ... the net would be worse than "slow repository" or "down for the weekend repository", but of course the only time I would need the old code is in an emergency...
>
> A repository like love is supposed to be forever:)
>
> A more palatable solution would be to have the active projects move to a different (smaller) project with fewer versions, the older infrequently used projects wouldn't be triggered and the new more active projects wouldn't carry the burden of ancient history around...
>

Yep, pharo deemed to use separate repository per release version..
because the number of versions is
sky rocketing and MC started to get connection timeouts due to slow
server response...

> My $0.02...
>
> Dale
>

--
Best regards,
Igor Stasenko.

Oh. On a second reading I think you are confusing this with http://squeaksource.com/ .

Besides being different servers, it's a clash of terminology: What Monticello calls a "repository", SqueakSource calls a "project". Trunk and Inbox are Squeaks two main repositories, which are hosted as two projects on http://source.squeak.org/ . Makes sense?

But my question stands: are you referencing specific package versions in our trunk repository ("project")?

- Bert -

On 23.05.2012, at 19:48, Bert Freudenberg wrote:

Igor's suggestion is worth considering. One could make a new trunk repository for each release.

2012/5/23 Bert Freudenberg <[hidden email]>:
I would prefer a delimitation per release than per year...
The previous release files would stay in trunk until next release...

I would also duplicate the last file if release n-1 as first file in release n.

Nicolas

On 23.05.2012, at 20:14, Nicolas Cellier wrote:

What about Eliot's argument that limiting by year makes it simpler to find the right repository for a specific version?

- Bert -

I misunderstood ... I thought you were talking about SqueakSource .... Never Mind:)

Dale

----- Original Message -----
| From: "Bert Freudenberg" <[hidden email]>
| To: "The general-purpose Squeak developers list" <[hidden email]>
| Sent: Wednesday, May 23, 2012 10:48:58 AM
| Subject: Re: [squeak-dev] Making trunk repo faster
|
| Hi Dale,
|
| that's interesting. I wouldn't have expected anyone to use our trunk
| repository as a production host.
|
| So you are linking to specific filenames in
| http://source.squeak.org/trunk/ ? Do you have an example somewhere?
|
| - Bert -
|
| On 23.05.2012, at 19:37, Dale Henrichs wrote:
|
| > I know that I have projects in production that are using code that
| > is older than 1 year ... it would not be easy for me to scramble
| > and identify all of the old packages, configurations and/or build
| > scripts that would have to be modified if you were to start
| > removing packages from the repository ... the net would be worse
| > than "slow repository" or "down for the weekend repository", but
| > of course the only time I would need the old code is in an
| > emergency...
| >
| > A repository like love is supposed to be forever:)
| >
| > A more palatable solution would be to have the active projects move
| > to a different (smaller) project with fewer versions, the older
| > infrequently used projects wouldn't be triggered and the new more
| > active projects wouldn't carry the burden of ancient history
| > around...
| >
| > My $0.02...
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "Bert Freudenberg" <[hidden email]>
| > | To: "The general-purpose Squeak developers list"
| > | <[hidden email]>
| > | Sent: Wednesday, May 23, 2012 10:28:29 AM
| > | Subject: [squeak-dev] Making trunk repo faster
| > |
| > | Accessing the repository gets slower and slower due to the ever
| > | increasing number of versions there.
| > |
| > | To make it faster we could remove old package versions. Say,
| > | everything older than a year (but keep at least one version of
| > | each
| > | package). We could move those to an "attic" repository.
| > |
| > | Sounds good? Other ideas?
| > |
| > | - Bert -
| > |
| > |
| > |
| >
|
|
|

2012/5/23 Bert Freudenberg <[hidden email]>:
Maybe...

We will need richer criteria than simply age.  Consider a package that
didn't change in the last year -- purging it would provide little gain
while seriously limiting the repository's ability to do its SCM duty
(compare, merge, revert, etc. with prior versions).

Before we knee-jerk react by hacking down our persistent model just to
accomodate MC's unsustainable design, may we first at least consider
to upgrade MC to be more scalable?

For example, one of the things that is "slow" is due to merely
applying the underlining and highlighting in the Versions browser.
One thing that can be done here is quit going back all versions and
instead just the prior 20 or so...




On Wed, May 23, 2012 at 12:28 PM, Bert Freudenberg <[hidden email]> wrote:
> Accessing the repository gets slower and slower due to the ever increasing number of versions there.
>
> To make it faster we could remove old package versions. Say, everything older than a year (but keep at least one version of each package). We could move those to an "attic" repository.
>
> Sounds good? Other ideas?
>
> - Bert -
>
>

On 23.05.2012, at 21:28, Chris Muller wrote:


I don't think that's enough. Try clicking the trunk repository on the website.

- Bert -

That's why I said "for example", it's not a silver bullet.  My only
point was that unless we address the inherent scalability issue then
the suffering will be on-going even as we scramble to discuss it and
move files around and then hunt for them later for basic SCM needs.
Seems worthy of a little thoughtful discussion at least..


On Wed, May 23, 2012 at 2:32 PM, Bert Freudenberg <[hidden email]> wrote:

On 23.05.2012, at 22:05, Chris Muller wrote:

> That's why I said "for example", it's not a silver bullet.  My only
> point was that unless we address the inherent scalability issue then
> the suffering will be on-going even as we scramble to discuss it and
> move files around and then hunt for them later for basic SCM needs.
> Seems worthy of a little thoughtful discussion at least..


Agreed. So, as I wrote in the original post: "Other ideas?" :)

- Bert -

>> That's why I said "for example", it's not a silver bullet.  My only
>> point was that unless we address the inherent scalability issue then
>> the suffering will be on-going even as we scramble to discuss it and
>> move files around and then hunt for them later for basic SCM needs.
>> Seems worthy of a little thoughtful discussion at least..
>
> Agreed. So, as I wrote in the original post: "Other ideas?" :)

At a minimum please start more conservatively.  Older than two years
but ensuring to keep a minimum of 10 versions of each package
regardless of age.  That way we can at least keep SOME history easily
accessible for all packages and let's see how much speed that gains
us.

The older ones should still be accessible in a separate repository.

===

As far as other ideas:  Change SqueakSource's implementation to stop
using MCFileBasedRepository which I suspect is the root of the
scalability issue.  A Magma-based MC backend performs steadily
regardless of size, I'd love to test this if someone can get
SqueakSource going on Squeak 4.3.  I also know there are other
solutions like ss3 and SmalltalkHub although SH may be too different
to easily migrate..

On 23.05.2012, at 22:52, Chris Muller wrote:

But you are aware that all the history meta data is in each and every MC package? How often do you actually need to look at a package as it was more than a year ago? And, in that case, how much additional effort would it be in that case to select the "attic" repository instead of the "trunk" repository? (that is, if we don't change the tools to automatically look in attic for stuff not found in trunk).

> The older ones should still be accessible in a separate repository.

That would be the proposed "attic", yes.

> ===
>
> As far as other ideas:  Change SqueakSource's implementation to stop
> using MCFileBasedRepository which I suspect is the root of the
> scalability issue.

Monticello is a purely client-side SCM. You should take a look at the SqueakSource code base. It doesn't use Monticello code for its basic operation at all. It's just a WEBDAV server treating MCZ files as binary blobs.

Only in very rare circumstances does SqueakSource actually look inside a MCZ file (e.g. to produce diffs or show the history in the web view). Not even the date or author you see at the website is taken from inside the MCZ file, it's the uploading user and date.

>  A Magma-based MC backend performs steadily
> regardless of size, I'd love to test this if someone can get
> SqueakSource going on Squeak 4.3.

I would be surprised if it didn't work on 4.3.

>  I also know there are other
> solutions like ss3 and SmalltalkHub although SH may be too different
> to easily migrate..

Assuming that the HTTP listing of gazillions of versions is the bottleneck, neither of those would help significantly. (of course, we need to profile where the slowness actually comes from).

- Bert -

On Wed, May 23, 2012 at 10:28 AM, Bert Freudenberg <[hidden email]> wrote:
> Accessing the repository gets slower and slower due to the ever increasing number of versions there.
>
> To make it faster we could remove old package versions. Say, everything older than a year (but keep at least one version of each package). We could move those to an "attic" repository.
>
> Sounds good? Other ideas?
>
> - Bert -

When you need the configuration file to load specific versions of
packages, I would assume we'd want to keep those versions.
Also, are you considering removing those configuration files as well?

-Chris C (not Cunnington)

>> At a minimum please start more conservatively.  Older than two years
>> but ensuring to keep a minimum of 10 versions of each package
>> regardless of age.  That way we can at least keep SOME history easily
>> accessible for all packages and let's see how much speed that gains
>> us.
>
> But you are aware that all the history meta data is in each and every MC package?

Yes, I am familiar with the entire MC model.

> How often do you actually need to look at a package as it was more than a year ago? And, in that case, how much additional
> effort would it be in that case to select the "attic" repository instead of the "trunk" repository?

We have ~75 packages in the trunk.  Minimum 10 versions per isn't too
much to expect the system to handle, is it?  Sure, the concession of
having to check the archive repository is not-so-bad but I just
concerned cutting to the bone (down to 1 version) would rear its
inconvenient head too often for a not-enough gain in speed.

> (that is, if we don't change the tools to automatically look in attic for stuff not found in trunk).

Yeah, maybe one or two select tool hacks would make it more
transparent.  But we'd have to choose carefully or it would defeat the
speed up.

> Monticello is a purely client-side SCM. You should take a look at the SqueakSource code base. It doesn't use Monticello code for its basic operation at all. It's just a WEBDAV server treating MCZ files as binary blobs.

Ok, didn't know that.

> Assuming that the HTTP listing of gazillions of versions is the bottleneck, neither of those would help significantly. (of course, we need to profile where the slowness actually comes from).

Bad bad -- what use-case thinks it needs to list gazillion versions?
Oh, is that, "Updating trunk..." I see?

On 24.05.2012, at 04:15, Chris Muller wrote:

Under which circumstances did you have to load a package version from trunk that's more than a year old? Honest question.

>> (that is, if we don't change the tools to automatically look in attic for stuff not found in trunk).
>
> Yeah, maybe one or two select tool hacks would make it more
> transparent.  But we'd have to choose carefully or it would defeat the
> speed up.

The normal update process does not access older versions. Neither does the committing of new versions. So I can't see how we would loose speed in regular operation.

>> Monticello is a purely client-side SCM. You should take a look at the SqueakSource code base. It doesn't use Monticello code for its basic operation at all. It's just a WEBDAV server treating MCZ files as binary blobs.
>
> Ok, didn't know that.
>
>> Assuming that the HTTP listing of gazillions of versions is the bottleneck, neither of those would help significantly. (of course, we need to profile where the slowness actually comes from).
>
> Bad bad -- what use-case thinks it needs to list gazillion versions?
> Oh, is that, "Updating trunk..." I see?

Yep.

I see this a lot more often than it used to happen. At some point not too long ago someone put in an update at apparently every operation. If I click a specific version, there is *no need* to do a server listing. Can someone remember when and why that came in?

- Bert -