More speedups for Store coming up

Hello...

  I just finished another round of speedups for Store.  This time, the
  changes are not as simple as the previous time.  In short, the
  benefits are:

  Up to 20% faster reconciling to the database.
  Between 1.5-2x faster Base64ReadStream.
  Up to 20% less scavenges.

  Coming up...

--
Best regards,
 Andres                          mailto:[hidden email]

Last November Andres published few store enhancements which did speed up
things along.

I wonder did any of those fixes found their way into the released VW7.5 ?

--Mark

Andres Valloud wrote:

Mark Pirogovsky wrote:
> Last November Andres published few store enhancements which did speed
> up things along.
>
> I wonder did any of those fixes found their way into the released VW7.5 ?
I know some of them made it to 7.5, and I saw some others get integrated
into the next release we are currently working on.  I will check to make
sure all of them are taken into account as appropriate.

Thanks,
Andres.

PS: I left the text below for reference.

Andres Valloud wrote:
> Mark Pirogovsky wrote:
>> Last November Andres published few store enhancements which did speed
>> up things along.
>>
>> I wonder did any of those fixes found their way into the released VW7.5 ?
> I know some of them made it to 7.5, and I saw some others get integrated
> into the next release we are currently working on.  I will check to make
> sure all of them are taken into account as appropriate.

Unfortunately, working with the Public Repository with 7.5 is currently
really slow, even slower than 7.4.1. For example, I stopped loading a
new version of Widgetry after waiting for an hour.
While loading a new bundle version, for each modification,
#checkForOverridesInBundleLoad: is called. It contains the following code:

    leafNames := self leafItems collect: [ :pkg | pkg name ]

which results in a DB query for each contained package/bundle, so this
alone involves [number of modifications] * [number of pundles].

Talking of  #checkForOverridesInBundleLoad:, if "Create Overrides in
Bundles when Loading" is turned on, the above check is not performed.
This speeds up loading significantly, but after upgrading the Widgetry
bundle, I had dozens of overrides. There's a note at the end of the help
text of this item talking about this, but I think there should be a
major warning when this option is turned on, as it's obviously dangerous.

> which results in a DB query for each contained package/bundle, so this
> alone involves [number of modifications] * [number of pundles].

bugfixed version:

...so this alone involves [number of modifications] * [number of
pundles] DB queries

Holger Kleinsorgen wrote:

> Unfortunately, working with the Public Repository with 7.5 is currently
> really slow, even slower than 7.4.1. For example, I stopped loading a
> new version of Widgetry after waiting for an hour.

I have a same problem loading into VW 7.5 from my own Store server.
Speed comparing to 7.4.1 is a factor 10 an more slower!

When looking for a cause I found that my PostgreSql server is eating
100% CPU while obviously running some long SQL query for Store. And my
server is very very old and slow but it was good until 7.5. It seems
that on 7.5 some SQL queries are optimized, but on newest hardware, not
considering that many Store servers (including public repository) are
running on much slower hardware.

For me I'm looking for obvious solution - moving Store to a faster
server, maybe Cincom should also?

Best regards
Janko


--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si

Janko Mivšek wrote:
> Holger Kleinsorgen wrote:
>
>> Unfortunately, working with the Public Repository with 7.5 is
>> currently really slow, even slower than 7.4.1. For example, I stopped
>> loading a new version of Widgetry after waiting for an hour.
>
> I have a same problem loading into VW 7.5 from my own Store server.
> Speed comparing to 7.4.1 is a factor 10 an more slower!
>
This is most likely due to the lack of proper indexing in the DB model.
Careful selection of index columns significantly speeds up things. I did
not yet have the time to investigate more on this, however.

Andre

We've had performance issues a long time ago (I can't remember in what cases exactly). Putting some more indexes on the database certainly improved performance drastically.

From what I can see we've at least added these indexes:

On TW_METHODS an index on PACKAGEREF and an index on METHODREF.


-----Original Message-----
From: [hidden email] [mailto:[hidden email]]
Sent: woensdag 4 juli 2007 12:29
To: Janko Mivšek
Cc: 'VWNC'
Subject: Re: More speedups for Store ?



Janko Mivšek wrote:
> Holger Kleinsorgen wrote:
>
>> Unfortunately, working with the Public Repository with 7.5 is
>> currently really slow, even slower than 7.4.1. For example, I stopped
>> loading a new version of Widgetry after waiting for an hour.
>
> I have a same problem loading into VW 7.5 from my own Store server.
> Speed comparing to 7.4.1 is a factor 10 an more slower!
>
This is most likely due to the lack of proper indexing in the DB model.
Careful selection of index columns significantly speeds up things. I did
not yet have the time to investigate more on this, however.

Andre

On 04/07/07, Janko Mivšek <[hidden email]> wrote:
> When looking for a cause I found that my PostgreSql server is eating
> 100% CPU while obviously running some long SQL query for Store. And my
> server is very very old and slow but it was good until 7.5. It seems
> that on 7.5 some SQL queries are optimized, but on newest hardware, not
> considering that many Store servers (including public repository) are
> running on much slower hardware.

Have you run vacuum analyze on the database?  This can help quite a
bit depending on the version of PostgreSQL you are using.

--
Make the most of your skills - with OpenSkills
http://www.openskills.org/

Hello,

I checked the speedups below.  The one in which selectorName was added
to Method is no longer applicable since the code has changed in Store to
the point that the optimization no longer makes sense.  However, the
base 64 decoding stream does apply.  I will follow up on this.

Thanks,
Andres.

-----Original Message-----
From: Andres Valloud [mailto:[hidden email]]
Sent: Tuesday, July 03, 2007 10:58 PM
To: Mark Pirogovsky
Cc: [hidden email]; Valloud, Andres
Subject: Re: More speedups for Store ?

Mark Pirogovsky wrote:
> Last November Andres published few store enhancements which did speed
> up things along.
>
> I wonder did any of those fixes found their way into the released
VW7.5 ?
I know some of them made it to 7.5, and I saw some others get integrated
into the next release we are currently working on.  I will check to make
sure all of them are taken into account as appropriate.

Thanks,
Andres.

PS: I left the text below for reference.
> Andres Valloud wrote:
>> Hello...
>>
>>   I just finished another round of speedups for Store.  This time,
the