Performance regression

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance regression

Jakob Reschke-2
Hi all,

On Monday I noticed that some Squot tests started to fail on Travis CI
due to timeouts (test cases exceeding the 5 sec default timeout). The
problems are not due to changes in Squot itself because if I retrigger
an older build which was already green, it also fails. So I suspect a
change in Squeak Trunk, since the Squeak 5.2 builds are also not
affected.

I did not yet find the cause, but would like to make known what I
found so far. Does anyone have an idea what could have caused this? Of
course I don't expect you to debug my test cases, but maybe you have a
hint for me.

- The Mac builds are much more affected than the Linux builds. This
could be due to differences in Travis CI's setup for the operating
systems. The Linux test runs also got slower, but not by so much.
- Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones
scratch at the 5 sec mark, but did not exceed it yet.
- If I repeat older Mac builds, they experience the same slowdown.
- The tests on Squeak 5.2 are unaffected on either platform.
- Some tests take about 6x up to 20x more time than before. On Linux
it is only up to x3 times slower and in sum hardly noticeable. For the
rest of this, I will only talk about the Mac Trunk builds.
- testTrackingAPackage has the worst ratio, whereas testTrackingAClass
is virtually unaffected.
- Only tests that involve whole packages, not individual objects or
single classes, seem to be affected. Is the regression in something
that PackageInfo does (enumerating classes, methods, accessing
categories)?
- Surprisingly, the tests with Git are less worse than the tests with
a stub in-memory repository (these are the same test cases, but with a
different backend object). I think the Git tests by chance ran after
the in-memory tests, maybe that affects things, I don't know yet. The
Git tests did not reach the 5 sec mark.
- Tests that involve Monticello (such as the Monticello->Git
conversion) and do not only create MCDefinitions and pass them around,
take about 2x as long now.

For reference:
Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144
Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404
The runtimes are annotated to the test cases when you expand them.
If you do want to look at the code, just load the Git Browser, then
you will also have the test cases.

If I find out more on the weekend, I will post again.

Kind regards,
Jakob


Reply | Threaded
Open this post in threaded view
|

Re: Performance regression

Jakob Reschke-2
Forgot the time frame: I am looking for something that changed between
2020-06-14 and 2020-06-22.

The builds run unconditionally every weekend to notice Trunk breakages
and were fine on 2020-06-14.

Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke
<[hidden email]>:

>
> Hi all,
>
> On Monday I noticed that some Squot tests started to fail on Travis CI
> due to timeouts (test cases exceeding the 5 sec default timeout). The
> problems are not due to changes in Squot itself because if I retrigger
> an older build which was already green, it also fails. So I suspect a
> change in Squeak Trunk, since the Squeak 5.2 builds are also not
> affected.
>
> I did not yet find the cause, but would like to make known what I
> found so far. Does anyone have an idea what could have caused this? Of
> course I don't expect you to debug my test cases, but maybe you have a
> hint for me.
>
> - The Mac builds are much more affected than the Linux builds. This
> could be due to differences in Travis CI's setup for the operating
> systems. The Linux test runs also got slower, but not by so much.
> - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones
> scratch at the 5 sec mark, but did not exceed it yet.
> - If I repeat older Mac builds, they experience the same slowdown.
> - The tests on Squeak 5.2 are unaffected on either platform.
> - Some tests take about 6x up to 20x more time than before. On Linux
> it is only up to x3 times slower and in sum hardly noticeable. For the
> rest of this, I will only talk about the Mac Trunk builds.
> - testTrackingAPackage has the worst ratio, whereas testTrackingAClass
> is virtually unaffected.
> - Only tests that involve whole packages, not individual objects or
> single classes, seem to be affected. Is the regression in something
> that PackageInfo does (enumerating classes, methods, accessing
> categories)?
> - Surprisingly, the tests with Git are less worse than the tests with
> a stub in-memory repository (these are the same test cases, but with a
> different backend object). I think the Git tests by chance ran after
> the in-memory tests, maybe that affects things, I don't know yet. The
> Git tests did not reach the 5 sec mark.
> - Tests that involve Monticello (such as the Monticello->Git
> conversion) and do not only create MCDefinitions and pass them around,
> take about 2x as long now.
>
> For reference:
> Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144
> Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404
> The runtimes are annotated to the test cases when you expand them.
> If you do want to look at the code, just load the Git Browser, then
> you will also have the test cases.
>
> If I find out more on the weekend, I will post again.
>
> Kind regards,
> Jakob


Reply | Threaded
Open this post in threaded view
|

Re: Performance regression

Nicolas Cellier
A wild guess: String comparison primitive disabled due to jit problem

Le mer. 24 juin 2020 à 08:58, Jakob Reschke <[hidden email]> a écrit :
Forgot the time frame: I am looking for something that changed between
2020-06-14 and 2020-06-22.

The builds run unconditionally every weekend to notice Trunk breakages
and were fine on 2020-06-14.

Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke
<[hidden email]>:
>
> Hi all,
>
> On Monday I noticed that some Squot tests started to fail on Travis CI
> due to timeouts (test cases exceeding the 5 sec default timeout). The
> problems are not due to changes in Squot itself because if I retrigger
> an older build which was already green, it also fails. So I suspect a
> change in Squeak Trunk, since the Squeak 5.2 builds are also not
> affected.
>
> I did not yet find the cause, but would like to make known what I
> found so far. Does anyone have an idea what could have caused this? Of
> course I don't expect you to debug my test cases, but maybe you have a
> hint for me.
>
> - The Mac builds are much more affected than the Linux builds. This
> could be due to differences in Travis CI's setup for the operating
> systems. The Linux test runs also got slower, but not by so much.
> - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones
> scratch at the 5 sec mark, but did not exceed it yet.
> - If I repeat older Mac builds, they experience the same slowdown.
> - The tests on Squeak 5.2 are unaffected on either platform.
> - Some tests take about 6x up to 20x more time than before. On Linux
> it is only up to x3 times slower and in sum hardly noticeable. For the
> rest of this, I will only talk about the Mac Trunk builds.
> - testTrackingAPackage has the worst ratio, whereas testTrackingAClass
> is virtually unaffected.
> - Only tests that involve whole packages, not individual objects or
> single classes, seem to be affected. Is the regression in something
> that PackageInfo does (enumerating classes, methods, accessing
> categories)?
> - Surprisingly, the tests with Git are less worse than the tests with
> a stub in-memory repository (these are the same test cases, but with a
> different backend object). I think the Git tests by chance ran after
> the in-memory tests, maybe that affects things, I don't know yet. The
> Git tests did not reach the 5 sec mark.
> - Tests that involve Monticello (such as the Monticello->Git
> conversion) and do not only create MCDefinitions and pass them around,
> take about 2x as long now.
>
> For reference:
> Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144
> Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404
> The runtimes are annotated to the test cases when you expand them.
> If you do want to look at the code, just load the Git Browser, then
> you will also have the test cases.
>
> If I find out more on the weekend, I will post again.
>
> Kind regards,
> Jakob




Reply | Threaded
Open this post in threaded view
|

Re: Performance regression

Levente Uzonyi
String comparison still uses primitives.
When there's a collation specified, primitive 158 is used
(Collections-ul.900). When there is no collation specified, there's an
additional message send to the "old" code which uses the "old" primitive
when possible.


Levente

On Wed, 24 Jun 2020, Nicolas Cellier wrote:

> A wild guess: String comparison primitive disabled due to jit problem
>
> Le mer. 24 juin 2020 à 08:58, Jakob Reschke <[hidden email]> a écrit :
>       Forgot the time frame: I am looking for something that changed between
>       2020-06-14 and 2020-06-22.
>
>       The builds run unconditionally every weekend to notice Trunk breakages
>       and were fine on 2020-06-14.
>
>       Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke
>       <[hidden email]>:
>       >
>       > Hi all,
>       >
>       > On Monday I noticed that some Squot tests started to fail on Travis CI
>       > due to timeouts (test cases exceeding the 5 sec default timeout). The
>       > problems are not due to changes in Squot itself because if I retrigger
>       > an older build which was already green, it also fails. So I suspect a
>       > change in Squeak Trunk, since the Squeak 5.2 builds are also not
>       > affected.
>       >
>       > I did not yet find the cause, but would like to make known what I
>       > found so far. Does anyone have an idea what could have caused this? Of
>       > course I don't expect you to debug my test cases, but maybe you have a
>       > hint for me.
>       >
>       > - The Mac builds are much more affected than the Linux builds. This
>       > could be due to differences in Travis CI's setup for the operating
>       > systems. The Linux test runs also got slower, but not by so much.
>       > - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones
>       > scratch at the 5 sec mark, but did not exceed it yet.
>       > - If I repeat older Mac builds, they experience the same slowdown.
>       > - The tests on Squeak 5.2 are unaffected on either platform.
>       > - Some tests take about 6x up to 20x more time than before. On Linux
>       > it is only up to x3 times slower and in sum hardly noticeable. For the
>       > rest of this, I will only talk about the Mac Trunk builds.
>       > - testTrackingAPackage has the worst ratio, whereas testTrackingAClass
>       > is virtually unaffected.
>       > - Only tests that involve whole packages, not individual objects or
>       > single classes, seem to be affected. Is the regression in something
>       > that PackageInfo does (enumerating classes, methods, accessing
>       > categories)?
>       > - Surprisingly, the tests with Git are less worse than the tests with
>       > a stub in-memory repository (these are the same test cases, but with a
>       > different backend object). I think the Git tests by chance ran after
>       > the in-memory tests, maybe that affects things, I don't know yet. The
>       > Git tests did not reach the 5 sec mark.
>       > - Tests that involve Monticello (such as the Monticello->Git
>       > conversion) and do not only create MCDefinitions and pass them around,
>       > take about 2x as long now.
>       >
>       > For reference:
>       > Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144
>       > Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404
>       > The runtimes are annotated to the test cases when you expand them.
>       > If you do want to look at the code, just load the Git Browser, then
>       > you will also have the test cases.
>       >
>       > If I find out more on the weekend, I will post again.
>       >
>       > Kind regards,
>       > Jakob
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Performance regression

Jakob Reschke-2
In reply to this post by Jakob Reschke-2
I just debugged a smalltalkCI run under Windows Subsystem For Linux
and it also fails two of the test cases which got slower. This might
not be the real reason for the Mac failures (because this is WSL...),
but it might give a clue.

On 17th of June, Marcel pushed Monticello-mt.724, which changes
MCWorkingCopy>>uniqueVersionName such that name candidates are checked
against the package cache as well.

The test cases I profiled spent >95% of the time writing backup
Monticello versions to the package cache for each package saved with
Squot during this test (it is a safeguard function from three years
ago when Squot or the Git I/O were new and unreliable). The time was
lost in FileDirectory>>primLookupEntryIn:index:. The package cache is
rather full at this point with what has been loaded during the image
update at the start of the test run and, more importantly, with all
the test package versions saved in other tests before. They are not
cleaned up after each test.

If this is the problem on the Mac as well, it turns out I have to fix
my own code after all. Marcel's change seems sound to me and no disk
I/O vs. repeated directory enumeration makes a difference in test
cases after all. I could try to clean up the test packages from the
cache directory. Or I could just drop that Monticello backup after
all... I have never looked at the backup versions in the cache for
three years, and their ancestry is wrong anyway (ignore switching
between branches).

Still, may I suggest to optimize includesVersionNamed: for
MCDirectoryRepository? It currently reads all the directory entries
just to check afterwards whether one of them fits the version name.
Instead it could look up files for the sought version name directly:
there is a method allFileNamesForVersionNamed:.

Am Mi., 24. Juni 2020 um 08:58 Uhr schrieb Jakob Reschke
<[hidden email]>:

>
> Forgot the time frame: I am looking for something that changed between
> 2020-06-14 and 2020-06-22.
>
> The builds run unconditionally every weekend to notice Trunk breakages
> and were fine on 2020-06-14.
>
> Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke
> <[hidden email]>:
> >
> > Hi all,
> >
> > On Monday I noticed that some Squot tests started to fail on Travis CI
> > due to timeouts (test cases exceeding the 5 sec default timeout). The
> > problems are not due to changes in Squot itself because if I retrigger
> > an older build which was already green, it also fails. So I suspect a
> > change in Squeak Trunk, since the Squeak 5.2 builds are also not
> > affected.
> >
> > I did not yet find the cause, but would like to make known what I
> > found so far. Does anyone have an idea what could have caused this? Of
> > course I don't expect you to debug my test cases, but maybe you have a
> > hint for me.
> >
> > - The Mac builds are much more affected than the Linux builds. This
> > could be due to differences in Travis CI's setup for the operating
> > systems. The Linux test runs also got slower, but not by so much.
> > - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones
> > scratch at the 5 sec mark, but did not exceed it yet.
> > - If I repeat older Mac builds, they experience the same slowdown.
> > - The tests on Squeak 5.2 are unaffected on either platform.
> > - Some tests take about 6x up to 20x more time than before. On Linux
> > it is only up to x3 times slower and in sum hardly noticeable. For the
> > rest of this, I will only talk about the Mac Trunk builds.
> > - testTrackingAPackage has the worst ratio, whereas testTrackingAClass
> > is virtually unaffected.
> > - Only tests that involve whole packages, not individual objects or
> > single classes, seem to be affected. Is the regression in something
> > that PackageInfo does (enumerating classes, methods, accessing
> > categories)?
> > - Surprisingly, the tests with Git are less worse than the tests with
> > a stub in-memory repository (these are the same test cases, but with a
> > different backend object). I think the Git tests by chance ran after
> > the in-memory tests, maybe that affects things, I don't know yet. The
> > Git tests did not reach the 5 sec mark.
> > - Tests that involve Monticello (such as the Monticello->Git
> > conversion) and do not only create MCDefinitions and pass them around,
> > take about 2x as long now.
> >
> > For reference:
> > Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144
> > Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404
> > The runtimes are annotated to the test cases when you expand them.
> > If you do want to look at the code, just load the Git Browser, then
> > you will also have the test cases.
> >
> > If I find out more on the weekend, I will post again.
> >
> > Kind regards,
> > Jakob

Reply | Threaded
Open this post in threaded view
|

Re: Performance regression

Levente Uzonyi
Hi Jakob,

On Sun, 28 Jun 2020, Jakob Reschke wrote:

> I just debugged a smalltalkCI run under Windows Subsystem For Linux
> and it also fails two of the test cases which got slower. This might
> not be the real reason for the Mac failures (because this is WSL...),
> but it might give a clue.
>
> On 17th of June, Marcel pushed Monticello-mt.724, which changes
> MCWorkingCopy>>uniqueVersionName such that name candidates are checked
> against the package cache as well.
>
> The test cases I profiled spent >95% of the time writing backup
> Monticello versions to the package cache for each package saved with
> Squot during this test (it is a safeguard function from three years
> ago when Squot or the Git I/O were new and unreliable). The time was
> lost in FileDirectory>>primLookupEntryIn:index:. The package cache is
> rather full at this point with what has been loaded during the image
> update at the start of the test run and, more importantly, with all
> the test package versions saved in other tests before. They are not
> cleaned up after each test.
>
> If this is the problem on the Mac as well, it turns out I have to fix
> my own code after all. Marcel's change seems sound to me and no disk
> I/O vs. repeated directory enumeration makes a difference in test
> cases after all. I could try to clean up the test packages from the
> cache directory. Or I could just drop that Monticello backup after
> all... I have never looked at the backup versions in the cache for
> three years, and their ancestry is wrong anyway (ignore switching
> between branches).
>
> Still, may I suggest to optimize includesVersionNamed: for
> MCDirectoryRepository? It currently reads all the directory entries
> just to check afterwards whether one of them fits the version name.
> Instead it could look up files for the sought version name directly:
> there is a method allFileNamesForVersionNamed:.

That sounds like a good idea, but it's not easy to do that. Why?
You can't know for sure what file name to search for.
E.g.: when you're looking for Collections-mt.896, your package cache may
only contain Collections-mt.896(nice.895).mcd, but not
Collections-mt.896.mcz.
Figuring out the right file name without reading the contents of the
directory is not easy.
If the image transformed all .mcd to .mcz upon download, that wouldn't be
a problem.

One case can still be improved though: when the file names are not cached,
look up the .mcz file and if it's present, return true. See
Monticello-ul.726 in the Inbox with that patch.

If possible, try to add a few #cacheAllFileNamesDuring: sends to the right
places. That should improve performance in all cases. Remember to
#flushAllFilenames when a new version is written while the argument of
#cacheAllFileNamesDuring: is being evaluated.


Levente

>
> Am Mi., 24. Juni 2020 um 08:58 Uhr schrieb Jakob Reschke
> <[hidden email]>:
>>
>> Forgot the time frame: I am looking for something that changed between
>> 2020-06-14 and 2020-06-22.
>>
>> The builds run unconditionally every weekend to notice Trunk breakages
>> and were fine on 2020-06-14.
>>
>> Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke
>> <[hidden email]>:
>> >
>> > Hi all,
>> >
>> > On Monday I noticed that some Squot tests started to fail on Travis CI
>> > due to timeouts (test cases exceeding the 5 sec default timeout). The
>> > problems are not due to changes in Squot itself because if I retrigger
>> > an older build which was already green, it also fails. So I suspect a
>> > change in Squeak Trunk, since the Squeak 5.2 builds are also not
>> > affected.
>> >
>> > I did not yet find the cause, but would like to make known what I
>> > found so far. Does anyone have an idea what could have caused this? Of
>> > course I don't expect you to debug my test cases, but maybe you have a
>> > hint for me.
>> >
>> > - The Mac builds are much more affected than the Linux builds. This
>> > could be due to differences in Travis CI's setup for the operating
>> > systems. The Linux test runs also got slower, but not by so much.
>> > - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones
>> > scratch at the 5 sec mark, but did not exceed it yet.
>> > - If I repeat older Mac builds, they experience the same slowdown.
>> > - The tests on Squeak 5.2 are unaffected on either platform.
>> > - Some tests take about 6x up to 20x more time than before. On Linux
>> > it is only up to x3 times slower and in sum hardly noticeable. For the
>> > rest of this, I will only talk about the Mac Trunk builds.
>> > - testTrackingAPackage has the worst ratio, whereas testTrackingAClass
>> > is virtually unaffected.
>> > - Only tests that involve whole packages, not individual objects or
>> > single classes, seem to be affected. Is the regression in something
>> > that PackageInfo does (enumerating classes, methods, accessing
>> > categories)?
>> > - Surprisingly, the tests with Git are less worse than the tests with
>> > a stub in-memory repository (these are the same test cases, but with a
>> > different backend object). I think the Git tests by chance ran after
>> > the in-memory tests, maybe that affects things, I don't know yet. The
>> > Git tests did not reach the 5 sec mark.
>> > - Tests that involve Monticello (such as the Monticello->Git
>> > conversion) and do not only create MCDefinitions and pass them around,
>> > take about 2x as long now.
>> >
>> > For reference:
>> > Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144
>> > Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404
>> > The runtimes are annotated to the test cases when you expand them.
>> > If you do want to look at the code, just load the Git Browser, then
>> > you will also have the test cases.
>> >
>> > If I find out more on the weekend, I will post again.
>> >
>> > Kind regards,
>> > Jakob

Reply | Threaded
Open this post in threaded view
|

Re: Performance regression

Jakob Reschke-2
Am Mo., 29. Juni 2020 um 01:50 Uhr schrieb Levente Uzonyi
<[hidden email]>:

>
> Hi Jakob,
>
> On Sun, 28 Jun 2020, Jakob Reschke wrote:
> >
> > Still, may I suggest to optimize includesVersionNamed: for
> > MCDirectoryRepository? It currently reads all the directory entries
> > just to check afterwards whether one of them fits the version name.
> > Instead it could look up files for the sought version name directly:
> > there is a method allFileNamesForVersionNamed:.
>
> That sounds like a good idea, but it's not easy to do that. Why?
> You can't know for sure what file name to search for.
> E.g.: when you're looking for Collections-mt.896, your package cache may
> only contain Collections-mt.896(nice.895).mcd, but not
> Collections-mt.896.mcz.
> Figuring out the right file name without reading the contents of the
> directory is not easy.
> If the image transformed all .mcd to .mcz upon download, that wouldn't be
> a problem.
>

I thought the repository would already do the name transformations,
and that MCFileBasedRepository>>#allFileNamesForVersionNamed: would
use that, but it gets all the file names, too, and then checks which
of them matches the MCVersionName. Moreover,
MCFileBasedRepository>>#basicStoreVersion: does not dictate the
filename scheme, but asks the MCVersion for the fileName. That sounds
odd to me, why should a version know in which format it is stored?
Should that not be the business of the repository and the
readers/writers it uses? (It may still dispatch through the version to
distinguish diffy versions.)

Anyway...

> One case can still be improved though: when the file names are not cached,
> look up the .mcz file and if it's present, return true. See
> Monticello-ul.726 in the Inbox with that patch.

Sounds reasonable. I guess I should still fix my tests, but it should
cut the slowdown for the cases where a number candidate is already
taken (trying new version names 1; 1, 2; 1, 2, 3; 1, ..., n - 1; 1,
..., n).

>
> If possible, try to add a few #cacheAllFileNamesDuring: sends to the right
> places. That should improve performance in all cases. Remember to
> #flushAllFilenames when a new version is written while the argument of
> #cacheAllFileNamesDuring: is being evaluated.

I hope I'll be able to try that during the week.