Performance regression
Locked
 
|
Hi all,
On Monday I noticed that some Squot tests started to fail on Travis CI due to timeouts (test cases exceeding the 5 sec default timeout). The problems are not due to changes in Squot itself because if I retrigger an older build which was already green, it also fails. So I suspect a change in Squeak Trunk, since the Squeak 5.2 builds are also not affected. I did not yet find the cause, but would like to make known what I found so far. Does anyone have an idea what could have caused this? Of course I don't expect you to debug my test cases, but maybe you have a hint for me. - The Mac builds are much more affected than the Linux builds. This could be due to differences in Travis CI's setup for the operating systems. The Linux test runs also got slower, but not by so much. - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones scratch at the 5 sec mark, but did not exceed it yet. - If I repeat older Mac builds, they experience the same slowdown. - The tests on Squeak 5.2 are unaffected on either platform. - Some tests take about 6x up to 20x more time than before. On Linux it is only up to x3 times slower and in sum hardly noticeable. For the rest of this, I will only talk about the Mac Trunk builds. - testTrackingAPackage has the worst ratio, whereas testTrackingAClass is virtually unaffected. - Only tests that involve whole packages, not individual objects or single classes, seem to be affected. Is the regression in something that PackageInfo does (enumerating classes, methods, accessing categories)? - Surprisingly, the tests with Git are less worse than the tests with a stub in-memory repository (these are the same test cases, but with a different backend object). I think the Git tests by chance ran after the in-memory tests, maybe that affects things, I don't know yet. The Git tests did not reach the 5 sec mark. - Tests that involve Monticello (such as the Monticello->Git conversion) and do not only create MCDefinitions and pass them around, take about 2x as long now. For reference: Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144 Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404 The runtimes are annotated to the test cases when you expand them. If you do want to look at the code, just load the Git Browser, then you will also have the test cases. If I find out more on the weekend, I will post again. Kind regards, Jakob |
|
|
Forgot the time frame: I am looking for something that changed between
2020-06-14 and 2020-06-22. The builds run unconditionally every weekend to notice Trunk breakages and were fine on 2020-06-14. Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke <[hidden email]>: > > Hi all, > > On Monday I noticed that some Squot tests started to fail on Travis CI > due to timeouts (test cases exceeding the 5 sec default timeout). The > problems are not due to changes in Squot itself because if I retrigger > an older build which was already green, it also fails. So I suspect a > change in Squeak Trunk, since the Squeak 5.2 builds are also not > affected. > > I did not yet find the cause, but would like to make known what I > found so far. Does anyone have an idea what could have caused this? Of > course I don't expect you to debug my test cases, but maybe you have a > hint for me. > > - The Mac builds are much more affected than the Linux builds. This > could be due to differences in Travis CI's setup for the operating > systems. The Linux test runs also got slower, but not by so much. > - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones > scratch at the 5 sec mark, but did not exceed it yet. > - If I repeat older Mac builds, they experience the same slowdown. > - The tests on Squeak 5.2 are unaffected on either platform. > - Some tests take about 6x up to 20x more time than before. On Linux > it is only up to x3 times slower and in sum hardly noticeable. For the > rest of this, I will only talk about the Mac Trunk builds. > - testTrackingAPackage has the worst ratio, whereas testTrackingAClass > is virtually unaffected. > - Only tests that involve whole packages, not individual objects or > single classes, seem to be affected. Is the regression in something > that PackageInfo does (enumerating classes, methods, accessing > categories)? > - Surprisingly, the tests with Git are less worse than the tests with > a stub in-memory repository (these are the same test cases, but with a > different backend object). I think the Git tests by chance ran after > the in-memory tests, maybe that affects things, I don't know yet. The > Git tests did not reach the 5 sec mark. > - Tests that involve Monticello (such as the Monticello->Git > conversion) and do not only create MCDefinitions and pass them around, > take about 2x as long now. > > For reference: > Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144 > Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404 > The runtimes are annotated to the test cases when you expand them. > If you do want to look at the code, just load the Git Browser, then > you will also have the test cases. > > If I find out more on the weekend, I will post again. > > Kind regards, > Jakob |
|
|
A wild guess: String comparison primitive disabled due to jit problem Le mer. 24 juin 2020 à 08:58, Jakob Reschke <[hidden email]> a écrit : Forgot the time frame: I am looking for something that changed between |
|
|
String comparison still uses primitives.
When there's a collation specified, primitive 158 is used (Collections-ul.900). When there is no collation specified, there's an additional message send to the "old" code which uses the "old" primitive when possible. Levente On Wed, 24 Jun 2020, Nicolas Cellier wrote: > A wild guess: String comparison primitive disabled due to jit problem > > Le mer. 24 juin 2020 à 08:58, Jakob Reschke <[hidden email]> a écrit : > Forgot the time frame: I am looking for something that changed between > 2020-06-14 and 2020-06-22. > > The builds run unconditionally every weekend to notice Trunk breakages > and were fine on 2020-06-14. > > Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke > <[hidden email]>: > > > > Hi all, > > > > On Monday I noticed that some Squot tests started to fail on Travis CI > > due to timeouts (test cases exceeding the 5 sec default timeout). The > > problems are not due to changes in Squot itself because if I retrigger > > an older build which was already green, it also fails. So I suspect a > > change in Squeak Trunk, since the Squeak 5.2 builds are also not > > affected. > > > > I did not yet find the cause, but would like to make known what I > > found so far. Does anyone have an idea what could have caused this? Of > > course I don't expect you to debug my test cases, but maybe you have a > > hint for me. > > > > - The Mac builds are much more affected than the Linux builds. This > > could be due to differences in Travis CI's setup for the operating > > systems. The Linux test runs also got slower, but not by so much. > > - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones > > scratch at the 5 sec mark, but did not exceed it yet. > > - If I repeat older Mac builds, they experience the same slowdown. > > - The tests on Squeak 5.2 are unaffected on either platform. > > - Some tests take about 6x up to 20x more time than before. On Linux > > it is only up to x3 times slower and in sum hardly noticeable. For the > > rest of this, I will only talk about the Mac Trunk builds. > > - testTrackingAPackage has the worst ratio, whereas testTrackingAClass > > is virtually unaffected. > > - Only tests that involve whole packages, not individual objects or > > single classes, seem to be affected. Is the regression in something > > that PackageInfo does (enumerating classes, methods, accessing > > categories)? > > - Surprisingly, the tests with Git are less worse than the tests with > > a stub in-memory repository (these are the same test cases, but with a > > different backend object). I think the Git tests by chance ran after > > the in-memory tests, maybe that affects things, I don't know yet. The > > Git tests did not reach the 5 sec mark. > > - Tests that involve Monticello (such as the Monticello->Git > > conversion) and do not only create MCDefinitions and pass them around, > > take about 2x as long now. > > > > For reference: > > Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144 > > Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404 > > The runtimes are annotated to the test cases when you expand them. > > If you do want to look at the code, just load the Git Browser, then > > you will also have the test cases. > > > > If I find out more on the weekend, I will post again. > > > > Kind regards, > > Jakob > > > > |
|
|
In reply to this post by Jakob Reschke
I just debugged a smalltalkCI run under Windows Subsystem For Linux
and it also fails two of the test cases which got slower. This might not be the real reason for the Mac failures (because this is WSL...), but it might give a clue. On 17th of June, Marcel pushed Monticello-mt.724, which changes MCWorkingCopy>>uniqueVersionName such that name candidates are checked against the package cache as well. The test cases I profiled spent >95% of the time writing backup Monticello versions to the package cache for each package saved with Squot during this test (it is a safeguard function from three years ago when Squot or the Git I/O were new and unreliable). The time was lost in FileDirectory>>primLookupEntryIn:index:. The package cache is rather full at this point with what has been loaded during the image update at the start of the test run and, more importantly, with all the test package versions saved in other tests before. They are not cleaned up after each test. If this is the problem on the Mac as well, it turns out I have to fix my own code after all. Marcel's change seems sound to me and no disk I/O vs. repeated directory enumeration makes a difference in test cases after all. I could try to clean up the test packages from the cache directory. Or I could just drop that Monticello backup after all... I have never looked at the backup versions in the cache for three years, and their ancestry is wrong anyway (ignore switching between branches). Still, may I suggest to optimize includesVersionNamed: for MCDirectoryRepository? It currently reads all the directory entries just to check afterwards whether one of them fits the version name. Instead it could look up files for the sought version name directly: there is a method allFileNamesForVersionNamed:. Am Mi., 24. Juni 2020 um 08:58 Uhr schrieb Jakob Reschke <[hidden email]>: > > Forgot the time frame: I am looking for something that changed between > 2020-06-14 and 2020-06-22. > > The builds run unconditionally every weekend to notice Trunk breakages > and were fine on 2020-06-14. > > Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke > <[hidden email]>: > > > > Hi all, > > > > On Monday I noticed that some Squot tests started to fail on Travis CI > > due to timeouts (test cases exceeding the 5 sec default timeout). The > > problems are not due to changes in Squot itself because if I retrigger > > an older build which was already green, it also fails. So I suspect a > > change in Squeak Trunk, since the Squeak 5.2 builds are also not > > affected. > > > > I did not yet find the cause, but would like to make known what I > > found so far. Does anyone have an idea what could have caused this? Of > > course I don't expect you to debug my test cases, but maybe you have a > > hint for me. > > > > - The Mac builds are much more affected than the Linux builds. This > > could be due to differences in Travis CI's setup for the operating > > systems. The Linux test runs also got slower, but not by so much. > > - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones > > scratch at the 5 sec mark, but did not exceed it yet. > > - If I repeat older Mac builds, they experience the same slowdown. > > - The tests on Squeak 5.2 are unaffected on either platform. > > - Some tests take about 6x up to 20x more time than before. On Linux > > it is only up to x3 times slower and in sum hardly noticeable. For the > > rest of this, I will only talk about the Mac Trunk builds. > > - testTrackingAPackage has the worst ratio, whereas testTrackingAClass > > is virtually unaffected. > > - Only tests that involve whole packages, not individual objects or > > single classes, seem to be affected. Is the regression in something > > that PackageInfo does (enumerating classes, methods, accessing > > categories)? > > - Surprisingly, the tests with Git are less worse than the tests with > > a stub in-memory repository (these are the same test cases, but with a > > different backend object). I think the Git tests by chance ran after > > the in-memory tests, maybe that affects things, I don't know yet. The > > Git tests did not reach the 5 sec mark. > > - Tests that involve Monticello (such as the Monticello->Git > > conversion) and do not only create MCDefinitions and pass them around, > > take about 2x as long now. > > > > For reference: > > Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144 > > Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404 > > The runtimes are annotated to the test cases when you expand them. > > If you do want to look at the code, just load the Git Browser, then > > you will also have the test cases. > > > > If I find out more on the weekend, I will post again. > > > > Kind regards, > > Jakob |
|
|
Hi Jakob,
On Sun, 28 Jun 2020, Jakob Reschke wrote: > I just debugged a smalltalkCI run under Windows Subsystem For Linux > and it also fails two of the test cases which got slower. This might > not be the real reason for the Mac failures (because this is WSL...), > but it might give a clue. > > On 17th of June, Marcel pushed Monticello-mt.724, which changes > MCWorkingCopy>>uniqueVersionName such that name candidates are checked > against the package cache as well. > > The test cases I profiled spent >95% of the time writing backup > Monticello versions to the package cache for each package saved with > Squot during this test (it is a safeguard function from three years > ago when Squot or the Git I/O were new and unreliable). The time was > lost in FileDirectory>>primLookupEntryIn:index:. The package cache is > rather full at this point with what has been loaded during the image > update at the start of the test run and, more importantly, with all > the test package versions saved in other tests before. They are not > cleaned up after each test. > > If this is the problem on the Mac as well, it turns out I have to fix > my own code after all. Marcel's change seems sound to me and no disk > I/O vs. repeated directory enumeration makes a difference in test > cases after all. I could try to clean up the test packages from the > cache directory. Or I could just drop that Monticello backup after > all... I have never looked at the backup versions in the cache for > three years, and their ancestry is wrong anyway (ignore switching > between branches). > > Still, may I suggest to optimize includesVersionNamed: for > MCDirectoryRepository? It currently reads all the directory entries > just to check afterwards whether one of them fits the version name. > Instead it could look up files for the sought version name directly: > there is a method allFileNamesForVersionNamed:. That sounds like a good idea, but it's not easy to do that. Why? You can't know for sure what file name to search for. E.g.: when you're looking for Collections-mt.896, your package cache may only contain Collections-mt.896(nice.895).mcd, but not Collections-mt.896.mcz. Figuring out the right file name without reading the contents of the directory is not easy. If the image transformed all .mcd to .mcz upon download, that wouldn't be a problem. One case can still be improved though: when the file names are not cached, look up the .mcz file and if it's present, return true. See Monticello-ul.726 in the Inbox with that patch. If possible, try to add a few #cacheAllFileNamesDuring: sends to the right places. That should improve performance in all cases. Remember to #flushAllFilenames when a new version is written while the argument of #cacheAllFileNamesDuring: is being evaluated. Levente > > Am Mi., 24. Juni 2020 um 08:58 Uhr schrieb Jakob Reschke > <[hidden email]>: >> >> Forgot the time frame: I am looking for something that changed between >> 2020-06-14 and 2020-06-22. >> >> The builds run unconditionally every weekend to notice Trunk breakages >> and were fine on 2020-06-14. >> >> Am Mi., 24. Juni 2020 um 08:43 Uhr schrieb Jakob Reschke >> <[hidden email]>: >> > >> > Hi all, >> > >> > On Monday I noticed that some Squot tests started to fail on Travis CI >> > due to timeouts (test cases exceeding the 5 sec default timeout). The >> > problems are not due to changes in Squot itself because if I retrigger >> > an older build which was already green, it also fails. So I suspect a >> > change in Squeak Trunk, since the Squeak 5.2 builds are also not >> > affected. >> > >> > I did not yet find the cause, but would like to make known what I >> > found so far. Does anyone have an idea what could have caused this? Of >> > course I don't expect you to debug my test cases, but maybe you have a >> > hint for me. >> > >> > - The Mac builds are much more affected than the Linux builds. This >> > could be due to differences in Travis CI's setup for the operating >> > systems. The Linux test runs also got slower, but not by so much. >> > - Both the 64 bit and 32 bit Mac builds got slower. The 64 bit ones >> > scratch at the 5 sec mark, but did not exceed it yet. >> > - If I repeat older Mac builds, they experience the same slowdown. >> > - The tests on Squeak 5.2 are unaffected on either platform. >> > - Some tests take about 6x up to 20x more time than before. On Linux >> > it is only up to x3 times slower and in sum hardly noticeable. For the >> > rest of this, I will only talk about the Mac Trunk builds. >> > - testTrackingAPackage has the worst ratio, whereas testTrackingAClass >> > is virtually unaffected. >> > - Only tests that involve whole packages, not individual objects or >> > single classes, seem to be affected. Is the regression in something >> > that PackageInfo does (enumerating classes, methods, accessing >> > categories)? >> > - Surprisingly, the tests with Git are less worse than the tests with >> > a stub in-memory repository (these are the same test cases, but with a >> > different backend object). I think the Git tests by chance ran after >> > the in-memory tests, maybe that affects things, I don't know yet. The >> > Git tests did not reach the 5 sec mark. >> > - Tests that involve Monticello (such as the Monticello->Git >> > conversion) and do not only create MCDefinitions and pass them around, >> > take about 2x as long now. >> > >> > For reference: >> > Good build: https://travis-ci.org/github/hpi-swa/Squot/jobs/698280144 >> > Bad build: https://travis-ci.org/github/hpi-swa/Squot/jobs/700640404 >> > The runtimes are annotated to the test cases when you expand them. >> > If you do want to look at the code, just load the Git Browser, then >> > you will also have the test cases. >> > >> > If I find out more on the weekend, I will post again. >> > >> > Kind regards, >> > Jakob |
|
|
Am Mo., 29. Juni 2020 um 01:50 Uhr schrieb Levente Uzonyi
<[hidden email]>: > > Hi Jakob, > > On Sun, 28 Jun 2020, Jakob Reschke wrote: > > > > Still, may I suggest to optimize includesVersionNamed: for > > MCDirectoryRepository? It currently reads all the directory entries > > just to check afterwards whether one of them fits the version name. > > Instead it could look up files for the sought version name directly: > > there is a method allFileNamesForVersionNamed:. > > That sounds like a good idea, but it's not easy to do that. Why? > You can't know for sure what file name to search for. > E.g.: when you're looking for Collections-mt.896, your package cache may > only contain Collections-mt.896(nice.895).mcd, but not > Collections-mt.896.mcz. > Figuring out the right file name without reading the contents of the > directory is not easy. > If the image transformed all .mcd to .mcz upon download, that wouldn't be > a problem. > I thought the repository would already do the name transformations, and that MCFileBasedRepository>>#allFileNamesForVersionNamed: would use that, but it gets all the file names, too, and then checks which of them matches the MCVersionName. Moreover, MCFileBasedRepository>>#basicStoreVersion: does not dictate the filename scheme, but asks the MCVersion for the fileName. That sounds odd to me, why should a version know in which format it is stored? Should that not be the business of the repository and the readers/writers it uses? (It may still dispatch through the version to distinguish diffy versions.) Anyway... > One case can still be improved though: when the file names are not cached, > look up the .mcz file and if it's present, return true. See > Monticello-ul.726 in the Inbox with that patch. Sounds reasonable. I guess I should still fix my tests, but it should cut the slowdown for the cases where a number candidate is already taken (trying new version names 1; 1, 2; 1, 2, 3; 1, ..., n - 1; 1, ..., n). > > If possible, try to add a few #cacheAllFileNamesDuring: sends to the right > places. That should improve performance in all cases. Remember to > #flushAllFilenames when a new version is written while the argument of > #cacheAllFileNamesDuring: is being evaluated. I hope I'll be able to try that during the week. |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |