Too many update configs in trunk?
Locked
12
12
Too many update configs in trunk?
 
|
Hi folks,
I just tried to update an older trunk image to the current one. It took ages. I have the impression that people are committing way more update configs than necessary. These are not cheap, but they force an incremental update process. Instead of just loading the latest version, they force me to load many many intermediate versions. I cannot imagine it is necessary to have so many. We are close to 200 now. Normally we only need a new config map when the load order of packages needs to change. But I see many configs where the order does not change at all. Attached below. Sometimes it might be necessary to force a specific version to be loaded before a subsequent version. But is that really necessary that often? If it is indeed necessary, then we need to figure out a way to only force-loading a specific package, not all packages. That would speed up the updates quite a bit. - Bert - ---- update-ul.118.mcm ---- update-ul.119.mcm ---- update-nice.120.mcm ---- update-nice.121.mcm ---- update-ul.122.mcm ---- update-ul.123.mcm ---- update-ul.124.mcm ---- update-nice.125.mcm ---- update-nice.126.mcm ---- update-nice.127.mcm ---- update-ar.128.mcm + Monticello Compiler EToys Exceptions Files FlexibleVocabularies Graphics GraphicsTests Installer-Core Kernel KernelTests MonticelloConfigurations - Monticello ---- update-ar.129.mcm ---- update-ar.130.mcm ---- update-nice.131.mcm ---- update-nice.132.mcm ---- update-ul.133.mcm ---- update-laza.134.mcm ---- update-nice.135.mcm ---- update-nice.136.mcm ---- update-ul.137.mcm ---- update-ul.138.mcm + Kernel Compiler EToys Exceptions Files FlexibleVocabularies Graphics GraphicsTests Installer-Core - Kernel ---- update-ar.139.mcm + System Morphic Multilingual MorphicExtras MorphicTests MultilingualTests Nebraska Network NetworkTests PackageInfo-Base PreferenceBrowser Protocols SMBase SMLoader ST80 SUnit SUnitGUI ScriptLoader Services-Base Sound - System ---- update-nice.140.mcm ---- update-ul.141.mcm ---- update-ul.142.mcm ---- update-nice.143.mcm ---- update-ul.144.mcm ---- update-ar.145.mcm ---- update-ul.146.mcm ---- update-bf.147.mcm + HelpSystem-Core + HelpSystem-Tests + Help-Squeak-Project ---- update-ul.148.mcm ---- update-ar.149.mcm ---- update-nice.150.mcm ---- update-ar.151.mcm + Network Collections CollectionsTests Compression Monticello Kernel Compiler EToys Exceptions Files FlexibleVocabularies Graphics GraphicsTests Installer-Core KernelTests MonticelloConfigurations System Morphic Multilingual MorphicExtras MorphicTests MultilingualTests Nebraska - Network ---- update-ar.152.mcm + Compression Collections CollectionsTests - Compression ---- update-ul.153.mcm ---- update-ul.154.mcm ---- update-ul.155.mcm ---- update-ul.156.mcm ---- update-ul.157.mcm ---- update-ul.158.mcm ---- update-ul.159.mcm ---- update-ul.160.mcm ---- update-ul.161.mcm ---- update-ul.162.mcm ---- update-ul.163.mcm ---- update-dtl.164.mcm + Help-Squeak-TerseGuide ---- update-ul.165.mcm ---- update-ul.166.mcm ---- update-ul.167.mcm ---- update-ul.168.mcm ---- update-nice.169.mcm ---- update-dtl.170.mcm + SystemReporter ---- update-ul.171.mcm ---- update-ul.172.mcm ---- update-ul.173.mcm ---- update-ul.174.mcm ---- update-ul.175.mcm ---- update-dtl.176.mcm ---- update-ul.177.mcm + BalloonTests ---- update-ul.178.mcm ---- update-cmm.179.mcm + TrunkScript ---- update-cmm.180.mcm - TrunkScript ---- update-ul.182.mcm ---- update-cmm.183.mcm ---- update-nice.184.mcm ---- update-ul.185.mcm ---- update-ul.186.mcm ---- update-ul.187.mcm ---- update-kb.188.mcm ---- update-nice.189.mcm ---- update-ul.190.mcm ---- update-ul.191.mcm repo := MCRepositoryGroup default repositories detect: [:r | r description = 'http://source.squeak.org/trunk']. updates := repo allFileNames select: [ :f | (f beginsWith: 'update-') and: [f endsWith: '.mcm'] ]. updates := updates asSortedCollection: [ :a :b | (a allButLast: 4) numericSuffix < (b allButLast: 4) numericSuffix]. Transcript clear. updates do: [ :update | Transcript show: '----'; cr; show: update; cr. config := repo versionNamed: update. pkgs := String streamContents: [:s | config dependencies do: [:dep | s nextPutAll: dep package name; cr] ]. prev ifNotNil: [ context := nil. (TextDiffBuilder from: prev to: pkgs) patchSequenceDoIfMatch: [ :string | context ifNotNil: [context := context, string ] ] ifInsert: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '+ ', string. context := ''] ifRemove: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '- ', string. context := '']. ]. prev := pkgs]. |
Re: Too many update configs in trunk?
|
|
On Thu, 9 Jun 2011, Bert Freudenberg wrote:
> Hi folks, > > I just tried to update an older trunk image to the current one. It took ages. > > I have the impression that people are committing way more update configs than necessary. These are not cheap, but they force an incremental update process. Instead of just loading the latest version, they force me to load many many intermediate versions. I cannot imagine it is necessary to have so many. We are close to 200 now. The intermediate versions for some packages have to be loaded otherwise your image may break. > > Normally we only need a new config map when the load order of packages needs to change. But I see many configs where the order does not change at all. Attached below. > > Sometimes it might be necessary to force a specific version to be loaded before a subsequent version. But is that really necessary that often? In most cases it's necessary. Before update-ul.16x.mcm I didn't rely on the package order inside mcms, so some mcms could have been avoided, but I guess their number is less than 10. > > If it is indeed necessary, then we need to figure out a way to only force-loading a specific package, not all packages. That would speed up the updates quite a bit. If we can mark the packages which require a specific version to be loaded, then we'll have the faster update process. One possible way is to only include those packages in the mcms which require a specific version. At the end of the update process update to the latest version all in-image packages which are also present in the repository. But changing the update process by the update process is a PITA as Chris found it out recently. Levente <rant> Try updating a Pharo image and you'll see how snappy our update process is. </rant> > > > - Bert - > > > ---- > update-ul.118.mcm > ---- > update-ul.119.mcm > ---- > update-nice.120.mcm > ---- > update-nice.121.mcm > ---- > update-ul.122.mcm > ---- > update-ul.123.mcm > ---- > update-ul.124.mcm > ---- > update-nice.125.mcm > ---- > update-nice.126.mcm > ---- > update-nice.127.mcm > ---- > update-ar.128.mcm > + Monticello > Compiler > EToys > Exceptions > Files > FlexibleVocabularies > Graphics > GraphicsTests > Installer-Core > Kernel > KernelTests > MonticelloConfigurations > - Monticello > ---- > update-ar.129.mcm > ---- > update-ar.130.mcm > ---- > update-nice.131.mcm > ---- > update-nice.132.mcm > ---- > update-ul.133.mcm > ---- > update-laza.134.mcm > ---- > update-nice.135.mcm > ---- > update-nice.136.mcm > ---- > update-ul.137.mcm > ---- > update-ul.138.mcm > + Kernel > Compiler > EToys > Exceptions > Files > FlexibleVocabularies > Graphics > GraphicsTests > Installer-Core > - Kernel > ---- > update-ar.139.mcm > + System > Morphic > Multilingual > MorphicExtras > MorphicTests > MultilingualTests > Nebraska > Network > NetworkTests > PackageInfo-Base > PreferenceBrowser > Protocols > SMBase > SMLoader > ST80 > SUnit > SUnitGUI > ScriptLoader > Services-Base > Sound > - System > ---- > update-nice.140.mcm > ---- > update-ul.141.mcm > ---- > update-ul.142.mcm > ---- > update-nice.143.mcm > ---- > update-ul.144.mcm > ---- > update-ar.145.mcm > ---- > update-ul.146.mcm > ---- > update-bf.147.mcm > + HelpSystem-Core > + HelpSystem-Tests > + Help-Squeak-Project > ---- > update-ul.148.mcm > ---- > update-ar.149.mcm > ---- > update-nice.150.mcm > ---- > update-ar.151.mcm > + Network > Collections > CollectionsTests > Compression > Monticello > Kernel > Compiler > EToys > Exceptions > Files > FlexibleVocabularies > Graphics > GraphicsTests > Installer-Core > KernelTests > MonticelloConfigurations > System > Morphic > Multilingual > MorphicExtras > MorphicTests > MultilingualTests > Nebraska > - Network > ---- > update-ar.152.mcm > + Compression > Collections > CollectionsTests > - Compression > ---- > update-ul.153.mcm > ---- > update-ul.154.mcm > ---- > update-ul.155.mcm > ---- > update-ul.156.mcm > ---- > update-ul.157.mcm > ---- > update-ul.158.mcm > ---- > update-ul.159.mcm > ---- > update-ul.160.mcm > ---- > update-ul.161.mcm > ---- > update-ul.162.mcm > ---- > update-ul.163.mcm > ---- > update-dtl.164.mcm > + Help-Squeak-TerseGuide > ---- > update-ul.165.mcm > ---- > update-ul.166.mcm > ---- > update-ul.167.mcm > ---- > update-ul.168.mcm > ---- > update-nice.169.mcm > ---- > update-dtl.170.mcm > + SystemReporter > ---- > update-ul.171.mcm > ---- > update-ul.172.mcm > ---- > update-ul.173.mcm > ---- > update-ul.174.mcm > ---- > update-ul.175.mcm > ---- > update-dtl.176.mcm > ---- > update-ul.177.mcm > + BalloonTests > ---- > update-ul.178.mcm > ---- > update-cmm.179.mcm > + TrunkScript > ---- > update-cmm.180.mcm > - TrunkScript > ---- > update-ul.182.mcm > ---- > update-cmm.183.mcm > ---- > update-nice.184.mcm > ---- > update-ul.185.mcm > ---- > update-ul.186.mcm > ---- > update-ul.187.mcm > ---- > update-kb.188.mcm > ---- > update-nice.189.mcm > ---- > update-ul.190.mcm > ---- > update-ul.191.mcm > > > repo := MCRepositoryGroup default repositories detect: [:r | r description = 'http://source.squeak.org/trunk']. > updates := repo allFileNames select: [ :f | (f beginsWith: 'update-') and: [f endsWith: '.mcm'] ]. > updates := updates asSortedCollection: [ :a :b | (a allButLast: 4) numericSuffix < (b allButLast: 4) numericSuffix]. > Transcript clear. > updates do: [ :update | > Transcript show: '----'; cr; show: update; cr. > config := repo versionNamed: update. > pkgs := String streamContents: [:s | config dependencies do: [:dep | s nextPutAll: dep package name; cr] ]. > prev ifNotNil: [ > context := nil. > (TextDiffBuilder from: prev to: pkgs) > patchSequenceDoIfMatch: [ :string | context ifNotNil: [context := context, string ] ] > ifInsert: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '+ ', string. context := ''] > ifRemove: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '- ', string. context := '']. > ]. > prev := pkgs]. > > > > |
Re: Too many update configs in trunk?
|
|
On 09.06.2011, at 15:18, Levente Uzonyi wrote: > On Thu, 9 Jun 2011, Bert Freudenberg wrote: > >> Hi folks, >> >> I just tried to update an older trunk image to the current one. It took ages. >> >> I have the impression that people are committing way more update configs than necessary. These are not cheap, but they force an incremental update process. Instead of just loading the latest version, they force me to load many many intermediate versions. I cannot imagine it is necessary to have so many. We are close to 200 now. > > The intermediate versions for some packages have to be loaded otherwise your image may break. > >> Normally we only need a new config map when the load order of packages needs to change. But I see many configs where the order does not change at all. Attached below. >> >> Sometimes it might be necessary to force a specific version to be loaded before a subsequent version. But is that really necessary that often? > > In most cases it's necessary. Before update-ul.16x.mcm I didn't rely on the package order inside mcms, so some mcms could have been avoided, but I guess their number is less than 10. > >> >> If it is indeed necessary, then we need to figure out a way to only force-loading a specific package, not all packages. That would speed up the updates quite a bit. > > If we can mark the packages which require a specific version to be loaded, then we'll have the faster update process. > One possible way is to only include those packages in the mcms which require a specific version. At the end of the update process update to the latest version all in-image packages which are also present in the repository. But changing the update process by the update process is a PITA as Chris found it out recently. Loading all packages found is not a good idea. We only want to load the packages mentioned in the config, and do it in the order given. What should work, IMHO, is when uploading a new config, only update those package versions that need changing. That is, take latest config, change version for the package you need updated, publish. There currently is no UI for this, it only allows "update from image" and "update from repository" which updates all packages. But if individual packages could be updated that should work faster. > Levente > > <rant> > Try updating a Pharo image and you'll see how snappy our update process is. > </rant> Never tried. You mean it's even slower? I agree that for day-to-day development the mcm-and-mcd updates are fast enough. We did go to some effort to make it fast, after all. However, when updating only once a month or so I think it could be faster. - Bert - >> >> ---- >> update-ul.118.mcm >> ---- >> update-ul.119.mcm >> ---- >> update-nice.120.mcm >> ---- >> update-nice.121.mcm >> ---- >> update-ul.122.mcm >> ---- >> update-ul.123.mcm >> ---- >> update-ul.124.mcm >> ---- >> update-nice.125.mcm >> ---- >> update-nice.126.mcm >> ---- >> update-nice.127.mcm >> ---- >> update-ar.128.mcm >> + Monticello >> Compiler >> EToys >> Exceptions >> Files >> FlexibleVocabularies >> Graphics >> GraphicsTests >> Installer-Core >> Kernel >> KernelTests >> MonticelloConfigurations >> - Monticello >> ---- >> update-ar.129.mcm >> ---- >> update-ar.130.mcm >> ---- >> update-nice.131.mcm >> ---- >> update-nice.132.mcm >> ---- >> update-ul.133.mcm >> ---- >> update-laza.134.mcm >> ---- >> update-nice.135.mcm >> ---- >> update-nice.136.mcm >> ---- >> update-ul.137.mcm >> ---- >> update-ul.138.mcm >> + Kernel >> Compiler >> EToys >> Exceptions >> Files >> FlexibleVocabularies >> Graphics >> GraphicsTests >> Installer-Core >> - Kernel >> ---- >> update-ar.139.mcm >> + System >> Morphic >> Multilingual >> MorphicExtras >> MorphicTests >> MultilingualTests >> Nebraska >> Network >> NetworkTests >> PackageInfo-Base >> PreferenceBrowser >> Protocols >> SMBase >> SMLoader >> ST80 >> SUnit >> SUnitGUI >> ScriptLoader >> Services-Base >> Sound >> - System >> ---- >> update-nice.140.mcm >> ---- >> update-ul.141.mcm >> ---- >> update-ul.142.mcm >> ---- >> update-nice.143.mcm >> ---- >> update-ul.144.mcm >> ---- >> update-ar.145.mcm >> ---- >> update-ul.146.mcm >> ---- >> update-bf.147.mcm >> + HelpSystem-Core >> + HelpSystem-Tests >> + Help-Squeak-Project >> ---- >> update-ul.148.mcm >> ---- >> update-ar.149.mcm >> ---- >> update-nice.150.mcm >> ---- >> update-ar.151.mcm >> + Network >> Collections >> CollectionsTests >> Compression >> Monticello >> Kernel >> Compiler >> EToys >> Exceptions >> Files >> FlexibleVocabularies >> Graphics >> GraphicsTests >> Installer-Core >> KernelTests >> MonticelloConfigurations >> System >> Morphic >> Multilingual >> MorphicExtras >> MorphicTests >> MultilingualTests >> Nebraska >> - Network >> ---- >> update-ar.152.mcm >> + Compression >> Collections >> CollectionsTests >> - Compression >> ---- >> update-ul.153.mcm >> ---- >> update-ul.154.mcm >> ---- >> update-ul.155.mcm >> ---- >> update-ul.156.mcm >> ---- >> update-ul.157.mcm >> ---- >> update-ul.158.mcm >> ---- >> update-ul.159.mcm >> ---- >> update-ul.160.mcm >> ---- >> update-ul.161.mcm >> ---- >> update-ul.162.mcm >> ---- >> update-ul.163.mcm >> ---- >> update-dtl.164.mcm >> + Help-Squeak-TerseGuide >> ---- >> update-ul.165.mcm >> ---- >> update-ul.166.mcm >> ---- >> update-ul.167.mcm >> ---- >> update-ul.168.mcm >> ---- >> update-nice.169.mcm >> ---- >> update-dtl.170.mcm >> + SystemReporter >> ---- >> update-ul.171.mcm >> ---- >> update-ul.172.mcm >> ---- >> update-ul.173.mcm >> ---- >> update-ul.174.mcm >> ---- >> update-ul.175.mcm >> ---- >> update-dtl.176.mcm >> ---- >> update-ul.177.mcm >> + BalloonTests >> ---- >> update-ul.178.mcm >> ---- >> update-cmm.179.mcm >> + TrunkScript >> ---- >> update-cmm.180.mcm >> - TrunkScript >> ---- >> update-ul.182.mcm >> ---- >> update-cmm.183.mcm >> ---- >> update-nice.184.mcm >> ---- >> update-ul.185.mcm >> ---- >> update-ul.186.mcm >> ---- >> update-ul.187.mcm >> ---- >> update-kb.188.mcm >> ---- >> update-nice.189.mcm >> ---- >> update-ul.190.mcm >> ---- >> update-ul.191.mcm >> >> >> repo := MCRepositoryGroup default repositories detect: [:r | r description = 'http://source.squeak.org/trunk']. >> updates := repo allFileNames select: [ :f | (f beginsWith: 'update-') and: [f endsWith: '.mcm'] ]. >> updates := updates asSortedCollection: [ :a :b | (a allButLast: 4) numericSuffix < (b allButLast: 4) numericSuffix]. >> Transcript clear. >> updates do: [ :update | >> Transcript show: '----'; cr; show: update; cr. >> config := repo versionNamed: update. >> pkgs := String streamContents: [:s | config dependencies do: [:dep | s nextPutAll: dep package name; cr] ]. >> prev ifNotNil: [ >> context := nil. >> (TextDiffBuilder from: prev to: pkgs) >> patchSequenceDoIfMatch: [ :string | context ifNotNil: [context := context, string ] ] >> ifInsert: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '+ ', string. context := ''] >> ifRemove: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '- ', string. context := '']. >> ]. >> prev := pkgs]. >> >> >> >> > |
Re: Too many update configs in trunk?
|
|
In reply to this post by Bert Freudenberg
Well, my comment is perpendicular, but people have been asking me for a new trunk image for a few weeks now, which could be related.
I don't have a real computer with me right now, otherwise I'd just do it tonight. I can probably find some time to do it tomorrow, unless someone wants to beat me to the punch. Doesn't solve the underlying problem, but does take some of the pain away. On Jun 9, 2011, at 2:51 PM, Bert Freudenberg <[hidden email]> wrote: > Hi folks, > > I just tried to update an older trunk image to the current one. It took ages. > > I have the impression that people are committing way more update configs than necessary. These are not cheap, but they force an incremental update process. Instead of just loading the latest version, they force me to load many many intermediate versions. I cannot imagine it is necessary to have so many. We are close to 200 now. > > Normally we only need a new config map when the load order of packages needs to change. But I see many configs where the order does not change at all. Attached below. > > Sometimes it might be necessary to force a specific version to be loaded before a subsequent version. But is that really necessary that often? > > If it is indeed necessary, then we need to figure out a way to only force-loading a specific package, not all packages. That would speed up the updates quite a bit. > > > - Bert - > > > ---- > update-ul.118.mcm > ---- > update-ul.119.mcm > ---- > update-nice.120.mcm > ---- > update-nice.121.mcm > ---- > update-ul.122.mcm > ---- > update-ul.123.mcm > ---- > update-ul.124.mcm > ---- > update-nice.125.mcm > ---- > update-nice.126.mcm > ---- > update-nice.127.mcm > ---- > update-ar.128.mcm > + Monticello > Compiler > EToys > Exceptions > Files > FlexibleVocabularies > Graphics > GraphicsTests > Installer-Core > Kernel > KernelTests > MonticelloConfigurations > - Monticello > ---- > update-ar.129.mcm > ---- > update-ar.130.mcm > ---- > update-nice.131.mcm > ---- > update-nice.132.mcm > ---- > update-ul.133.mcm > ---- > update-laza.134.mcm > ---- > update-nice.135.mcm > ---- > update-nice.136.mcm > ---- > update-ul.137.mcm > ---- > update-ul.138.mcm > + Kernel > Compiler > EToys > Exceptions > Files > FlexibleVocabularies > Graphics > GraphicsTests > Installer-Core > - Kernel > ---- > update-ar.139.mcm > + System > Morphic > Multilingual > MorphicExtras > MorphicTests > MultilingualTests > Nebraska > Network > NetworkTests > PackageInfo-Base > PreferenceBrowser > Protocols > SMBase > SMLoader > ST80 > SUnit > SUnitGUI > ScriptLoader > Services-Base > Sound > - System > ---- > update-nice.140.mcm > ---- > update-ul.141.mcm > ---- > update-ul.142.mcm > ---- > update-nice.143.mcm > ---- > update-ul.144.mcm > ---- > update-ar.145.mcm > ---- > update-ul.146.mcm > ---- > update-bf.147.mcm > + HelpSystem-Core > + HelpSystem-Tests > + Help-Squeak-Project > ---- > update-ul.148.mcm > ---- > update-ar.149.mcm > ---- > update-nice.150.mcm > ---- > update-ar.151.mcm > + Network > Collections > CollectionsTests > Compression > Monticello > Kernel > Compiler > EToys > Exceptions > Files > FlexibleVocabularies > Graphics > GraphicsTests > Installer-Core > KernelTests > MonticelloConfigurations > System > Morphic > Multilingual > MorphicExtras > MorphicTests > MultilingualTests > Nebraska > - Network > ---- > update-ar.152.mcm > + Compression > Collections > CollectionsTests > - Compression > ---- > update-ul.153.mcm > ---- > update-ul.154.mcm > ---- > update-ul.155.mcm > ---- > update-ul.156.mcm > ---- > update-ul.157.mcm > ---- > update-ul.158.mcm > ---- > update-ul.159.mcm > ---- > update-ul.160.mcm > ---- > update-ul.161.mcm > ---- > update-ul.162.mcm > ---- > update-ul.163.mcm > ---- > update-dtl.164.mcm > + Help-Squeak-TerseGuide > ---- > update-ul.165.mcm > ---- > update-ul.166.mcm > ---- > update-ul.167.mcm > ---- > update-ul.168.mcm > ---- > update-nice.169.mcm > ---- > update-dtl.170.mcm > + SystemReporter > ---- > update-ul.171.mcm > ---- > update-ul.172.mcm > ---- > update-ul.173.mcm > ---- > update-ul.174.mcm > ---- > update-ul.175.mcm > ---- > update-dtl.176.mcm > ---- > update-ul.177.mcm > + BalloonTests > ---- > update-ul.178.mcm > ---- > update-cmm.179.mcm > + TrunkScript > ---- > update-cmm.180.mcm > - TrunkScript > ---- > update-ul.182.mcm > ---- > update-cmm.183.mcm > ---- > update-nice.184.mcm > ---- > update-ul.185.mcm > ---- > update-ul.186.mcm > ---- > update-ul.187.mcm > ---- > update-kb.188.mcm > ---- > update-nice.189.mcm > ---- > update-ul.190.mcm > ---- > update-ul.191.mcm > > > repo := MCRepositoryGroup default repositories detect: [:r | r description = 'http://source.squeak.org/trunk']. > updates := repo allFileNames select: [ :f | (f beginsWith: 'update-') and: [f endsWith: '.mcm'] ]. > updates := updates asSortedCollection: [ :a :b | (a allButLast: 4) numericSuffix < (b allButLast: 4) numericSuffix]. > Transcript clear. > updates do: [ :update | > Transcript show: '----'; cr; show: update; cr. > config := repo versionNamed: update. > pkgs := String streamContents: [:s | config dependencies do: [:dep | s nextPutAll: dep package name; cr] ]. > prev ifNotNil: [ > context := nil. > (TextDiffBuilder from: prev to: pkgs) > patchSequenceDoIfMatch: [ :string | context ifNotNil: [context := context, string ] ] > ifInsert: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '+ ', string. context := ''] > ifRemove: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '- ', string. context := '']. > ]. > prev := pkgs]. > > > |
Re: Too many update configs in trunk?
|
|
In reply to this post by Bert Freudenberg
On 6/10/2011 0:39, Bert Freudenberg wrote:
> Loading all packages found is not a good idea. We only want to load the packages mentioned in the config, and do it in the order given. > > What should work, IMHO, is when uploading a new config, only update those package versions that need changing. That is, take latest config, change version for the package you need updated, publish. There currently is no UI for this, it only allows "update from image" and "update from repository" which updates all packages. But if individual packages could be updated that should work faster. I have a few doubts about how well this would work. Here is why: Let's assume the last config had Kernel-Foo.1.mcz in it. You update happily and when you're done (w/o any update.mcm) you're at Kernel-Bar.99.mcz. Now you find that you have an ordering problem which makes it so that Sockets-MeToo.123 needs to be loaded before updating Sockets further. According to your logic, this would now mean that the MCM combines Kernel-Foo.1.mcz with Sockets-MeToo.123 even though you've most likely developed Sockets-MeToo.123 on top of Kernel-Bar.99. And if loading those two together will work you'll only find out if someone actually updates from first principles; if people keep their images updated regularly they could be quite a ways past Kernel-Foo.1 already and may not even notice the problem. I'll also point out that we've had this particular problem a few of times already and it was a major pain to go back and fix the configs so that updating from first principles works again. I really wouldn't want to make this worse. That said, let me ask a different question: Why is it a problem to have 200 MCMs? If you're updating from 4.0 to 4.3 you are updating through two and a half versions of Squeak, work that took two years total by the entire community. That doesn't strike me as too much. The only thing I'd like to have is to be able to update such images to particular checkpoints (i.e., load the updates from 4.0 -> 4.1, 4.1 -> 4.2, etc). Cheers, - Andreas |
Re: Too many update configs in trunk?
|
|
On 09.06.2011, at 23:54, Andreas Raab wrote: > On 6/10/2011 0:39, Bert Freudenberg wrote: >> Loading all packages found is not a good idea. We only want to load the packages mentioned in the config, and do it in the order given. >> >> What should work, IMHO, is when uploading a new config, only update those package versions that need changing. That is, take latest config, change version for the package you need updated, publish. There currently is no UI for this, it only allows "update from image" and "update from repository" which updates all packages. But if individual packages could be updated that should work faster. > > I have a few doubts about how well this would work. Here is why: Let's assume the last config had Kernel-Foo.1.mcz in it. You update happily and when you're done (w/o any update.mcm) you're at Kernel-Bar.99.mcz. Now you find that you have an ordering problem which makes it so that Sockets-MeToo.123 needs to be loaded before updating Sockets further. According to your logic, this would now mean that the MCM combines Kernel-Foo.1.mcz with Sockets-MeToo.123 even though you've most likely developed Sockets-MeToo.123 on top of Kernel-Bar.99. And if loading those two together will work you'll only find out if someone actually updates from first principles; if people keep their images updated regularly they could be quite a ways past Kernel-Foo.1 already and may not even notice the problem. I'll also point out that we've had this particular problem a few of times already and it was a major pain to go back and fix the configs so that updating from first principles works again. I really wouldn't want to make this worse. I see your point. We don't handle dependencies between package versions, so problems with dependencies getting out of sync should be smaller with more frequent checkpoints. It still seems somewhat accidental. A way to discover the problems you mention would be to actually update from first principles every day. Build server, anyone? ;) > That said, let me ask a different question: Why is it a problem to have 200 MCMs? If you're updating from 4.0 to 4.3 you are updating through two and a half versions of Squeak, work that took two years total by the entire community. That doesn't strike me as too much. Well, it could be even more efficient :) > The only thing I'd like to have is to be able to update such images to particular checkpoints (i.e., load the updates from 4.0 -> 4.1, 4.1 -> 4.2, etc). That should not be hard if we knew which trunk config corresponds to which release. Which we don't, unfortunately. There's no tagging as in other SCMs. - Bert - |
Re: Too many update configs in trunk?
|
|
Ouch. Yeah, understanding what release a config corresponds to is probably about the most important part of the "community supported" idea (unless it's the tests.) Would it be possible to start keeping track of this stuff in the following release?
Other than the painful update thing, are there any other actual technical issues? Can we just add a note somewhere that says "BTW these bits correspond with Squeak 4.4"?
On Fri, Jun 10, 2011 at 8:12 AM, Bert Freudenberg <[hidden email]> wrote:
-- Casey Ransberger |
Re: Too many update configs in trunk?
|
|
In reply to this post by Bert Freudenberg
On Thu, 9 Jun 2011, Bert Freudenberg wrote:
> > On 09.06.2011, at 15:18, Levente Uzonyi wrote: > >> On Thu, 9 Jun 2011, Bert Freudenberg wrote: >> >>> Hi folks, >>> >>> I just tried to update an older trunk image to the current one. It took ages. >>> >>> I have the impression that people are committing way more update configs than necessary. These are not cheap, but they force an incremental update process. Instead of just loading the latest version, they force me to load many many intermediate versions. I cannot imagine it is necessary to have so many. We are close to 200 now. >> >> The intermediate versions for some packages have to be loaded otherwise your image may break. >> >>> Normally we only need a new config map when the load order of packages needs to change. But I see many configs where the order does not change at all. Attached below. >>> >>> Sometimes it might be necessary to force a specific version to be loaded before a subsequent version. But is that really necessary that often? >> >> In most cases it's necessary. Before update-ul.16x.mcm I didn't rely on the package order inside mcms, so some mcms could have been avoided, but I guess their number is less than 10. >> >>> >>> If it is indeed necessary, then we need to figure out a way to only force-loading a specific package, not all packages. That would speed up the updates quite a bit. >> >> If we can mark the packages which require a specific version to be loaded, then we'll have the faster update process. >> One possible way is to only include those packages in the mcms which require a specific version. At the end of the update process update to the latest version all in-image packages which are also present in the repository. But changing the update process by the update process is a PITA as Chris found it out recently. > > Loading all packages found is not a good idea. We only want to load the packages mentioned in the config, and do it in the order given. > > What should work, IMHO, is when uploading a new config, only update those package versions that need changing. That is, take latest config, change version for the package you need updated, publish. There currently is no UI for this, it only allows "update from image" and "update from repository" which updates all packages. But if individual packages could be updated that should work faster. I agree. > > >> Levente >> >> <rant> >> Try updating a Pharo image and you'll see how snappy our update process is. >> </rant> > > Never tried. You mean it's even slower? I agree that for day-to-day development the mcm-and-mcd updates are fast enough. We did go to some effort to make it fast, after all. It feels a lot slower. AFAIK they use an update stream with quite a lot of mczs. > > However, when updating only once a month or so I think it could be faster. No, loading an mcz for almost every change is not faster. Levente > > - Bert - > >>> >>> ---- >>> update-ul.118.mcm >>> ---- >>> update-ul.119.mcm >>> ---- >>> update-nice.120.mcm >>> ---- >>> update-nice.121.mcm >>> ---- >>> update-ul.122.mcm >>> ---- >>> update-ul.123.mcm >>> ---- >>> update-ul.124.mcm >>> ---- >>> update-nice.125.mcm >>> ---- >>> update-nice.126.mcm >>> ---- >>> update-nice.127.mcm >>> ---- >>> update-ar.128.mcm >>> + Monticello >>> Compiler >>> EToys >>> Exceptions >>> Files >>> FlexibleVocabularies >>> Graphics >>> GraphicsTests >>> Installer-Core >>> Kernel >>> KernelTests >>> MonticelloConfigurations >>> - Monticello >>> ---- >>> update-ar.129.mcm >>> ---- >>> update-ar.130.mcm >>> ---- >>> update-nice.131.mcm >>> ---- >>> update-nice.132.mcm >>> ---- >>> update-ul.133.mcm >>> ---- >>> update-laza.134.mcm >>> ---- >>> update-nice.135.mcm >>> ---- >>> update-nice.136.mcm >>> ---- >>> update-ul.137.mcm >>> ---- >>> update-ul.138.mcm >>> + Kernel >>> Compiler >>> EToys >>> Exceptions >>> Files >>> FlexibleVocabularies >>> Graphics >>> GraphicsTests >>> Installer-Core >>> - Kernel >>> ---- >>> update-ar.139.mcm >>> + System >>> Morphic >>> Multilingual >>> MorphicExtras >>> MorphicTests >>> MultilingualTests >>> Nebraska >>> Network >>> NetworkTests >>> PackageInfo-Base >>> PreferenceBrowser >>> Protocols >>> SMBase >>> SMLoader >>> ST80 >>> SUnit >>> SUnitGUI >>> ScriptLoader >>> Services-Base >>> Sound >>> - System >>> ---- >>> update-nice.140.mcm >>> ---- >>> update-ul.141.mcm >>> ---- >>> update-ul.142.mcm >>> ---- >>> update-nice.143.mcm >>> ---- >>> update-ul.144.mcm >>> ---- >>> update-ar.145.mcm >>> ---- >>> update-ul.146.mcm >>> ---- >>> update-bf.147.mcm >>> + HelpSystem-Core >>> + HelpSystem-Tests >>> + Help-Squeak-Project >>> ---- >>> update-ul.148.mcm >>> ---- >>> update-ar.149.mcm >>> ---- >>> update-nice.150.mcm >>> ---- >>> update-ar.151.mcm >>> + Network >>> Collections >>> CollectionsTests >>> Compression >>> Monticello >>> Kernel >>> Compiler >>> EToys >>> Exceptions >>> Files >>> FlexibleVocabularies >>> Graphics >>> GraphicsTests >>> Installer-Core >>> KernelTests >>> MonticelloConfigurations >>> System >>> Morphic >>> Multilingual >>> MorphicExtras >>> MorphicTests >>> MultilingualTests >>> Nebraska >>> - Network >>> ---- >>> update-ar.152.mcm >>> + Compression >>> Collections >>> CollectionsTests >>> - Compression >>> ---- >>> update-ul.153.mcm >>> ---- >>> update-ul.154.mcm >>> ---- >>> update-ul.155.mcm >>> ---- >>> update-ul.156.mcm >>> ---- >>> update-ul.157.mcm >>> ---- >>> update-ul.158.mcm >>> ---- >>> update-ul.159.mcm >>> ---- >>> update-ul.160.mcm >>> ---- >>> update-ul.161.mcm >>> ---- >>> update-ul.162.mcm >>> ---- >>> update-ul.163.mcm >>> ---- >>> update-dtl.164.mcm >>> + Help-Squeak-TerseGuide >>> ---- >>> update-ul.165.mcm >>> ---- >>> update-ul.166.mcm >>> ---- >>> update-ul.167.mcm >>> ---- >>> update-ul.168.mcm >>> ---- >>> update-nice.169.mcm >>> ---- >>> update-dtl.170.mcm >>> + SystemReporter >>> ---- >>> update-ul.171.mcm >>> ---- >>> update-ul.172.mcm >>> ---- >>> update-ul.173.mcm >>> ---- >>> update-ul.174.mcm >>> ---- >>> update-ul.175.mcm >>> ---- >>> update-dtl.176.mcm >>> ---- >>> update-ul.177.mcm >>> + BalloonTests >>> ---- >>> update-ul.178.mcm >>> ---- >>> update-cmm.179.mcm >>> + TrunkScript >>> ---- >>> update-cmm.180.mcm >>> - TrunkScript >>> ---- >>> update-ul.182.mcm >>> ---- >>> update-cmm.183.mcm >>> ---- >>> update-nice.184.mcm >>> ---- >>> update-ul.185.mcm >>> ---- >>> update-ul.186.mcm >>> ---- >>> update-ul.187.mcm >>> ---- >>> update-kb.188.mcm >>> ---- >>> update-nice.189.mcm >>> ---- >>> update-ul.190.mcm >>> ---- >>> update-ul.191.mcm >>> >>> >>> repo := MCRepositoryGroup default repositories detect: [:r | r description = 'http://source.squeak.org/trunk']. >>> updates := repo allFileNames select: [ :f | (f beginsWith: 'update-') and: [f endsWith: '.mcm'] ]. >>> updates := updates asSortedCollection: [ :a :b | (a allButLast: 4) numericSuffix < (b allButLast: 4) numericSuffix]. >>> Transcript clear. >>> updates do: [ :update | >>> Transcript show: '----'; cr; show: update; cr. >>> config := repo versionNamed: update. >>> pkgs := String streamContents: [:s | config dependencies do: [:dep | s nextPutAll: dep package name; cr] ]. >>> prev ifNotNil: [ >>> context := nil. >>> (TextDiffBuilder from: prev to: pkgs) >>> patchSequenceDoIfMatch: [ :string | context ifNotNil: [context := context, string ] ] >>> ifInsert: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '+ ', string. context := ''] >>> ifRemove: [ :string | context ifNotNil: [Transcript show: context]. Transcript show: '- ', string. context := '']. >>> ]. >>> prev := pkgs]. >>> >>> >>> >>> >> > > > |
Re: Too many update configs in trunk?
|
|
In reply to this post by Andreas.Raab
On 6/10/11, Andreas Raab <[hidden email]> wrote:
..... > > That said, let me ask a different question: Why is it a problem to have > 200 MCMs? If you're updating from 4.0 to 4.3 you are updating through > two and a half versions of Squeak, work that took two years total by the > entire community. That doesn't strike me as too much. The only thing I'd > like to have is to be able to update such images to particular > checkpoints (i.e., load the updates from 4.0 -> 4.1, 4.1 -> 4.2, etc). +1 Hannes |
Re: Too many update configs in trunk?
|
|
In reply to this post by Bert Freudenberg
On 6/9/11, Bert Freudenberg <[hidden email]> wrote:
> Hi folks, > > I just tried to update an older trunk image to the current one. It took > ages. > Bert Which version was the older trunk image? How long is "ages"? 20 min, 1 hour, 2 hours, 4 hours? How often are you going to repeat the exercise in the future? Hannes |
Re: Too many update configs in trunk?
|
|
In reply to this post by Casey Ransberger-2
On 6/10/11, Casey Ransberger <[hidden email]> wrote:
> Well, my comment is perpendicular, but people have been asking me for a new > trunk image for a few weeks now, which could be related. > > I don't have a real computer with me right now, otherwise I'd just do it > tonight. I can probably find some time to do it tomorrow, unless someone > wants to beat me to the punch. Probably not; it is still fine if you do it some time during the next week. > Doesn't solve the underlying problem, but does take some of the pain away. Yes. The last one which is there is from the 17th of April ftp://ftp.squeak.org/4.3alpha/ It would be great to have an updated image there around every 3rd day of each month with the updates of the month before. --Hannes |
Re: Too many update configs in trunk?
|
|
On 6/11/11 3:08 AM, "Hannes Hirzel" <[hidden email]> wrote: > On 6/10/11, Casey Ransberger <[hidden email]> wrote: >> Well, my comment is perpendicular, but people have been asking me for a new >> trunk image for a few weeks now, which could be related. >> >> I don't have a real computer with me right now, otherwise I'd just do it >> tonight. I can probably find some time to do it tomorrow, unless someone >> wants to beat me to the punch. > > Probably not; it is still fine if you do it some time during the next week. > >> Doesn't solve the underlying problem, but does take some of the pain away. > > Yes. > > The last one which is there is from the 17th of April > ftp://ftp.squeak.org/4.3alpha/ > > It would be great to have an updated image there around every 3rd day > of each month with the updates of the month before. > > --Hannes If no one is against, this is a task which I could do. Edgar |
Re: Too many update configs in trunk?
|
|
On 6/11/11, Edgar J. De Cleene <[hidden email]> wrote:
> On 6/11/11 3:08 AM, "Hannes Hirzel" <[hidden email]> wrote: > >> On 6/10/11, Casey Ransberger <[hidden email]> wrote: >>> Well, my comment is perpendicular, but people have been asking me for a >>> new >>> trunk image for a few weeks now, which could be related. .. >> >> Probably not; it is still fine if you do it some time during the next >> week. >> ... >> >> The last one which is there is from the 17th of April >> ftp://ftp.squeak.org/4.3alpha/ >> >> It would be great to have an updated image there around every 3rd day >> of each month with the updates of the month before. >> >> --Hannes > > > If no one is against, this is a task which I could do. > > > Edgar > This is very much welcome.... Please do so. --Hannes |
Re: Too many update configs in trunk?
|
|
On Sat, Jun 11, 2011 at 11:13:02AM +0000, Hannes Hirzel wrote:
> On 6/11/11, Edgar J. De Cleene <[hidden email]> wrote: > > On 6/11/11 3:08 AM, "Hannes Hirzel" <[hidden email]> wrote: > > > >> On 6/10/11, Casey Ransberger <[hidden email]> wrote: > >>> Well, my comment is perpendicular, but people have been asking me for a > >>> new > >>> trunk image for a few weeks now, which could be related. > .. > >> > >> Probably not; it is still fine if you do it some time during the next > >> week. > >> > ... > >> > >> The last one which is there is from the 17th of April > >> ftp://ftp.squeak.org/4.3alpha/ > >> > >> It would be great to have an updated image there around every 3rd day > >> of each month with the updates of the month before. > >> > >> --Hannes > > > > > > If no one is against, this is a task which I could do. > > > > > > Edgar > > > > This is very much welcome.... > > Please do so. > > --Hannes +1 Dave |
Re: Too many update configs in trunk?
|
|
In reply to this post by Hannes Hirzel
On 10.06.2011, at 22:59, Hannes Hirzel wrote:
> On 6/9/11, Bert Freudenberg <[hidden email]> wrote: >> Hi folks, >> >> I just tried to update an older trunk image to the current one. It took >> ages. >> > Bert > > Which version was the older trunk image? Update config 168. > How long is "ages"? 20 min, 1 hour, 2 hours, 4 hours? 15 minutes maybe? I didn't watch the clock. > How often are you going to repeat the exercise in the future? Not very often. I guess I'm just spoiled by how snappy the update process usually is ;) About 50 packages changed since 168.mcm. So ideally the update process would only have to download 50 mcds. Instead, it downloaded about 240, forced by the update maps. Which is still better than downloading all 650 mcds corresponding to each commit individually. And a lot better than downloading full mczs, as Levente pointed out. So, it's not that bad after all. - Bert - |
11 June Building report (was Re: [squeak-dev] Too many update configs in trunk?)
|
|
In reply to this post by David T. Lewis
On 6/11/11 9:56 AM, "David T. Lewis" <[hidden email]> wrote: > On Sat, Jun 11, 2011 at 11:13:02AM +0000, Hannes Hirzel wrote: >> On 6/11/11, Edgar J. De Cleene <[hidden email]> wrote: >>> On 6/11/11 3:08 AM, "Hannes Hirzel" <[hidden email]> wrote: >>> >>>> On 6/10/11, Casey Ransberger <[hidden email]> wrote: >>>>> Well, my comment is perpendicular, but people have been asking me for a >>>>> new >>>>> trunk image for a few weeks now, which could be related. >> .. >>>> >>>> Probably not; it is still fine if you do it some time during the next >>>> week. >>>> >> ... >>>> >>>> The last one which is there is from the 17th of April >>>> ftp://ftp.squeak.org/4.3alpha/ >>>> >>>> It would be great to have an updated image there around every 3rd day >>>> of each month with the updates of the month before. >>>> >>>> --Hannes >>> >>> >>> If no one is against, this is a task which I could do. >>> >>> >>> Edgar >>> >> >> This is very much welcome.... >> >> Please do so. >> >> --Hannes > > +1 > > Dave > On Mac , the image builds without any troubles. I do a Profiled Sunit test and have 3007 run, 2982 passes, 7 expected failures, 14 failures, 2 errors, 2 unexpected passes Failure list CompilerExceptionsTest>>#testUnusedVariable ExceptionTests>>#testHandlerFromAction MCFileInTest>>#testStWriter MCMczInstallerTest>>#testInstallFromFile MCMczInstallerTest>>#testInstallFromStream MirrorPrimitiveTests>>#testMirrorAt MirrorPrimitiveTests>>#testMirrorInstVarAt PackageDependencyTest>>#testExceptions PackageDependencyTest>>#testMonticello PackageDependencyTest>>#testST80 PackageDependencyTest>>#testToolBuilderMVC ProcessTest>>#testAtomicSuspend RWBinaryOrTextStreamTest>>#testExisiting ReleaseTest>>#testNoObsoleteClasses SocketTest>>#testLocalAddress SocketTest>>#testRemoteAddress Error list CompiledMethodTest>>#testPerformCanExecutelongMethodWithTemps CompiledMethodTest>>#testPerformInSuperclassCanExecutelongMethodWithTemps Now I transfer to Windows XP computer and run the test, then store on ftp Edgar |
Re: 11 June Building report (was Re: [squeak-dev] Too many update configs in trunk?)
|
|
Bonus points for running the tests: thank you for taking the time to do this.
On Jun 11, 2011, at 12:07 PM, "Edgar J. De Cleene" <[hidden email]> wrote: > > > > On 6/11/11 9:56 AM, "David T. Lewis" <[hidden email]> wrote: > >> On Sat, Jun 11, 2011 at 11:13:02AM +0000, Hannes Hirzel wrote: >>> On 6/11/11, Edgar J. De Cleene <[hidden email]> wrote: >>>> On 6/11/11 3:08 AM, "Hannes Hirzel" <[hidden email]> wrote: >>>> >>>>> On 6/10/11, Casey Ransberger <[hidden email]> wrote: >>>>>> Well, my comment is perpendicular, but people have been asking me for a >>>>>> new >>>>>> trunk image for a few weeks now, which could be related. >>> .. >>>>> >>>>> Probably not; it is still fine if you do it some time during the next >>>>> week. >>>>> >>> ... >>>>> >>>>> The last one which is there is from the 17th of April >>>>> ftp://ftp.squeak.org/4.3alpha/ >>>>> >>>>> It would be great to have an updated image there around every 3rd day >>>>> of each month with the updates of the month before. >>>>> >>>>> --Hannes >>>> >>>> >>>> If no one is against, this is a task which I could do. >>>> >>>> >>>> Edgar >>>> >>> >>> This is very much welcome.... >>> >>> Please do so. >>> >>> --Hannes >> >> +1 >> >> Dave >> > > On Mac , the image builds without any troubles. > I do a Profiled Sunit test and have > > 3007 run, 2982 passes, 7 expected failures, 14 failures, 2 errors, 2 > unexpected passes > > Failure list > CompilerExceptionsTest>>#testUnusedVariable > ExceptionTests>>#testHandlerFromAction > MCFileInTest>>#testStWriter > MCMczInstallerTest>>#testInstallFromFile > MCMczInstallerTest>>#testInstallFromStream > MirrorPrimitiveTests>>#testMirrorAt > MirrorPrimitiveTests>>#testMirrorInstVarAt > PackageDependencyTest>>#testExceptions > PackageDependencyTest>>#testMonticello > PackageDependencyTest>>#testST80 > PackageDependencyTest>>#testToolBuilderMVC > ProcessTest>>#testAtomicSuspend > RWBinaryOrTextStreamTest>>#testExisiting > ReleaseTest>>#testNoObsoleteClasses > SocketTest>>#testLocalAddress > SocketTest>>#testRemoteAddress > > Error list > CompiledMethodTest>>#testPerformCanExecutelongMethodWithTemps > CompiledMethodTest>>#testPerformInSuperclassCanExecutelongMethodWithTemps > > Now I transfer to Windows XP computer and run the test, then store on ftp > > Edgar > > > |
|
|
In reply to this post by David T. Lewis
On Windows Xp with Cog I have the attached crash dump trying to do same
Sunit test as in Mac And I do not have enough privileges to store the .zip in sftp://[hidden email]/var/www/files/4.3alpha Could store in Experiments and some could move to or wait some update privileges for storing the file. Edgar |
crash.dmp (16K)
Re: 11 June Building report (2)
|
|
Hi Edgar,
if you included Croquet.map then the C stack backtrace would be more informative. Looks like its crashing in perform:, which may indicate heap corruption. HTH
Eliot On Sat, Jun 11, 2011 at 12:27 PM, Edgar J. De Cleene <[hidden email]> wrote: On Windows Xp with Cog I have the attached crash dump trying to do same |
Re: 11 June Building report (2)
|
|
On 6/11/11 5:35 PM, "Eliot Miranda" <[hidden email]> wrote: Hi Edgar, Sure I miss something, so please tell how to use Croquet.map, the file i have reads Aug 18 2010 as date modified so seems not used by the test Edgar |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |