At yesterday's board meeting we discussed the possibility of condensing the sources file for the new release; we don't do this often, and usually for major releases but it after all five years since the last time and the changes file has got quite large.
So as an experiment we thought of - take a 5.2-19299 image (ie just the start of post-5.2) - condense the sources - update to latest (19335) ... hopefully resulting in a small changes file of everything changed from 5.2 to now. Happily this process worked without any hitches on my iMac and I now have: image -> 44.4MB changes -> 144KB sources -> 50MB The sources file has grown rather a lot from 35.2 to 50MB. I had thought that might be due to adding the EToys stuff but that was there in the 5.0 release anyway. The image has also grown from 5.0 (35.6MB) to 44.4. That's quite a lot. Now an interesting question is the most effective way to make a 32bit image release to use the same source file. Let's see if condensing sources & updating a 18229-32bit image results in an image that can use the same sources file... tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim ...now touch these wires to your tongue! |
> On 2020-02-06, at 1:05 PM, tim Rowledge <[hidden email]> wrote: > > > So as an experiment we thought of > - take a 5.2-19299 image (ie just the start of post-5.2) > - condense the sources > - update to latest (19335) > ... hopefully resulting in a small changes file of everything changed from 5.2 to now. Dammit, that's the wrong image to start from. The principle holds (I hope) but the details are just wrong. OK, out-of-cheese error +++ redo from start tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim Strange OpCodes: WFL: Wave FLag |
In reply to this post by timrowledge
Another possibility is to hook an image to a sources file: Open a newChanges file Pass 1: scan the newSources file, and for each method/class comment found and also in image, if sourceCode+timestamp matches, record method->new source position else append sourceCode to new changes, record method->new changes position. Pass 2: scan all existing method/class comment in image if found in recorded, alter remembered sourceCode position else append sourceCode to new changes, alter remembered sourceCode position changes the sources and changes to newSources and newChanges Le jeu. 6 févr. 2020 à 22:05, tim Rowledge <[hidden email]> a écrit : At yesterday's board meeting we discussed the possibility of condensing the sources file for the new release; we don't do this often, and usually for major releases but it after all five years since the last time and the changes file has got quite large. |
In reply to this post by timrowledge
On Thu, Feb 6, 2020 at 1:05 PM tim Rowledge <[hidden email]> wrote: At yesterday's board meeting we discussed the possibility of condensing the sources file for the new release; we don't do this often, and usually for major releases but it after all five years since the last time and the changes file has got quite large. - condense changes ... so we do not have more than 1 version of a method in the changes file. ... hopefully resulting in a small changes file of everything changed from 5.2 to now. - Bert - |
In reply to this post by timrowledge
Right, so to correct the details
- start from an 18299 image - set update url to trunk - condenseSources - update to 19355 - condenseChanges ... for both a 32 bit and 64 bit image. 64bit image -> 46.6MB, down from 49.6 changes -> 3.3MB, down from 14.4MB sources -> 47.4MB, up from 35.2MB 32bit image -> 38.1MB, down from 41.3MB changes -> 3.3MB, down from 14.4MB sources 47.4MB, up from 35.2MB However, life is a touch more complicated because I notice that the file sizes are not identical changes 64 bit (3302430) 32bit (3307752) sources 64bit (47374222) 32bit (47375119) It is clear that the files differ significantly, though mostly in the order that methods get written out. The different sizes are a puzzle but without the methods in the same order it is a bit difficult to work out exactly what is really different. I'm going to guess right now that it has something to do with the hashing differing slightly when building the list of methods in, for example, ClassDescription>>#moveChangesTo: and the classes/traits in SmalltalkImage>>#allClassesAndTraitsDo: .... and yes, that seems to be the case. If we eval - `String streamContents:[:s| Smalltalk allClassesAndTraitsDo: [:classOrTrait | s nextPutAll: classOrTrait name; cr]]` in a 32bit image we get a quite different list to in a 64bit image. Changing to - `String streamContents: [:s| (Smalltalk allClassesAndTraits sorted: [:a :b | a name < b name]) do:[:nm| s nextPutAll: nm name; cr] ]` provides us with the identical results we need. At last! If I edit the condenseChanges method to do this sorting, then we almost get identical changes files. THe differences are a bit strange: - in the 32bit file, we have two copies of EFontBDFFontReaderForRanges>>#readCharactersInRanges2:storeInto: one after the other and then two copies of EFontBDFFontReaderForRanges>>#readRangesForSimplifiedChinese:overrideWith:otherRanges:additionalOverrideRange: - this despite there only being one copy of each in the image? - we have different versions of SMSqueakMap>>#loadFullFrom: - in the 32bit system we have a newer version than in the 64bit (my 1/18/2019 15:55 version as opposed to goran's 5/5/2006 02:07). Both 'original' 18229 images have the older method. Looks like for some reason my SMBase-tpr.141.mcz exists only in the squeak52 repository and never made it to the trunk repository! So sometihng faintly odd must have happened somewhere. NO idea... This is my solution so far - And just to make life a bit more fun, attempting to catch the moving of #readCharactersInRanges2:storeInto: by sticking a halt in provides a very, very, broken debugger stack. Rather like the problems recently discussed. Basically bad enough that we really can't make this a release. tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim If at first you don't succeed, destroy all evidence that you tried. SourceCodeCondensing.1.cs (5K) Download Attachment |
In reply to this post by timrowledge
Hi Tim,
> On Feb 6, 2020, at 1:05 PM, tim Rowledge <[hidden email]> wrote: > > At yesterday's board meeting we discussed the possibility of condensing the sources file for the new release; we don't do this often, and usually for major releases but it after all five years since the last time and the changes file has got quite large. > > So as an experiment we thought of > - take a 5.2-19299 image (ie just the start of post-5.2) > - condense the sources > - update to latest (19335) > ... hopefully resulting in a small changes file of everything changed from 5.2 to now. > > Happily this process worked without any hitches on my iMac and I now have: > image -> 44.4MB > changes -> 144KB > sources -> 50MB > > The sources file has grown rather a lot from 35.2 to 50MB. I had thought that might be due to adding the EToys stuff but that was there in the 5.0 release anyway. The image has also grown from 5.0 (35.6MB) to 44.4. That's quite a lot. > > Now an interesting question is the most effective way to make a 32bit image release to use the same source file. Let's see if condensing sources & updating a 18229-32bit image results in an image that can use the same sources file... The “right” way, until we have written a 64-bit to 32-bit image converter, is to do the release in a 32-bit image and then run it through the 32-bit yo 64-bit converter yo derive the 64-bit pre-release image, which should be started up and snapshotted before being used as a release image. Another way is to do a release on the 32-bit image and then just slam in the right source pointers. Dumping the source pointers from the 64-bit image is a trivial undertaking. Reading in the dumped source pointers and applying them should be only slightly less trivial. But given that soon macs that run 32-bit images (on spur vms) won’t be available I guess I really should try and write the converter 64-bit to 32-bit converter. > tim > -- > tim Rowledge; [hidden email]; http://www.rowledge.org/tim > ...now touch these wires to your tongue! _,,,^..^,,,_ (phone) |
On Thu, Feb 06, 2020 at 05:02:36PM -0800, Eliot Miranda wrote:
> Hi Tim, > > > On Feb 6, 2020, at 1:05 PM, tim Rowledge <[hidden email]> wrote: > > > > ???At yesterday's board meeting we discussed the possibility of condensing the sources file for the new release; we don't do this often, and usually for major releases but it after all five years since the last time and the changes file has got quite large. > > > > So as an experiment we thought of > > - take a 5.2-19299 image (ie just the start of post-5.2) > > - condense the sources > > - update to latest (19335) > > ... hopefully resulting in a small changes file of everything changed from 5.2 to now. > > > > Happily this process worked without any hitches on my iMac and I now have: > > image -> 44.4MB > > changes -> 144KB > > sources -> 50MB > > > > The sources file has grown rather a lot from 35.2 to 50MB. I had thought that might be due to adding the EToys stuff but that was there in the 5.0 release anyway. The image has also grown from 5.0 (35.6MB) to 44.4. That's quite a lot. > > > > Now an interesting question is the most effective way to make a 32bit image release to use the same source file. Let's see if condensing sources & updating a 18229-32bit image results in an image that can use the same sources file... > > The ???right??? way, until we have written a 64-bit to 32-bit image converter, is to do the release in a 32-bit image and then run it through the 32-bit yo 64-bit converter yo derive the 64-bit pre-release image, which should be started up and snapshotted before being used as a release image. > Yes that is the right way to do it. And if it is a lot of work and/or confusion, then the best thing for 5.3 is don't worry about condensing the sources for this release. It's definitely a "nice to have" and not a requirement for Squeak 5.3. > Another way is to do a release on the 32-bit image and then just slam in the right source pointers. Dumping the source pointers from the 64-bit image is a trivial undertaking. Reading in the dumped source pointers and applying them should be only slightly less trivial. > > But given that soon macs that run 32-bit images (on spur vms) won???t be available I guess I really should try and write the converter 64-bit to 32-bit converter. > A 64-bit to 32-bit image converter would be a very good thing to have in our toolkit :-) Dave |
> On 2020-02-06, at 6:03 PM, David T. Lewis <[hidden email]> wrote: > And if it is a lot of work and/or > confusion, then the best thing for 5.3 is don't worry about condensing > the sources for this release. It's definitely a "nice to have" and not > a requirement for Squeak 5.3. Oh, definitely. The *important* thing here is actually that we have three problems revealed by this experiment - a) somehow an update ended up in the 52 update stream but did not get into the trunk stream and so gets missed out. It's not a fantastically crucial change fortunately but it illuminates a process fault we need to care about b) somehow a couple of methods get written out twice by the condense code but only in the 32bit system. Now *that* is the sort of thing to make you lose sleep. c) worst of all, it exposed another major debugger problem, possibly related to the stuff recently discussed in the Fixing the infinite debugger chains? (was: Code simulation error (was Re: I broke the debugger?)) thread(s). A debugger that is buggered is sometihng I claim is a show stopper. tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim If you must choose between two evils, pick the one you've never tried before. |
In reply to this post by David T. Lewis
> On 07.02.2020, at 03:03, David T. Lewis <[hidden email]> wrote: > > On Thu, Feb 06, 2020 at 05:02:36PM -0800, Eliot Miranda wrote: >> Hi Tim, >> >>> On Feb 6, 2020, at 1:05 PM, tim Rowledge <[hidden email]> wrote: >>> >>> ???At yesterday's board meeting we discussed the possibility of condensing the sources file for the new release; we don't do this often, and usually for major releases but it after all five years since the last time and the changes file has got quite large. >>> >>> So as an experiment we thought of >>> - take a 5.2-19299 image (ie just the start of post-5.2) >>> - condense the sources >>> - update to latest (19335) >>> ... hopefully resulting in a small changes file of everything changed from 5.2 to now. >>> >>> Happily this process worked without any hitches on my iMac and I now have: >>> image -> 44.4MB >>> changes -> 144KB >>> sources -> 50MB >>> >>> The sources file has grown rather a lot from 35.2 to 50MB. I had thought that might be due to adding the EToys stuff but that was there in the 5.0 release anyway. The image has also grown from 5.0 (35.6MB) to 44.4. That's quite a lot. >>> >>> Now an interesting question is the most effective way to make a 32bit image release to use the same source file. Let's see if condensing sources & updating a 18229-32bit image results in an image that can use the same sources file... >> >> The ???right??? way, until we have written a 64-bit to 32-bit image converter, is to do the release in a 32-bit image and then run it through the 32-bit yo 64-bit converter yo derive the 64-bit pre-release image, which should be started up and snapshotted before being used as a release image. >> > > Yes that is the right way to do it. And if it is a lot of work and/or > confusion, then the best thing for 5.3 is don't worry about condensing > the sources for this release. It's definitely a "nice to have" and not > a requirement for Squeak 5.3. > >> Another way is to do a release on the 32-bit image and then just slam in the right source pointers. Dumping the source pointers from the 64-bit image is a trivial undertaking. Reading in the dumped source pointers and applying them should be only slightly less trivial. >> >> But given that soon macs that run 32-bit images (on spur vms) won???t be available I guess I really should try and write the converter 64-bit to 32-bit converter. >> > > A 64-bit to 32-bit image converter would be a very good thing to have > in our toolkit :-) Or, you know, making the VMs understand both formats ;) Best regards -Tobias |
In reply to this post by timrowledge
On Thu, Feb 06, 2020 at 10:13:53PM -0800, tim Rowledge wrote:
> > > > On 2020-02-06, at 6:03 PM, David T. Lewis <[hidden email]> wrote: > > And if it is a lot of work and/or > > confusion, then the best thing for 5.3 is don't worry about condensing > > the sources for this release. It's definitely a "nice to have" and not > > a requirement for Squeak 5.3. > > Oh, definitely. The *important* thing here is actually that we have three problems revealed by this experiment - > a) somehow an update ended up in the 52 update stream but did not get into the trunk stream and so gets missed out. It's not a fantastically crucial change fortunately but it illuminates a process fault we need to care about > I looked through all the packages in the squeak52 repository, and found a number of inconsistencies. In the squeak52 repository from our last release, we have: EToys-asarch.338 - Not present in trunk, empty commit notice, and no code changes. GraphicsTests-jpl.47 - Not present in trunk, empty commit notice, and no code changes. Installer-Core-cmm.427 - Not present in trunk. I think this is actually the same as Installer-Core-dtl.425 in trunk. The one in squeak52 has an earlier commit date than the one in trunk. IIRC, this was an emergency fix to make the squeak52 update stream function, which explains why it appears there first. I think that I did the actual changes on this, and probably had them sitting in the inbox. Chris may have put them in squeak52 as a hot fix to get the update stream working. A bit messy, but at least it's explainable. SMBase-tpr.141 - Not present in trunk. Contains changes that presumably should be in trunk. WebClient-HTTP-ph.7 - Not present in trunk. Commit notice says "added a method for times feed" but there are no actual code changes. So there appear to be three junk commits with no actual content, one messy but explainable package discrepancy, and one commit that should be pulled into trunk immediately after the 5.3 release. The three junk commits should probably be left in place, since they are harmless and are probably present in a lot of package-cache directories, but it would be good to understand how they got there. Dave |
In reply to this post by Tobias Pape
On Fri, Feb 07, 2020 at 08:41:36AM +0100, Tobias Pape wrote:
> > > On 07.02.2020, at 03:03, David T. Lewis <[hidden email]> wrote: > > > > A 64-bit to 32-bit image converter would be a very good thing to have > > in our toolkit :-) > > Or, you know, making the VMs understand both formats ;) > Changing the subject line, because this is really a different topic. It is very easy to set up VM to run various image formats. Technically, it would be hard to do this from a single compiled VM executable, but fortunately that is not needed. The only requirement is an agreed naming convention for the installations. I've posted a run script to handle this several times over the years, but the key thing is this: # Scripts for running various interpreters INTERP_SCRIPT="squeak" # Context VM for 32 and 64 bit images COG_SCRIPT="cog" # Cog VM SPUR_SCRIPT="spur" # Spur VM for 32-bit Spur image SPUR64_SCRIPT="spur64" # Spur VM for 64-bit Spur image As long as each flavor of VM is installed in a distict directory under /usr/local/lib/squeak, and as long as the run scripts that point to those directories have distinct names in /usr/local/bin, then the selection of a correct VM executable for any given image format is handled from a single shell script. I call it /usr/local/bin/run on my PC, hence "$ run squeak.image" for any flavor of squeak.image. I can re-post my run script if there is an interest, but it's useless without establishing a naming convention. Is there any interest? Dave |
On Fri, Feb 7, 2020 at 11:07 AM David T. Lewis <[hidden email]> wrote: On Fri, Feb 07, 2020 at 08:41:36AM +0100, Tobias Pape wrote: Yes. I've already gone some way to producing a 64-bit to 32-bit Spur image converter. The refactored code which shares almost all methods between the 32->64 and 64->32 is committed, and 32->64 works. I'll work on finishing 64->32 when time allows. _,,,^..^,,,_ best, Eliot |
In reply to this post by David T. Lewis
> On 07.02.2020, at 20:07, David T. Lewis <[hidden email]> wrote: > > Technically, > it would be hard to do this from a single compiled VM executable, but > fortunately that is not needed. It's possible, tho. RSqueak at one point happily read 64bit and 32bit images all the same… Best regards -Tobias |
Tobias, On Fri, Feb 7, 2020 at 11:21 AM Tobias Pape <[hidden email]> wrote:
Of course it's possible. But it makes no sense to me. One only needs to convert once so why complicate the Vm including an extremely rarely used action that is slooow? If we include this facility soon enough people will complain that launching a 32-bit image takes ages. It is *much better engineering* to require people to convert once using a separate tool than complicating the existing production VM with a facility that is hardly ever used.
_,,,^..^,,,_ best, Eliot |
On Fri, Feb 7, 2020 at 11:33 AM Eliot Miranda <[hidden email]> wrote:
And if *you* want the facility, why don't *you* write it and test it, instead of expecting me to add it as yet another task to my high list of completely unfunded things to do for the VM? _,,,^..^,,,_ best, Eliot |
In reply to this post by Tobias Pape
On Fri, Feb 07, 2020 at 08:21:10PM +0100, Tobias Pape wrote:
> > > On 07.02.2020, at 20:07, David T. Lewis <[hidden email]> wrote: > > > > Technically, > > it would be hard to do this from a single compiled VM executable, but > > fortunately that is not needed. > > It's possible, tho. RSqueak at one point happily read 64bit and 32bit images all the same??? > Yes, and I expect that SqueakJS can do it. It's a bit trickier with the compiled VMs. I expect that a motivated person could make it happen for the compiled VMs, but these VMs depend heavily on source inlining (Slang to C level inlining) for performance, and this complicates any scheme for invoking functions to handle two different object point sizes. In the end, I doubt that it would end up being worth the effort(*). For the interpreter VM, the executables to handle 32-bit V3 images and 64-bit V3 images are compiled from a single code base, and installed with a naming convention to make the two sets of executable files findable. The same principle applies to the Cog/Spur VMs. If the various flavors do not overwrite one another on installation, then you can install as many variations as you want. Dave (*) But don't let my comment stop anyone from trying! It would be an interesting challenge to see if it can be done. |
In reply to this post by Eliot Miranda-2
Hi Eliot,
On Fri, Feb 07, 2020 at 11:36:12AM -0800, Eliot Miranda wrote: > On Fri, Feb 7, 2020 at 11:33 AM Eliot Miranda <[hidden email]> > wrote: > > > Tobias, > > > > On Fri, Feb 7, 2020 at 11:21 AM Tobias Pape <[hidden email]> wrote: > > > >> > >> > On 07.02.2020, at 20:07, David T. Lewis <[hidden email]> wrote: > >> > > >> > Technically, > >> > it would be hard to do this from a single compiled VM executable, but > >> > fortunately that is not needed. > >> > >> It's possible, tho. RSqueak at one point happily read 64bit and 32bit > >> images all the same??? > >> > > > > Of course it's possible. But it makes no sense to me. One only needs to > > convert once so why complicate the Vm including an extremely rarely used > > action that is slooow? If we include this facility soon enough people will > > complain that launching a 32-bit image takes ages. It is *much better > > engineering* to require people to convert once using a separate tool than > > complicating the existing production VM with a facility that is hardly ever > > used. > > > > And if *you* want the facility, why don't *you* write it and test it, > instead of expecting me to add it as yet another task to my high list of > completely unfunded things to do for the VM? > Hi Eliot, I think you are reading too much into Tobias' reply :-) My "is there any interest" question was referring to the idea of coming up with distict naming conventions for the various VM installations. To say it another way, if I have Cog installed on my Linux box, and then I install 32-bit Spur, it overwrites the /usr/local/bin/squeak file. And if I then isntall 64-bit Spur, it overwrites the one for 32-bit Spur. They also conflict with the original /usr/local/bin/squeak. This is easily fixable if there is an interest, hence my question. Dave |
In reply to this post by David T. Lewis
On 08/02/20 12:37 AM, David T. Lewis wrote:
> As long as each flavor of VM is installed in a distict directory under > /usr/local/lib/squeak, and as long as the run scripts that point to > those directories have distinct names in /usr/local/bin, then the > selection of a correct VM executable for any given image format is > handled from a single shell script. I call it /usr/local/bin/run on > my PC, hence "$ run squeak.image" for any flavor of squeak.image. The default prefix path should be changed to something like $HOME/.squeak or $HOME/.config/squeak so that default installations will not require root permission. Package maintainers can always override this default to distro-specific paths while building system-wide installable packages. Regards .. Subbu |
On 08/02/20 5:58 PM, K K Subbu wrote:
> > The default prefix path should be changed to something like > $HOME/.squeak or $HOME/.config/squeak so that default installations > will not require root permission.I meant to write 'needs to be changed'. Pressed send too soon. Regards .. Subbu |
In reply to this post by Eliot Miranda-2
> On 07.02.2020, at 20:36, Eliot Miranda <[hidden email]> wrote: > > > > On Fri, Feb 7, 2020 at 11:33 AM Eliot Miranda <[hidden email]> wrote: > Tobias, > > On Fri, Feb 7, 2020 at 11:21 AM Tobias Pape <[hidden email]> wrote: > > > On 07.02.2020, at 20:07, David T. Lewis <[hidden email]> wrote: > > > > Technically, > > it would be hard to do this from a single compiled VM executable, but > > fortunately that is not needed. > > It's possible, tho. RSqueak at one point happily read 64bit and 32bit images all the same… > > Of course it's possible. But it makes no sense to me. One only needs to convert once so why complicate the Vm including an extremely rarely used action that is slooow? If we include this facility soon enough people will complain that launching a 32-bit image takes ages. It is *much better engineering* to require people to convert once using a separate tool than complicating the existing production VM with a facility that is hardly ever used. > > And if *you* want the facility, why don't *you* write it and test it, instead of expecting me to add it as yet another task to my high list of completely unfunded things to do for the VM? > Eliot, I did by no means imply that this be your duty. Not at all. In fact, if a VM gains this ability, I think it ought to be done that all conversion is (a) obvious to the user and (b) transparent to everything after interpret(). And I acknowledge that these kind of features are not the ones you should think about :) Best regards -Tobias > _,,,^..^,,,_ > best, Eliot > |
Free forum by Nabble | Edit this page |