Another metacello loading puzzle: xml
Locked
123
123
Re: Another metacello loading puzzle: xml
 
|
Hi guys,
On 1/8/18 6:35 AM, Mariano Martinez Peck wrote: > Just a small comment from the outside: that problem seems to be > known: http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html > Maybe Iwan can check if he has instances of QuadByteString too... > that might be the reason... On 08/01/2018 16:24, Dale Henrichs via Glass wrote: > Getting rid of the QuadByteStrings in the mcz file is one solution the > other solution is to save the package in Filetree format, since > Filetree uses utf8. Thanks Mariano and Dale, Yes, I eventually discovered the QuadByteString issue... I tried using filetree initially, but given the code that's there (that WonkaLoader thing) it is not straightforward to use filetree (I suspect because it uses FileTree to fetch FileTree as well, there may be a chicken-and-egg scenario going on...but I did not investigate). Instead I figured the QuadByteString issue must be a bug, since writeWordLike: is being called from code present in Gemstone, yet it is not implemented in Gemstone. I provided an implementation (should I log a bug somewhere?), and using that it actually loads using our scripts. Which is nice because it gives me a starting point of sorts from where I can refactor and simplify. Of course there's strong incentive to just leave it at that from the people who have to pay the $$ for this... But I would have liked to be able to load it all (via/not via network) using only what is provided by GsUpgrader and Metacello without all the various things in our WonkaLoader class.... It is painful to have to get all the right branches of all the right projects from all the right github repositories locally just to be able to load...being able to fetch the correct dependencies at their correct versions, from wherever, recursively is exactly the point of using metacello in the first place. Having to manage that ourselves defeats the point of using Metacello in the first place... (Besides, many of our dependencies are just Monticello packages anyways) If you want to play along, my answers to Dale: On 08/01/2018 16:24, Dale Henrichs via Glass wrote: > I highly recommend to the bootstrapGsDevKit approach, but if you don't > want to take that route, then you should try splitting your load into > two phases 1) fetch and load GLASS 2) fetch and load your application. That is actually exactly what our code does. When you call WonkaLoader class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork: (see the call in topaz/makeCIImage.topaz [1] ): - it first updates/loads GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that order, from locally cloned git repositories); and - then it loads group 'CI' of BaselineOfWonkaRuntime from wonkaRoot/monticello repository (this is where our own code sits). But, all the loading that takes place - whether loading a "bootstrap dependency" (the GLASS/Metacello stuff) or any other package to load - is done in the same method: WonkaLoader>>loadGroup [2], using the same code to create a Metacello [3] and [4]. In the first phase, (loading bootstrap dependencies), it sets up a new Monticello independently for each of those "bootstrap" projects in turn, on their own (cloned github) repository. ( See BaselineOfWOnkaRuntime class >> loadBootstrapBaseline:repository:modes: [5] ) In the second phase, it uses the same underlying method to set up a new Metacello on our own repository. What is shared in all of the above is how conflicts etc are handled; and the given repositoryOverrides are always ignored if you do it overNetwork. Further, if you DO load overNetwork, it sets the cacheRepository, and always to the same platform-dependent location (which we check into our git repo). What I understand our not-so-pretty code does in short (ignoring handling of conflicts etc) is: Metacello new baseline: xxx; repository: aLocalGitClone; cacheRepository: yyy; get. Metacello new baseline: xxx; repository: aLocalGitClone; cacheRepository: yyy; load: 'default'. For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that order) Then it does: Metacello new baseline: xxx; repository: ourOwnFileTreeRepo; cacheRepository: yyy; get. Metacello new baseline: xxx; repository: ourOwnFileTreeRepo; cacheRepository: yyy; load: 'CI'. Either way, this example (working, if done using all the complications in our scripts) contradicts the documentation [6] if my reasoning is correct. I have hastily put a version of the above (without any WonkaLoader stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github, with its transcript in logs/transcript.2018-01-09.log but have not applied my mind to that yet. Obviously this naive version still fails. > and it looks like you are running with a pure fetch load and you are > doing of fetch of the not only GLASS/GsDevKit, but your entire application. Not quite. WonkaLoader is used to load a group from BaselineOfWonkaRuntime, and the baseline only contains dependencies to stuff we need, not any of our actual own code (besides what is contained in BaselineOfWonkaRuntime package itself). ...and the two-phase load does happen, as I explained above. (I added some detail about our project layout at the bottom of the README.rst file in github) > My second thought is that I've recently been working on getting > GsDevKit/GLASS to be bootstrapped using only git repositories (i.e., no > network) and this script[2] (within the context of GsDevKit_home) does > just that. It should be straightforward to adapt the script to a > non-GsDevKit_home > > So if you are struggling with getting the networkless load to work it > may make sense for you to use the bootstrapGsDevKit script ... and then > work on getting all of the other monticello-based projects that you are > using up on github, so that your entire application load can be done > without hitting the network ... > > If you take this route then we can ignore the #'writeWordLike:' and also > ignore the cacheRepository:/repositoryOverrides issues. We are currently using git repos, at least for the GLASS bits, but we would prefer not to. We'd prefer going with cacheRepository:/repositoryOverrides and rely on Metacello to fetch whatever is needed without us having to know the details....as I explained above. Managing all those dependencies is what we understand Metacello's purpose in life is... Cant we help bash it until it works for this use case? There is a bit more to it: the way we handle load conflicts... but I'll leave that for a later discussion perhaps. [1] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImage.topaz [2] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/loadGroup..st [3] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/initializeBaseName.repository.modes.repositoryOverrides..st [4] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/metacelloForLoading.st [5] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/class/loadBootstrapBaseline.repository.modes..st [6] https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching [7] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImageUsingStandardTools.topaz -- Regards - Iwan _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 1/9/18 4:02 AM, Iwan Vosloo via Glass wrote: > Hi guys, > > On 1/8/18 6:35 AM, Mariano Martinez Peck wrote: > > Just a small comment from the outside: that problem seems to be > > known: > http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html > > Maybe Iwan can check if he has instances of QuadByteString too... > > that might be the reason... > > On 08/01/2018 16:24, Dale Henrichs via Glass wrote: >> Getting rid of the QuadByteStrings in the mcz file is one solution the > > other solution is to save the package in Filetree format, since > > Filetree uses utf8. > > > > Thanks Mariano and Dale, > > Yes, I eventually discovered the QuadByteString issue... I tried using > filetree initially, but given the code that's there (that WonkaLoader > thing) it is not straightforward to use filetree (I suspect because it > uses FileTree to fetch FileTree as well, there may be a > chicken-and-egg scenario going on...but I did not investigate). > > Instead I figured the QuadByteString issue must be a bug, since > writeWordLike: is being called from code present in Gemstone, yet it > is not implemented in Gemstone. > > I provided an implementation (should I log a bug somewhere?), and > using that it actually loads using our scripts. Which is nice because > it gives me a starting point of sorts from where I can refactor and > simplify. WideString and QuadByteString ... if it is working it is likely to be luck ... or the corruption is only showing up in a comment ... or the code compiles without error, but will fail when run ... really the only solution in your case is to "edit the Monticello .mcz file" and remove your patch ... it is much safer to have the load fail than debug the potential downstream problems... To edit a Monticello .mcz file with WideStrings, you should load the mcz file into a pharo (or squeak) image ... from there your best bet would seem to be to inspect the package structure and get the snapshot definitions and find look for MCMethodDefinitions whose #source is a kind of WideString also look for a class definition #comment that might contain a WideString ... if you find a comment or method with a WideString, then edit the method(s) and comment(s) to eliminate the WideString character(s) save a new version of the .mcz file and then edit the appropriate ConfigurationOf to point to the new .mcz version and try the load again ... If you are still getting an error or you didn't find any methods or comments with WideStrings, then it is likely that a commit comment in the .mcz contains the offending character (and there may be more than one comment with the problem) or the problem might be another of the fields ... all it takes is one WideString to spoil the entire package. Here's some sample code: | basePackageName wc comments methods versions badDefs | basePackageName := 'Metacello-Base'. wc := MCWorkingCopy allManagers detect: [:each | each packageName = basePackageName]. comments := OrderedCollection new.. methods := OrderedCollection new. wc snapshot definitions do: [:def | def isClassDefinition ifTrue: [ (def comment isKindOf: WideString) ifTrue: [ comments add: def]]. def isMethodDefinition ifTrue: [ (def source isKindOf: WideString) ifTrue: [ comments add: def]]]. badDefs := OrderedCollection new. comments isEmpty ifFalse: [ badDefs add: comments ]. methods isEmpty ifFalse: [ badDefs add: methods ]. badDefs isEmpty ifFalse: [ ^ badDefs ]. "now scan wc ancestry for WideStrings" versions := OrderedCollection new. wc ancestry allAncestorsDo: [:aMCVersionInfo | (aMCVersionInfo message isKindOf: WideString) ifTrue: [ versions add: aMCVersionInfo ] ]. versions > > Of course there's strong incentive to just leave it at that from the > people who have to pay the $$ for this... But I would have liked to be > able to load it all (via/not via network) using only what is provided > by GsUpgrader and Metacello without all the various things in our > WonkaLoader class.... Yeah the FinWorks WonkaLoader likely predates Metacello let alone GsUpgrader and GsDevKit_home... > > It is painful to have to get all the right branches of all the right > projects from all the right github repositories locally just to be > able to load... Hmmm, it seems that there is some pain associated with the current scheme and once you do clone the projects and get them to the right version then you will be able to load without network access all of the time and when it is a good time for you to update the version of a project you are using, you can do it on your own schedule when you are prepared for the porting effort ... A quick scan o f the http: projects that you are using several different http: repositories ... Other than Seaside which turns into a github download for GemStone anyway and Metacello (which doesn't need http: access if you use bootstrapGsDevKit[2]) , the other http: repositories I've identified are: http://seaside.gemtalksystems.com/ss/hyper http://smalltalkhub.com/mc/hernan/XMLPullParser/main http://smalltalkhub.com/mc/PharoExtras/XMLParser/main http://smalltalkhub.com/mc/PharoExtras/OrderPreservingDictionary/main http://www.smalltalkhub.com/mc/PharoExtras/BitmapCharacterSet/main http://smalltalkhub.com/mc/Swazoo/Swazoo/main http://seaside.gemtalksystems.com/ss/swazoo2 hyper also uses a local clone for bootstrapGsDevKit[2], but the others have not been moved to Github yet ... are you using Swazoo for your web server? If not then you are down to only 4 different projects based on http: and I'm thinking that the XML projects need to move to GitHub soon and have already heard from Monty that he would be willing to do that ... The other two OrderPreservingDictionary and BitmapCharacterSet presumably aren't changing that frequently to become difficult to manage as a git clone .... As for the github: projects that aren't already covered by bootstrapGsDevKit[2] it seems that only SeasideSt/Seaside is being used ... I don't know, but from where I am sitting, it would be easier to manage git repositories than try to figure out why your scripts aren't working:) Oh well... [1] https://raw.githubusercontent.com/finworks/BaselineOfWonkaRuntime/master/logs/transcript.2018-01-01.log [2] https://github.com/GsDevKit/GsDevKit_home/blob/issue_200/bin/bootstrapGsDevKit > being able to fetch the correct dependencies at their correct > versions, from wherever, recursively is exactly the point of using > metacello in the first place. Having to manage that ourselves defeats > the point of using Metacello in the first place.. > (Besides, many of our dependencies are just Monticello packages anyways) Things are not quite that simple ... and I would say that if you downloading individual package versions, then I would claim that a git pull is much simpler than hand editting ConfigurationOf files to change the version of the package you are using ... If you are not specifically managing the versions of packages and projects (i.e., not hand editing ConfigurationOf files) then you are at the mercy of the project developers as to which version of the project you pick up when you update and have no control (other than hand editting ConfigurationOf files) over which version you are actually using in your product ... I would argue that with git clones you know exactly which version of each of the projects you are using ... and you are in complete control over when you want to use a new version ... With all of the said, you are working with a legacy production loading scheme and if you are going to pay the price of converting the load system, I would recommend that you wait until GemStone 3.5 (planned, not committed) comes out with Rowan support ... at which time package loading, Metacello and git support will be incorporated in the GemStone kernel and be supported by GemTalk Systems instead of on community support, while still maintaining open source status for those tools... > > If you want to play along, my answers to Dale: > > > On 08/01/2018 16:24, Dale Henrichs via Glass wrote: > > I highly recommend to the bootstrapGsDevKit approach, but if you don't > > want to take that route, then you should try splitting your load into > > two phases 1) fetch and load GLASS 2) fetch and load your application. > > That is actually exactly what our code does. When you call WonkaLoader > class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork: > (see the call in topaz/makeCIImage.topaz [1] ): > - it first updates/loads GsUpgrader, FileTree, Metacello, Grease, > GLASS1 (in that order, from locally cloned git repositories); and > - then it loads group 'CI' of BaselineOfWonkaRuntime from > wonkaRoot/monticello repository (this is where our own code sits). ... dumping into the Transcript the steps that are performed by Wonka (and the Metacello or other load expression used) would help me understand where the transitions are rather than trying to interpret things from a pure Metacello log ... > > But, all the loading that takes place - whether loading a "bootstrap > dependency" (the GLASS/Metacello stuff) or any other package to load - > is done in the same method: WonkaLoader>>loadGroup [2], using the same > code to create a Metacello [3] and [4]. Okay ... skipping ahead a bit I don't see a Smalltalk stack and without the wonka loader markers in the new log[3], I don't really know when or what happened so save reading the various code snippets until I can get my hands on a stack ... [3] https://github.com/finworks/BaselineOfWonkaRuntime/commit/6316b77d63189f5c47407403d6a9f7f1e37f048e > > In the first phase, (loading bootstrap dependencies), it sets up a new > Monticello independently for each of those "bootstrap" projects in > turn, on their own (cloned github) repository. ( See > BaselineOfWOnkaRuntime class >> > loadBootstrapBaseline:repository:modes: [5] ) > > In the second phase, it uses the same underlying method to set up a > new Metacello on our own repository. > > What is shared in all of the above is how conflicts etc are handled; > and the given repositoryOverrides are always ignored if you do it > overNetwork. Further, if you DO load overNetwork, it sets the > cacheRepository, and always to the same platform-dependent location > (which we check into our git repo). > > > What I understand our not-so-pretty code does in short (ignoring > handling of conflicts etc) is: > > Metacello new > baseline: xxx; > repository: aLocalGitClone; > cacheRepository: yyy; > get. > > Metacello new > baseline: xxx; > repository: aLocalGitClone; > cacheRepository: yyy; > load: 'default'. > > For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that > order) > > Then it does: > > Metacello new > baseline: xxx; > repository: ourOwnFileTreeRepo; > cacheRepository: yyy; > get. > > Metacello new > baseline: xxx; > repository: ourOwnFileTreeRepo; > cacheRepository: yyy; > load: 'CI'. > > > Either way, this example (working, if done using all the complications > in our scripts) contradicts the documentation [6] if my reasoning is > correct. "working" to me as there is an error at the very end and I have not seen a stack ... > > > I have hastily put a version of the above (without any WonkaLoader > stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github, > with its transcript in logs/transcript.2018-01-09.log but have not > applied my mind to that yet. Obviously this naive version still fails. it's not obvious to me and without a stack I guess I cannot comment with any authority. If the load process is actually working, then I'm not exactly sure why I spent an hour or so on my weekend answering this email message only to find out that you are apparently saying "never mind at the end" ... Dale.... _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Hi Dale,
>> Either way, this example (working, if done using all the >> complications in our scripts) contradicts the documentation [6] if my >> reasoning is correct. > I guess I'm confused right now ... the log new log does not look like > "working" to me as there is an error at the very end and I have not > seen a stack ... [...] > If the load process is actually working, then I'm not exactly sure why > I spent an hour or so on my weekend answering this email message only > to find out that you are apparently saying "never mind at the end" ... Sorry, no, that's not what I meant, I did not express it clearly. (And I do appreciate your help!) What is working is: using the WonkaLoader, ie, in that code, running ./bootstrap_image.sh -n (a) The way I understand WonkaLoader code is that *in principle* it does do a version of my "almost-pseudocode" below (b), except (a) adds ways to deal with conflicts and a bunch of other stuff. (And, in (b) I am trying to use a filetree repo) >> Metacello new >> baseline: xxx; >> repository: aLocalGitClone; >> cacheRepository: yyy; >> get. >> >> Metacello new >> baseline: xxx; >> repository: aLocalGitClone; >> cacheRepository: yyy; >> load: 'default'. >> >> For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that >> order) >> >> Then it does: >> >> Metacello new >> baseline: xxx; >> repository: ourOwnFileTreeRepo; >> cacheRepository: yyy; >> get. >> >> Metacello new >> baseline: xxx; >> repository: ourOwnFileTreeRepo; >> cacheRepository: yyy; >> load: 'CI'. My reasoning is that, if (a) works, and I understand that it really does (b) with some complications added, I should be able to do (b). But just doing (b) straight up does not work (that's the log you saw) and I have not exposed its breakage for you properly last week - I ran out of time. I am busy delving deeper into it today and will report what I manage to get going. About the WideString issue and missing writeWordLike: you say: > This is not a correct solution ... the byte patterns differ between > WideString and QuadByteString ... if it is working it is likely to be > luck ... or the corruption is only showing up in a comment ... or the > code compiles without error, but will fail when run ... really the only > solution in your case is to "edit the Monticello .mcz file" and remove > your patch ... it is much safer to have the load fail than debug the > potential downstream problems... I understand, I did not try to build a proper solution - I just copied a Pharo implementation over to see what happens. Point is: is it not possible to provide a proper implementation for writeWordLike: (since there is code in GS that calls it)? This also means that whatever I *think* is working using our existing stuff might be breaking in weird ways, so I have to actually go back and sort this out better. > Here's some sample code: (thanks) > ... are you using Swazoo for your web server? I think I can ditch Swazoo, it apparently used to be used in some scenarios, but if I hit those I'll switch them to zinc. >> being able to fetch the correct dependencies at their correct >> versions, from wherever, recursively is exactly the point of using >> metacello in the first place. Having to manage that ourselves defeats >> the point of using Metacello in the first place.. >> (Besides, many of our dependencies are just Monticello packages anyways) > Things are not quite that simple ... and I would say that if you > downloading individual package versions, then I would claim that a git > pull is much simpler than hand editting ConfigurationOf files to change > the version of the package you are using ... > > If you are not specifically managing the versions of packages and > projects (i.e., not hand editing ConfigurationOf files) then you are at > the mercy of the project developers as to which version of the project > you pick up when you update and have no control (other than hand > editting ConfigurationOf files) over which version you are actually > using in your product ... > > I would argue that with git clones you know exactly which version of > each of the projects you are using ... and you are in complete control > over when you want to use a new version ... I understand your motivation for git clones. I suspect there's stuff I do not understand about "being at the mercy of project developers". I am also just used to similar tools in other ecosystems that do not rely on me having to clone git repos (apt, npm, maven, pip) - these do the recursive fetch (although some also have issues regarding conflicting dependencies). My "being used to" these tools, and from Metacello docs I thought this is what Metacello does too. What do you mean by hand-editing ConfigurationOf files? Do you mean editing our own, which includes project versions? If we are specific in that, won't we always pick up the same version files? Isn't this just like changing the debian/rules of your own deb package source? > > With all of the said, you are working with a legacy production loading > scheme and if you are going to pay the price of converting the load > system, I would recommend that you wait until GemStone 3.5 (planned, not > committed) comes out with Rowan support ... at which time package > loading, Metacello and git support will be incorporated in the GemStone > kernel and be supported by GemTalk Systems instead of on community > support, while still maintaining open source status for those tools... Thanks, I hear you. I am trying to understand the tools and my options at the moment, so this certainly is one option to keep in mind. Regarding the rest of your comments: I am working on this today again. I am hoping to delve deeper into it and will check in better logs and report back... Thank you -Iwan -- _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 1/15/18 3:11 AM, Iwan Vosloo via Glass wrote: > Hi Dale, > >>> Either way, this example (working, if done using all the >>> complications in our scripts) contradicts the documentation [6] if >>> my reasoning is correct. >> I guess I'm confused right now ... the log new log does not look like >> "working" to me as there is an error at the very end and I have not >> seen a stack ... > [...] >> If the load process is actually working, then I'm not exactly sure >> why I spent an hour or so on my weekend answering this email message >> only to find out that you are apparently saying "never mind at the >> end" ... > > Sorry, no, that's not what I meant, I did not express it clearly. (And > I do appreciate your help!) complicated problems .. > > What is working is: using the WonkaLoader, ie, in that code, running > ./bootstrap_image.sh -n (a) > > The way I understand WonkaLoader code is that *in principle* it does > do a version of my "almost-pseudocode" below (b), except (a) adds ways > to deal with conflicts and a bunch of other stuff. (And, in (b) I am > trying to use a filetree repo) > >>> Metacello new >>> baseline: xxx; >>> repository: aLocalGitClone; >>> cacheRepository: yyy; >>> get. >>> >>> Metacello new >>> baseline: xxx; >>> repository: aLocalGitClone; >>> cacheRepository: yyy; >>> load: 'default'. >>> >>> For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that >>> order) >>> >>> Then it does: >>> >>> Metacello new >>> baseline: xxx; >>> repository: ourOwnFileTreeRepo; >>> cacheRepository: yyy; >>> get. >>> >>> Metacello new >>> baseline: xxx; >>> repository: ourOwnFileTreeRepo; >>> cacheRepository: yyy; >>> load: 'CI'. > > > My reasoning is that, if (a) works, and I understand that it really > does (b) with some complications added, I should be able to do (b). > > But just doing (b) straight up does not work (that's the log you saw) > and I have not exposed its breakage for you properly last week - I ran > out of time. I am busy delving deeper into it today and will report > what I manage to get going. a log file ... from the stack I tell where in the code the error is coming from ... from the log file I should be able to tell which part of the process is being run ... and then that will connect with the scripts . I don't expect to be able to be an expert on how the overall WonkaLoad goes ... but with a stack I will be able to pinpoint where things are going wrong. > > > > > About the WideString issue and missing writeWordLike: you say: >> This is not a correct solution ... the byte patterns differ between >> > WideString and QuadByteString ... if it is working it is likely to be > > luck ... or the corruption is only showing up in a comment ... or > the > code compiles without error, but will fail when run ... really > the only > solution in your case is to "edit the Monticello .mcz file" > and remove > your patch ... it is much safer to have the load fail > than debug the > potential downstream problems... > > I understand, I did not try to build a proper solution - I just copied > a Pharo implementation over to see what happens. Point is: is it not > possible to provide a proper implementation for writeWordLike: (since > there is code in GS that calls it)? DataStream which serializes objects to a stream. Like most serializers the objects are copied to a byteStream recording class names and fields in a platform-specific fashion. With some luck and a little bit of hacking that made it possible for GemStone to be able to deserialize Monticello packages. The String implementation is common across all platforms, but when it comes to multi-byte characters each dialect does things a bit differently and in the case of Squeak/Pharo the WideString implementation is completely different from the GemStone multibyte implementations (DoubleTypeString, QuadByteString, Unicode16, and Unicode32) and I've never even attempted to create a proper converter. By the time multi-byte characters became common in pacakges, we had moved to FileTree which has proper Utf8 support and a cross-dialect serialization model ... It is probably possible to provide a converter between WideString and the GemStone character collection hierarchy, but I do not intend to spend a lot of time on Monticello ... it will be supported "forever" but I hope to make it an optional addon in GsDevKit. Over time, the kernel of GsDevKit will rely less and less on the pharo compatibility layer as the chunks of functionality are moved into the kernel. Monticello itself requires quite a bit of pharo compatibility and I have no interest in rewriting Monticello to run in the kernel -- tonel and filetree are better disk-based package formats as they are SCM neutral and Utf8 aware and we have already ported tonel, filetree and Metacello to the kernel so Monticello will be supported for folks who still rely on http: .mcz repositories and Monticello will run continue to run on top of the pharo compatibility layer (which will of course be kept and expanded --- support for specific pharo versions should be possible then ... > > This also means that whatever I *think* is working using our existing > stuff might be breaking in weird ways, so I have to actually go back > and sort this out better. You only have to worry about code that is loaded with writeWordLike: and if the issue is in the ancestry, then the corruption would affect the loaded code ... however, you may run into trouble saving packages that end up being non-String objects ... > > >> Here's some sample code: > (thanks) > > >> ... are you using Swazoo for your web server? > > I think I can ditch Swazoo, it apparently used to be used in some > scenarios, but if I hit those I'll switch them to zinc. > > >>> being able to fetch the correct dependencies at their correct >>> versions, from wherever, recursively is exactly the point of using >>> metacello in the first place. Having to manage that ourselves >>> defeats the point of using Metacello in the first place.. >>> (Besides, many of our dependencies are just Monticello packages >>> anyways) >> Things are not quite that simple ... and I would say that if you >> downloading individual package versions, then I would claim that a >> git pull is much simpler than hand editting ConfigurationOf files to >> change the version of the package you are using ... >> >> If you are not specifically managing the versions of packages and >> projects (i.e., not hand editing ConfigurationOf files) then you are >> at the mercy of the project developers as to which version of the >> project you pick up when you update and have no control (other than >> hand editting ConfigurationOf files) over which version you are >> actually using in your product ... >> >> I would argue that with git clones you know exactly which version of >> each of the projects you are using ... and you are in complete >> control over when you want to use a new version ... > > I understand your motivation for git clones. I suspect there's stuff I > do not understand about "being at the mercy of project developers". I > am also just used to similar tools in other ecosystems that do not > rely on me having to clone git repos (apt, npm, maven, pip) - these do > the recursive fetch (although some also have issues regarding > conflicting dependencies). My "being used to" these tools, and from > Metacello docs I thought this is what Metacello does too. be a replacing for the old github: to tar file download ... my concern is that if local git clones are not used, then contributing to a project that is managed by git on github becomes much more difficult ... if the git clones are loaded into the image from the start, then it is natural to be able to push any changes/bugfixes back to the originating project ... Isn't it true that for apt, npm, maven, pit, etc., that you are talking about deployment when you say that you don't need git clones? If you want to contribute to any of those projects you would need to use a git clone, right? With Smalltalk the development environment is always part of the deployment there isn't really a distinction between deployment and development ... we often develop in our deployed app .. > > What do you mean by hand-editing ConfigurationOf files? Do you mean > editing our own, which includes project versions? If we are specific > in that, won't we always pick up the same version files? Isn't this > just like changing the debian/rules of your own deb package source? If you are specific, you always get the same version files. If you want to move to a different version then you have to edit a ConfigurationOf file to change the version of each project ... and I actually think that pulling a different branch/tag/sha is actually easier than the process required to change the ConfigurationOf file ... > >> >> With all of the said, you are working with a legacy production >> loading scheme and if you are going to pay the price of converting >> the load system, I would recommend that you wait until GemStone 3.5 >> (planned, not committed) comes out with Rowan support ... at which >> time package loading, Metacello and git support will be incorporated >> in the GemStone kernel and be supported by GemTalk Systems instead of >> on community support, while still maintaining open source status for >> those tools... > > Thanks, I hear you. I am trying to understand the tools and my options > at the moment, so this certainly is one option to keep in mind. > > > Regarding the rest of your comments: I am working on this today again. > I am hoping to delve deeper into it and will check in better logs and > report back... error message ... at least for me;) Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Hi Dale.
I tried getting a version going that does kinda what WonkaLoader does, but not using the WonkaLoader class. That was tricky because it contains logic in onUpgrade/onDowngrade/onConflict which is not trivial [1]. Doing this, I managed to get it to load overNetwork using filetree repos and using GsUprader, GsDeployer etc. But, I could not get it to work without the network. I am starting to feel doing this is not productive.... So, I switched back to getting WonkaLoader to work. The issue there was that I had to make it use a filetree cache repository in order to bypass the character encoding issue with mcz files. This is now on github, but it breaks with log [2]. (Stack is included there, but not the WHOLE stack.) It would be great if you can advise on this... Just to recap (in case it helps with navigating the log output): This thing works in 2 phases as you also suggested. First phase it loads the basic projects, in this order: GsUpgrader (from a local git clone) MetacelloPreviewBootstrap (from a local mcz repo, possibly old [3]) MetacelloPreview (from a local git clone [4]) FileTree (from a local git clone) Metacello (from a local git clone) Grease (from a local git clone) GLASS1 (from a local git clone) These are prefixed with '*** Loading' in the log. The breakage happens while loading GLASS AFAIK. It never gets to the second phase, which is to load BaselineOfWonkaRuntime. [1] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImageUsingStandardTools.topaz [2] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.overNetwork.2018-01-16.log [3] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/class/loadGemStoneMetacelloPreviewBootstrap.st [4] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/class/loadGemStoneMetacelloPreview.st Thanks - Iwan -- _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 16/01/2018 14:34, Iwan Vosloo via Glass wrote: > This is now on github, but it breaks with log [2]. (Stack is included > there, but not the WHOLE stack.) It would be great if you can advise on > this... Hang on... as Murphy would have it I found this problem shortly after sending my email. It was also trying to read all the ConfigurationOfs but tried to read the (not filetree) repository as an MCDirectoryRepository. I'll report wat I achieve going past this... Thanks -Iwan -- _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 16/01/2018 14:49, Iwan Vosloo via Glass wrote:
> > I'll report wat I achieve going past this... > I have now managed to get it to load (barring a few warnings) using WonkaLoader via the network using filetree on GS3.4 [1]. When I load it bypassing the network though, things break [2] with: "Class category name 'Sport' for the class 'SpAbstractError' is inconsistent with the package name 'Sport3.010.v3'" [1] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.overNetwork.2018-01-16.2.log [2] https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.NOToverNetwork.2018-01-16.2.log Thanks - Iwan -- _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Iwan,
I see that you are using a filetree cache now, which is clever, but because .mcz files does not have a consistent file/package naming convention, when the Sport package is converted to filtree, which _does_ have an enforced package naming convention it runs into the consistency check error for filetree:( Between the "corrupt" .mcz files and improperly named .mcz files you are caught between a rock and a hard place. Now it turns out I've already moved Sport to a github project[1] and the github project for Sport does not have this inconsistency[2]. It looks like I renamed the Sport3.010.v3 package to Sport.v3 and you might be able to pull off the same trick by renaming the package in the hyper project[1] and whatever ConfigurationOf that references it - but this is wandering into complicated territory that is only becoming a problem because you are trying to use a filetree cache ... If this is the very last problem, then it might be worth it ... I think it would take an afternoon to clone all of the projects from github and create local git repos for the projects that you use that are not on github and I would recommend that you use GsDevKit[4] instead of GLASS[5], because I've already done the work of converting all of the dependent projects from .mcz to github projects that are loaded in a standard install ... the actual code in GsDevKit and GLASS is "identical" (GLASS is a fork of the GsDevKit project and pull requests are shared between the two) so it should be that difficult to switch out and pretty quickly I think that you would be networkless and loading all of your source from disk-based git repositories and you would be heading in the right direction. I have been doing builds using the GsDevKit repository since October and for the last month I've been doing active development in a GsDevKit (not GLASS) stone on a daily basis.. So I strongly suggest that you reconsider your decision to not follow my recommendation to use local git repos as a solution to your networkless load problems. Dale [1] https://github.com/GsDevKit/Sport [2] https://github.com/GsDevKit/Sport/blob/master/src/Sport.v3.package/SpAbstractError.class/properties.json [3] http://seaside.gemtalksystems.com/ss/hyper.html [4] https://github.com/GsDevKit/GsDevKit/tree/gs_port_200 [5] https://github.com/glassdb/glass On 1/16/18 5:38 AM, Iwan Vosloo via Glass wrote: > On 16/01/2018 14:49, Iwan Vosloo via Glass wrote: >> >> I'll report wat I achieve going past this... >> > > I have now managed to get it to load (barring a few warnings) using > WonkaLoader via the network using filetree on GS3.4 [1]. When I load > it bypassing the network though, things break [2] with: > > "Class category name 'Sport' for the class 'SpAbstractError' is > inconsistent with the package name 'Sport3.010.v3'" > > > > [1] > https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.overNetwork.2018-01-16.2.log > > [2] > https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.NOToverNetwork.2018-01-16.2.log > > > Thanks > - Iwan > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Hi Dale,
Thank you. I could spend more time on this today. On 17/01/2018 17:43, Dale Henrichs via Glass wrote: > I see that you are using a filetree cache now, which is clever, but > because .mcz files does not have a consistent file/package naming > convention, when the Sport package is converted to filtree, which _does_ > have an enforced package naming convention it runs into the consistency > check error for filetree:( Between the "corrupt" .mcz files and > improperly named .mcz files you are caught between a rock and a hard place. Yes, you're right - I sit on the one hand with the character encoding in some mcz's that won't be supported on gemstone, and then the naming of Sport which is not allowed by filetree. > > Now it turns out I've already moved Sport to a github project[1] and the > github project for Sport does not have this inconsistency[2]. It looks > like I renamed the Sport3.010.v3 package to Sport.v3 and you might be > able to pull off the same trick by renaming the package in the hyper > project[1] and whatever ConfigurationOf that references it - but this is > wandering into complicated territory that is only becoming a problem > because you are trying to use a filetree cache ... If this is the very > last problem, then it might be worth it ... I suspect this is indeed the last remaining issue, so I tried using your Sport from github. The only thing I am aware of that relies on Sport is GLASS itself. Since we have a local clone of the master branch of the github glassdb/glass repository, I edited its baseline to include a baseline: baseline: 'Sport' with: [ spec repository: 'github://GsDevKit/Sport:master/src' ]; ... instead of the package it currently pulls from http://seaside.gemtalksystems.com/ss/hyper In its #gs3.x section, I commented out the package it defined: " package: 'Sport' with: [ spec file: 'Sport3.010.v3-jupiter.33' ];" Having done this, on my extracted version on GitHub, I have managed to get it to load overNetwork and NOT overNetwork. I wasn't sure that commenting out the package above was the right thing to do. Perhaps you can advise? I suppose I could also use GsDevKit/GsDevkit, but am a little wary of doing that just yet, because (a) according to its readme it is experimental and (b) I don't know how using it will work when I load it into a stone that has an older version loaded from glassdb/glass. If you think I should rather switch, and have solutions to my worries here, I'd love to know. Anyways, I am still having trouble when I put all this stuff back into our project... a problem I had before also. If the above solution solves the basic loading issues, and I can get it to load in our project I would prefer to stick to this route for now and wait to see how your future plans turn out. That decision depends on whether I can get it to load when everything is copied back into our project. Something that is supposed to work, but for some reason just does not. I would appreciate it if you have any insights in to this issue....just maybe its not a big deal??? The transcript and stack for this is on pastbin [1] [1] https://pastebin.com/Jp3kxUAR Thank you - Iwan -- _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 01/22/2018 07:40 AM, Iwan Vosloo via
Glass wrote:
Hi Dale,replacing the package named Sport with a project named Sport is the right solution ... It _is_ still experimental, but it is also very close to being ready for use ... I have to figure out how to deal with git clones that have the same names but are of different versions before declaring it ready for use... (b) I don't know how using it will work when I load it into a stone that has an older version loaded from glassdb/glass.It is pretty much the same code base ... all of the glass/gsdevkit tests pass and I mentioned I've been using personally for development for a month or so and haven't hit any major issues (and fixed the minor ones I _have_hit_ --- for all intents and purposes the packages are the same with a few minor exceptions so it should work seamlessly ... If you think I should rather switch, and have solutions to my worries here, I'd love to know.I always hate to force folks to switch their production systems, but when things are "broken" how much time should be invested in patching the broken system versus investing in upgrading the production system to a more recent approach ... It seems that you are getting close to patching your system and I'd rather wait a month or so before finishing up GsDevKit/GsDevKit work Yeah we're awfully close to having you back in business, it seems ... So you are referring to the following MNU?: a MessageNotUnderstood occurred (error 2010), a GRGemStonePlatform does not understand #'bindingOf:'according to your log you are using: --transcript--'Fetched -> Grease-Core-JohanBrichau.94 --- filetree:///home/wonka/development/wonka/3rdparty/gemstone/grease/repository [00a7df6:HEAD] --- filetree:///home/wonka/development/wonka/3rdparty/gemstone/grease/repository'a git SHA of 00a7df6 In a fresh build of Seaside3.2 my image is using Grease-Core (Grease-Core-topa.109)and the method GRPlatform>>bindingOf: itself was committed 2.5 years ago commit 94ca45afcd1b00f7d320496adbaade48bdbf7ccfwhile the version of Grease you are using is from 3 years ago: commit 00a7df6f5c3e83d01d97dcb2785c2bd502e5e6adSo it looks like you need to update to a later version of Grease at a minimum or use an older version of Seaside --- from the transcript you are using the latest version of Seaside, so you really should be using the latest version of "everything", but I don't know what version of Seaisde your application was written against ... and the Seaside version should be your driving factor for moving on from here ... Dale
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Hi Dale,
On 23/01/2018 02:26, Dale Henrichs via Glass wrote: > So you are referring to the following MNU?: > > a MessageNotUnderstood occurred (error 2010), a GRGemStonePlatform > does not understand #'bindingOf:' [snip] > So it looks like you need to update to a later version of Grease at a > minimum or use an older version of Seaside --- from the transcript you > are using the latest version of Seaside, so you really should be using > the latest version of "everything", but I don't know what version of > Seaisde your application was written against ... and the Seaside version > should be your driving factor for moving on from here ... Thanks for that. I should have looked closer. This was due to our git clones arrangement - they were all reset to older commits as part of our build process. I have fixed that now and can happily load on/off network on GS3.4 I can also load the exact same baseline overNetwork on pharo3.0, but still have issues when I try it on pharo 6 or on pharo3 NOT overNetwork. I am ignoring pharo for the moment so that I can first sort out issues in our own code (this also upgrades seaside and magritte and the XML projects for us)..... and will then bug the Pharo-users list when I am ready to tackle that issue. Thanks for your patient attention! - Iwan -- _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 01/24/2018 07:35 AM, Iwan Vosloo via Glass wrote: > Hi Dale, > > > On 23/01/2018 02:26, Dale Henrichs via Glass wrote: >> So you are referring to the following MNU?: >> >> a MessageNotUnderstood occurred (error 2010), a GRGemStonePlatform >> does not understand #'bindingOf:' > [snip] >> So it looks like you need to update to a later version of Grease at a >> minimum or use an older version of Seaside --- from the transcript >> you are using the latest version of Seaside, so you really should be >> using the latest version of "everything", but I don't know what >> version of Seaisde your application was written against ... and the >> Seaside version should be your driving factor for moving on from here >> ... > > Thanks for that. I should have looked closer. This was due to our git > clones arrangement - they were all reset to older commits as part of > our build process. I have fixed that now and can happily load on/off > network on GS3.4 > > I can also load the exact same baseline overNetwork on pharo3.0, but > still have issues when I try it on pharo 6 or on pharo3 NOT overNetwork. I can help debug the pharo issues as well ... a transcript from the build and stack would help me understand what might be going wrong ... now in pharo the stack can be truncated when is quite inconvenient and I don't know of a solution to it ... but the first bit is to find out what the immediate error is in relation to the transcript information and if I need more stack screen shots are better than nothing:) > > I am ignoring pharo for the moment so that I can first sort out issues > in our own code (this also upgrades seaside and magritte and the XML > projects for us)..... and will then bug the Pharo-users list when I am > ready to tackle that issue. It is likely to be a Metacello/Monticello issue in pharo as well --- the actual error and stack would be critical in determining this --- and if it is I am probably the best person to handle it for Pharo as well :) Dale > > Thanks for your patient attention! > - Iwan > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
In reply to this post by GLASS mailing list
Dale, if you want to mirror these projects on github (like you do SS), that would be fine. The STHub PharoExtras projects are:
XMLParser BitmapCharacterSet OrderPreservingDictionary XMLWriter XPath XMLParserHTML XMLParserStAX They have no other dependencies. (XMLPullParser is unrelated and not supported by me; I recommend XMLParserStAX instead.) Upgrading within a stone isn't supported because to run on GS, these projects rely on GS compatibility .mczs and some Configuration postLoad doits (to explicitly initialize normally lazily initialized class state), and there are issues with Monticello and extension packages on GS where loading a new version of the core package deletes the extensions, and unless the extension package has a new version also, Metacello won't reload it. I wrote to you about this in 2016: > (simulating upgrading to a new version that uses the same GS compat .mcz). run the tests again and many fail, because extension methods under "*XML-Parser-GemstoneCompatibility" (eg in Collection or GsFile) disappeared when you reloaded the "XML-Parser" .mcz. This shouldn't happen. I think it's due to this bug (which was fixed on Pharo): http://lists.gforge.inria.fr/pipermail/pharo-bugtracker/2015-May/071136.html (Note: none of my extension packages overwrite any existing methods; they only add new methods and classes.) > Sent: Sunday, January 14, 2018 at 8:55 PM > From: "Dale Henrichs via Glass" <[hidden email]> > To: [hidden email] > Subject: Re: [Glass] Another metacello loading puzzle: xml > > > > On 1/9/18 4:02 AM, Iwan Vosloo via Glass wrote: > > Hi guys, > > > > On 1/8/18 6:35 AM, Mariano Martinez Peck wrote: > > > Just a small comment from the outside: that problem seems to be > > > known: > > http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html > > > Maybe Iwan can check if he has instances of QuadByteString too... > > > that might be the reason... > > > > On 08/01/2018 16:24, Dale Henrichs via Glass wrote: > >> Getting rid of the QuadByteStrings in the mcz file is one solution the > > > other solution is to save the package in Filetree format, since > > > Filetree uses utf8. > > > > > > > > Thanks Mariano and Dale, > > > > Yes, I eventually discovered the QuadByteString issue... I tried using > > filetree initially, but given the code that's there (that WonkaLoader > > thing) it is not straightforward to use filetree (I suspect because it > > uses FileTree to fetch FileTree as well, there may be a > > chicken-and-egg scenario going on...but I did not investigate). > > > > Instead I figured the QuadByteString issue must be a bug, since > > writeWordLike: is being called from code present in Gemstone, yet it > > is not implemented in Gemstone. > > > > I provided an implementation (should I log a bug somewhere?), and > > using that it actually loads using our scripts. Which is nice because > > it gives me a starting point of sorts from where I can refactor and > > simplify. > This is not a correct solution ... the byte patterns differ between > WideString and QuadByteString ... if it is working it is likely to be > luck ... or the corruption is only showing up in a comment ... or the > code compiles without error, but will fail when run ... really the only > solution in your case is to "edit the Monticello .mcz file" and remove > your patch ... it is much safer to have the load fail than debug the > potential downstream problems... > > To edit a Monticello .mcz file with WideStrings, you should load the mcz > file into a pharo (or squeak) image ... from there your best bet would > seem to be to inspect the package structure and get the snapshot > definitions and find look for MCMethodDefinitions whose #source is a > kind of WideString also look for a class definition #comment that might > contain a WideString ... if you find a comment or method with a > WideString, then edit the method(s) and comment(s) to eliminate the > WideString character(s) save a new version of the .mcz file and then > edit the appropriate ConfigurationOf to point to the new .mcz version > and try the load again ... > > If you are still getting an error or you didn't find any methods or > comments with WideStrings, then it is likely that a commit comment in > the .mcz contains the offending character (and there may be more than > one comment with the problem) or the problem might be another of the > fields ... all it takes is one WideString to spoil the entire package. > > Here's some sample code: > > | basePackageName wc comments methods versions badDefs | > basePackageName := 'Metacello-Base'. > wc := MCWorkingCopy allManagers detect: [:each | each packageName = > basePackageName]. > comments := OrderedCollection new.. > methods := OrderedCollection new. > wc snapshot definitions do: [:def | > def isClassDefinition > ifTrue: [ (def comment isKindOf: WideString) ifTrue: [ comments > add: def]]. > def isMethodDefinition > ifTrue: [ (def source isKindOf: WideString) ifTrue: [ comments > add: def]]]. > badDefs := OrderedCollection new. > comments isEmpty ifFalse: [ badDefs add: comments ]. > methods isEmpty ifFalse: [ badDefs add: methods ]. > badDefs isEmpty ifFalse: [ ^ badDefs ]. > "now scan wc ancestry for WideStrings" > versions := OrderedCollection new. > wc ancestry allAncestorsDo: [:aMCVersionInfo | > (aMCVersionInfo message isKindOf: WideString) ifTrue: [ versions > add: aMCVersionInfo ] ]. > versions > > > > Of course there's strong incentive to just leave it at that from the > > people who have to pay the $$ for this... But I would have liked to be > > able to load it all (via/not via network) using only what is provided > > by GsUpgrader and Metacello without all the various things in our > > WonkaLoader class.... > Yeah the FinWorks WonkaLoader likely predates Metacello let alone > GsUpgrader and GsDevKit_home... > > > > It is painful to have to get all the right branches of all the right > > projects from all the right github repositories locally just to be > > able to load... > Hmmm, it seems that there is some pain associated with the current > scheme and once you do clone the projects and get them to the right > version then you will be able to load without network access all of the > time and when it is a good time for you to update the version of a > project you are using, you can do it on your own schedule when you are > prepared for the porting effort ... > > A quick scan o f the http: projects that you are using several different > http: repositories ... Other than Seaside which turns into a github > download for GemStone anyway and Metacello (which doesn't need http: > access if you use bootstrapGsDevKit[2]) , the other http: repositories > I've identified are: > > http://seaside.gemtalksystems.com/ss/hyper > http://smalltalkhub.com/mc/hernan/XMLPullParser/main > http://smalltalkhub.com/mc/PharoExtras/XMLParser/main > http://smalltalkhub.com/mc/PharoExtras/OrderPreservingDictionary/main > http://www.smalltalkhub.com/mc/PharoExtras/BitmapCharacterSet/main > http://smalltalkhub.com/mc/Swazoo/Swazoo/main > http://seaside.gemtalksystems.com/ss/swazoo2 > > hyper also uses a local clone for bootstrapGsDevKit[2], but the others > have not been moved to Github yet ... are you using Swazoo for your web > server? If not then you are down to only 4 different projects based on > http: and I'm thinking that the XML projects need to move to GitHub soon > and have already heard from Monty that he would be willing to do that > ... The other two OrderPreservingDictionary and BitmapCharacterSet > presumably aren't changing that frequently to become difficult to manage > as a git clone .... > > As for the github: projects that aren't already covered by > bootstrapGsDevKit[2] it seems that only SeasideSt/Seaside is being used ... > > I don't know, but from where I am sitting, it would be easier to manage > git repositories than try to figure out why your scripts aren't working:) > > Oh well... > > [1] > https://raw.githubusercontent.com/finworks/BaselineOfWonkaRuntime/master/logs/transcript.2018-01-01.log > [2] > https://github.com/GsDevKit/GsDevKit_home/blob/issue_200/bin/bootstrapGsDevKit > > > being able to fetch the correct dependencies at their correct > > versions, from wherever, recursively is exactly the point of using > > metacello in the first place. Having to manage that ourselves defeats > > the point of using Metacello in the first place.. > > (Besides, many of our dependencies are just Monticello packages anyways) > Things are not quite that simple ... and I would say that if you > downloading individual package versions, then I would claim that a git > pull is much simpler than hand editting ConfigurationOf files to change > the version of the package you are using ... > > If you are not specifically managing the versions of packages and > projects (i.e., not hand editing ConfigurationOf files) then you are at > the mercy of the project developers as to which version of the project > you pick up when you update and have no control (other than hand > editting ConfigurationOf files) over which version you are actually > using in your product ... > > I would argue that with git clones you know exactly which version of > each of the projects you are using ... and you are in complete control > over when you want to use a new version ... > > With all of the said, you are working with a legacy production loading > scheme and if you are going to pay the price of converting the load > system, I would recommend that you wait until GemStone 3.5 (planned, not > committed) comes out with Rowan support ... at which time package > loading, Metacello and git support will be incorporated in the GemStone > kernel and be supported by GemTalk Systems instead of on community > support, while still maintaining open source status for those tools... > > > > If you want to play along, my answers to Dale: > > > > > > On 08/01/2018 16:24, Dale Henrichs via Glass wrote: > > > I highly recommend to the bootstrapGsDevKit approach, but if you don't > > > want to take that route, then you should try splitting your load into > > > two phases 1) fetch and load GLASS 2) fetch and load your application. > > > > That is actually exactly what our code does. When you call WonkaLoader > > class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork: > > (see the call in topaz/makeCIImage.topaz [1] ): > > - it first updates/loads GsUpgrader, FileTree, Metacello, Grease, > > GLASS1 (in that order, from locally cloned git repositories); and > > - then it loads group 'CI' of BaselineOfWonkaRuntime from > > wonkaRoot/monticello repository (this is where our own code sits). > Okay that makes sense, but I can't tell that from the loading Transcript > ... dumping into the Transcript the steps that are performed by Wonka > (and the Metacello or other load expression used) would help me > understand where the transitions are rather than trying to interpret > things from a pure Metacello log ... > > > > But, all the loading that takes place - whether loading a "bootstrap > > dependency" (the GLASS/Metacello stuff) or any other package to load - > > is done in the same method: WonkaLoader>>loadGroup [2], using the same > > code to create a Metacello [3] and [4]. > Okay ... skipping ahead a bit I don't see a Smalltalk stack and without > the wonka loader markers in the new log[3], I don't really know when or > what happened so save reading the various code snippets until I can get > my hands on a stack ... > > [3] > https://github.com/finworks/BaselineOfWonkaRuntime/commit/6316b77d63189f5c47407403d6a9f7f1e37f048e > > > > In the first phase, (loading bootstrap dependencies), it sets up a new > > Monticello independently for each of those "bootstrap" projects in > > turn, on their own (cloned github) repository. ( See > > BaselineOfWOnkaRuntime class >> > > loadBootstrapBaseline:repository:modes: [5] ) > > > > In the second phase, it uses the same underlying method to set up a > > new Metacello on our own repository. > > > > What is shared in all of the above is how conflicts etc are handled; > > and the given repositoryOverrides are always ignored if you do it > > overNetwork. Further, if you DO load overNetwork, it sets the > > cacheRepository, and always to the same platform-dependent location > > (which we check into our git repo). > > > > > > What I understand our not-so-pretty code does in short (ignoring > > handling of conflicts etc) is: > > > > Metacello new > > baseline: xxx; > > repository: aLocalGitClone; > > cacheRepository: yyy; > > get. > > > > Metacello new > > baseline: xxx; > > repository: aLocalGitClone; > > cacheRepository: yyy; > > load: 'default'. > > > > For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that > > order) > > > > Then it does: > > > > Metacello new > > baseline: xxx; > > repository: ourOwnFileTreeRepo; > > cacheRepository: yyy; > > get. > > > > Metacello new > > baseline: xxx; > > repository: ourOwnFileTreeRepo; > > cacheRepository: yyy; > > load: 'CI'. > > > > > > Either way, this example (working, if done using all the complications > > in our scripts) contradicts the documentation [6] if my reasoning is > > correct. > I guess I'm confused right now ... the log new log does not look like > "working" to me as there is an error at the very end and I have not seen > a stack ... > > > > > > I have hastily put a version of the above (without any WonkaLoader > > stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github, > > with its transcript in logs/transcript.2018-01-09.log but have not > > applied my mind to that yet. Obviously this naive version still fails. > it's not obvious to me and without a stack I guess I cannot comment with > any authority. > > If the load process is actually working, then I'm not exactly sure why I > spent an hour or so on my weekend answering this email message only to > find out that you are apparently saying "never mind at the end" ... > > Dale.... > > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass > Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 01/26/2018 04:52 PM, monty via Glass
wrote:
Monty,I wrote to you about this in 2016:(simulating upgrading to a new version that uses the same GS compat .mcz). run the tests again and many fail, because extension methods under "*XML-Parser-GemstoneCompatibility" (eg in Collection or GsFile) disappeared when you reloaded the "XML-Parser" .mcz. This shouldn't happen. I think it's due to this bug (which was fixed on Pharo): http://lists.gforge.inria.fr/pipermail/pharo-bugtracker/2015-May/071136.html(Note: none of my extension packages overwrite any existing methods; they only add new methods and classes.) This is not a "gemstone bug" per se. Monticello's package naming rules prohibit having two packages with a common prefix (i.e., XML-Parser and XML-Parser-GemstoneCompatibility cannot be used as names for two separate packages). This rule has been in effect since the beginning of Monticello. A couple years ago Pharo began unilaterally violating this "convention". I "accidentally" found out about this when I tried to port Voyage into GemStone ... it is not a simple matter of "fixing a bug" ... Pharo redefined their whole package definition model at this time and it took them a year to get it right and since this action was taken unilaterally it has not fit into my schedule to completely redesign the package management scheme in GemStone ... so this is actually a Pharo bug ... The simple solution is to go back and use the portable Monticello naming conventions for projects that are intended to be portable ... For GemStone 3.5 we are finally going to support a project/package management system in the base image of GemStone and it is likely that we will not be constrained by the Monticello naming conventions for Filetree and Tonel repositories, however, it is entirely possible that the Monticello support for GemStone (.mcz files) will continue to support only the portable naming conventions, since the new work is completely independent of the current Monticello implementation and Monticello support will not be in the base image, but added as an extension for GLASS/GsDevKit ... I appreciate the offer to let us mirror the repos and when I do I will have to rename the package to conform to the portable Monticello naming convention, but since there are folks that are using the .mcz versions perhaps you should rename your .mcz packages to conform to the portable naming convention as that will like resolve the problems that folks are apparently having ... It will be easier to maintain the mirror if the .mcz package names and filetree package names are the same ... Dale _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Let me look into renaming at least the compatibility packages. I assume you're referring to these guidelines: https://github.com/SeasideSt/Seaside/wiki/Package-Naming _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Thanks Monty ... I use a slightly different pattern myself, but the
Seaside guidelines are definitely portable. Using these guidelines you would rename XML-Parser-GemstoneCompatibility to XML-Gemstone-Parser and that would eliminate the conflict with XML-Parser Dale On 01/26/2018 07:37 PM, monty via Glass wrote: > Let me look into renaming at least the compatibility packages. > > I assume you're referring to these guidelines: https://github.com/SeasideSt/Seaside/wiki/Package-Naming > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
I renamed enough packages to eliminate Monticello conflicts on GS:
XML-Parser-GemstoneCompatibility -> XML-ParserGemstoneCompatibility XML-Parser-GTExtensions -> XML-ParserGTExtensions XML-Parser-HTML-Core -> XML-ParserHTML-Core XML-Parser-HTML-Tests -> XML-ParserHTML-Tests XML-Parser-HTML-GemstoneCompatibility -> XML-ParserHTML-GemstoneCompatibility XML-Parser-StAX-Core -> XML-ParserStAX-Core XML-Parser-StAX-Tests -> XML-ParserStAX-Tests Collections-OrderPreservingDictionary-GemstoneCompatibility -> Collections-OrderPreservingDictionaryGemstoneCompatibility It's still an ugly mix of SS and Pharo "XXX-Core" + "XXX-Tests" names, but I wanted to change them as little as possible while maximizing preserved package history. I also did some Config cleaning, and #validate is a really great feature. > Sent: Friday, January 26, 2018 at 10:45 PM > From: "Dale Henrichs via Glass" <[hidden email]> > To: [hidden email] > Subject: Re: [Glass] Another metacello loading puzzle: xml > > Thanks Monty ... I use a slightly different pattern myself, but the > Seaside guidelines are definitely portable. Using these guidelines you > would rename XML-Parser-GemstoneCompatibility to XML-Gemstone-Parser and > that would eliminate the conflict with XML-Parser > > Dale > > > On 01/26/2018 07:37 PM, monty via Glass wrote: > > Let me look into renaming at least the compatibility packages. > > > > I assume you're referring to these guidelines: https://github.com/SeasideSt/Seaside/wiki/Package-Naming > > _______________________________________________ > > Glass mailing list > > [hidden email] > > http://lists.gemtalksystems.com/mailman/listinfo/glass > > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass > Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
On 01/29/2018 01:23 AM, monty via Glass wrote: > I renamed enough packages to eliminate Monticello conflicts on GS: > XML-Parser-GemstoneCompatibility -> XML-ParserGemstoneCompatibility > XML-Parser-GTExtensions -> XML-ParserGTExtensions > XML-Parser-HTML-Core -> XML-ParserHTML-Core > XML-Parser-HTML-Tests -> XML-ParserHTML-Tests > XML-Parser-HTML-GemstoneCompatibility -> XML-ParserHTML-GemstoneCompatibility > XML-Parser-StAX-Core -> XML-ParserStAX-Core > XML-Parser-StAX-Tests -> XML-ParserStAX-Tests > Collections-OrderPreservingDictionary-GemstoneCompatibility -> Collections-OrderPreservingDictionaryGemstoneCompatibility > > It's still an ugly mix of SS and Pharo "XXX-Core" + "XXX-Tests" names, but I wanted to change them as little as possible while maximizing preserved package history. > I also did some Config cleaning, and #validate is a really great feature. Ah, I'm glad that you used it ... I found it to be a required feature early on when I was trying to help folks clean up their configurations:) Thanks for the effort, I will make time to create the mirror in the next week or so. Thanks again! Dale > >> Sent: Friday, January 26, 2018 at 10:45 PM >> From: "Dale Henrichs via Glass" <[hidden email]> >> To: [hidden email] >> Subject: Re: [Glass] Another metacello loading puzzle: xml >> >> Thanks Monty ... I use a slightly different pattern myself, but the >> Seaside guidelines are definitely portable. Using these guidelines you >> would rename XML-Parser-GemstoneCompatibility to XML-Gemstone-Parser and >> that would eliminate the conflict with XML-Parser >> >> Dale >> >> >> On 01/26/2018 07:37 PM, monty via Glass wrote: >>> Let me look into renaming at least the compatibility packages. >>> >>> I assume you're referring to these guidelines: https://github.com/SeasideSt/Seaside/wiki/Package-Naming >>> _______________________________________________ >>> Glass mailing list >>> [hidden email] >>> http://lists.gemtalksystems.com/mailman/listinfo/glass >> _______________________________________________ >> Glass mailing list >> [hidden email] >> http://lists.gemtalksystems.com/mailman/listinfo/glass >> > _______________________________________________ > Glass mailing list > [hidden email] > http://lists.gemtalksystems.com/mailman/listinfo/glass _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
In reply to this post by GLASS mailing list
Hi Dale,
On 26/01/2018 15:58, Dale Henrichs via Glass wrote: >> I am ignoring pharo for the moment so that I can first sort out issues >> in our own code (this also upgrades seaside and magritte and the XML >> projects for us)..... and will then bug the Pharo-users list when I am >> ready to tackle that issue. > It is likely to be a Metacello/Monticello issue in pharo as well --- the > actual error and stack would be critical in determining this --- and if > it is I am probably the best person to handle it for Pharo as well :) I am now turning to Pharo... and I managed to get Pharo 3.0 and 6.1 to load what I can load in Gemstone3.4 with/without a network connection, BUT... I now have issues loading our own project. Our own project is loaded via a BaselineOf which only contains internal packages on the local filetree repo. It breaks with a 'Apparent loop in before/after dependency definitions' while trying to do MetacelloMCVersionSpec(MetacelloVersionSpec)>>packageSpecsInLoadOrder I have put the (very short transcript) and stack from Pharo6.1 on pastebin [1]. The MetacelloPackagesSpec asString is also useful - but it contains information I'd rather keep confidential, so I will email that off-list. (I have also pushed the dependencies as per the output of that MetacelloPackagesSpec asString through a topological sort and found no circularity.) Can you shed any light? [1] https://pastebin.com/hUsQ4eSK Thanks Iwan -- _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Re: Another metacello loading puzzle: xml
|
|
Iwan,
The first thing I would ask you to do is to validate your baseline with the following: MetacelloToolBox validateBaseline: BaselineOf??? given that you have a circularity in your dependencies, the validate may not complete either, but if you do get some issues reported fixing those issues might clear up the circularity. It's been awhile since I've had to debug a circularity and I don't remember the drill of the top of my head ... so I will need to spend a little time refreshing my memory as to the best way to solve the problem ... I don't think that I have an explicit validation for circular dependencies and this may be an opportunity to add one ... Dale On 01/31/2018 07:05 AM, Iwan Vosloo via Glass wrote: > Hi Dale, > > On 26/01/2018 15:58, Dale Henrichs via Glass wrote: >>> I am ignoring pharo for the moment so that I can first sort out >>> issues in our own code (this also upgrades seaside and magritte and >>> the XML projects for us)..... and will then bug the Pharo-users list >>> when I am ready to tackle that issue. >> It is likely to be a Metacello/Monticello issue in pharo as well --- >> the actual error and stack would be critical in determining this --- >> and if it is I am probably the best person to handle it for Pharo as >> well :) > > I am now turning to Pharo... and I managed to get Pharo 3.0 and 6.1 to > load what I can load in Gemstone3.4 with/without a network connection, > BUT... I now have issues loading our own project. Our own project is > loaded via a BaselineOf which only contains internal packages on the > local filetree repo. > > It breaks with a 'Apparent loop in before/after dependency > definitions' while trying to do > MetacelloMCVersionSpec(MetacelloVersionSpec)>>packageSpecsInLoadOrder > > I have put the (very short transcript) and stack from Pharo6.1 on > pastebin [1]. The MetacelloPackagesSpec asString is also useful - but > it contains information I'd rather keep confidential, so I will email > that off-list. (I have also pushed the dependencies as per the output > of that MetacelloPackagesSpec asString through a topological sort and > found no circularity.) > > Can you shed any light? > > [1] https://pastebin.com/hUsQ4eSK > > > Thanks > Iwan > > _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
«
Return to GLASS
|
1 view|%1 views
| Free forum by Nabble | Edit this page |