Another metacello loading puzzle: xml

> Hi all,
>
> I am trying to load XMLParser (from
> http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main) into a new
> gemstone 3.4 image as a dependency of another BaselineOf.
>
> My reproducing baseline contains only that single dependency:
>
> baseline: spec
>   <baseline>
>   spec
>     for: #common do: [
>         spec
>             blessing: #baseline;
>             description: 'Example project';
>             configuration: 'XMLParser' with: [
>            spec
> repository: 'http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main';
>                  versionString: '3.2.8'.
>         ];
>             group: 'Core' with: #('XMLParser');
>             group: 'default' with: #('Core')
>     ].
>
>
> .. which I try to load with:
>
> Metacello new
>   baseline: 'Example';
>   repository: 'filetree:///tmp/examplerepository';
>   cacheRepository: '/tmp/github-cache';
>   load
>
>
> This seems to load fine, but breaks when it tries to run (as post load
> directive) ConfigurationOfXMLParser>>initializeClassesForGSBaseline321
> because the classes referred to in that method seem not to be loaded
> at this point. (The actual exception is "Set doesNotUnderstand
> #entityCache" - perhaps I understand that incorrectly...)
>
>
>
> If I just load it myself, it works:
>
> (Metacello new
>   configuration: 'XMLParser';
>   repository:
> 'http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main';
>   version: '3.2.8';
>   load)
>
>
> Any ideas? How can I debug the issue?
>
> Regards
> - Iwan
>
>
>

> On 19/12/2017 16:36, Iwan Vosloo via Glass wrote:
>> I am trying to load XMLParser (from
>> http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main) into a new
>> gemstone 3.4 image as a dependency of another BaselineOf.
>>
> [deleted]
>>
>> .. which I try to load with:
>>
>> Metacello new
>>    baseline: 'Example';
>>    repository: 'filetree:///tmp/examplerepository';
>>    cacheRepository: '/tmp/github-cache';
>>    load
>
> After some more experiments I found that I can load it successfully,
> but only if the cacheRepository is empty/is not specified.
>
>
> Regards
> - Iwan
>

> Hi Dale,
>
> Thanks for checking...
>
> On 22/12/2017 20:44, Dale Henrichs via Glass wrote:
>> What are you using the cacheRepository: for? It's not something that
>> you should use except when you are trying to explicitly load mcz
>> files from disk instead of the http:// repos...
>
> We fetch what we need once, and keep it thereafter locally (in the
> cache repository) so that we don't need to keep hitting the network
> during CI builds etc.

> Hi Dale,
>
> On 23/12/2017 00:11, Dale Henrichs via Glass wrote:
>>> We fetch what we need once, and keep it thereafter locally (in the
>>> cache repository) so that we don't need to keep hitting the network
>>> during CI builds etc.
> >
>> okay ... perhaps the fetch fails when attempting to download the xml
>> files? There shouldn't be anything special about he xml project to
>> make it fail, unless you haven't pre-fetched the XML mcz files ///
>
> Do you mean I should always first do :
>
> Metacello new
>   ....
>   cacheDirectory: '...';
>   fetch: 'ALL'.
>
> And THEN only a load: 'ALL'?
>
> I was under the impression doing just a load would fetch, thus
> populate the cache and then load; and on subsequent loads of the same
> thing would hit the cache. And indeed if I clean the cache out, and
> then just do a load I see that happening.
>
> Thanks
> Iwan
>
>
>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

> Well I think that if you want to avoid hitting the network all of the
> time, then you do a separate fetch into the cacheRepository[1] and then
> when you are doing a load, you use repositoryOverrides[2]. The links are
> broken in the urls below but the code is and explanations are still
> valid ...
>
> Dale
>
>
> [1]
> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_create_a_local_cache_repository_for_a_config 
>
> [2]
> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_override_the_repository_for_a_config 

>> .. which I try to load with:
>>
>> Metacello new
>>   baseline: 'Example';
>>   repository: 'filetree:///tmp/examplerepository';
>>   cacheRepository: '/tmp/github-cache';
>>   load
>>
>>
>> This seems to load fine, but breaks when it tries to run (as post load
>> directive) ConfigurationOfXMLParser>>initializeClassesForGSBaseline321
>> because the classes referred to in that method seem not to be loaded
>> at this point. (The actual exception is "Set doesNotUnderstand
>> #entityCache" - perhaps I understand that incorrectly...)

> On 26/12/2017 19:38, Dale Henrichs via Glass wrote:
>> Well I think that if you want to avoid hitting the network all of the
>> time, then you do a separate fetch into the cacheRepository[1] and
>> then when you are doing a load, you use repositoryOverrides[2]. The
>> links are broken in the urls below but the code is and explanations
>> are still valid ...
>>
>> Dale
>>
>>
>> [1]
>> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_create_a_local_cache_repository_for_a_config 
>>
>> [2]
>> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_override_the_repository_for_a_config 
>
>
>
> Thanks Dale,
>
> I have to use the Metacello scripting API though... and according to
> its docs[1] that means to simply fetch (with cacheRepository set) and
> thereafter a load (also with cacheRepository set).
>
> I have been trying to get just this bit to work as per the docs... Are
> those docs correct? Should this work via the scripting API?

> Hi Dale,
>
>
> On 27/12/2017 19:13, Dale Henrichs via Glass wrote:
>> Not sure what you mean by "scripting API"?
>
> I mean as per the heading of [1]. I thought that referred to working
> with via a Metacello instance as explained on that page instead of via
> projects or versions of a configuration.
>
> > Perhaps you could be more specific about exactly what you are trying
> > to achieve
>
> Thats complicated (by details). But, I have inherited the job of
> upgrading our systems. They, and the scripts that manage them are
> dependent on Metacello for loading things and for expressing
> dependencies between things....and have evolved over time which means
> I need to understand Metacello and our intricate context and make
> things work again on Gs3.4. (Expect lots more strange questions.)
>
> What the home-grown scripts do is basically: build a CI extent,
> starting from the seaside extent, using pretty much only a local
> BaselineOf that pulls a bunch of 3rd party projects in. Some of these
> are ConfigurationOfs with mcz files, some are just mcz projects with
> no Metacello, some are cloned repositories etc. There are a fair
> number of packages being pulled in, and we won't live in a
> git-repo-only world any time soon.
>
> The same scripts also upgrade production systems.
>
> The CI extents can be built using an option that fetches things via
> the network, but once that is done they typically just use what we
> fetched before. Also, the CI builds run continuously so we dont want
> them to hit the network all the time.
>
> It is handy to be able to say "go fetch all you need and save it on
> disk"... it is more cumbersome to clone several git repos... you have
> to know which ones, which branches and you have to distribute these to
> the build machines too. (We use git submodules for that.) If I can
> just say Metacello fetch, and save the resultant cache repository I
> dont need to even know which dependencies are all pulled in.
>
> We also depend on exact versions of things and use that info to make
> decisions when there is a conflict.
>
> I am still trying to figure Metacello out, but what would be nice is
> if I can, in my BaselineOf point the various bits to github repos, and
> still fetch and save the fetched result in a cacheRepository inside my
> own project's git repository. Then I can handle all the different
> kinds of packages the same way.
>
>
> [1]
> https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md
>
>
> Thanks
> Iwan
>

> Okay ...
>
> Now I understand what you are trying to do, so now I would like to
> understand the exact problems you are having ...
>
> So I would like to start over ... I still believe that you should be
> doing something like the following:
>
>   | version repo |
>   repo := MCDirectoryRepository new.
>   repo directory: FileDirectory on: '<path to directory>'.
>   Metacello new
>     configuration: 'XXX';
>     version: '1.0';
>     cacheRepository: repo;
>     fetch.
>   Metacello new
>     configuration: 'XXX';
>     version: '1.0';
>     repositoryOverrides: (OrderedCollection with: repo);
>     load
>
> But it sounds like you don't have a single master configuration or
> baseline to do your loads ... right? if this is the case, then the
> problem is just slightly more complicated ...
>
> Sooooo, please provide me with the set of Metacello load scripts that
> you use to load your application without involving any caching at all
> ... and the Transcript output for the full load. This will allow me to
> understand when and where the http repositories get hit as well as
> whether or not github repos with zip file downloads are involved or
> not (my suspicion is that things might not work as expected when gthub
> downloads are involved --- a use case for which I have not tests) ...
> rather than guess, I'd rather know ...
>
> Once I've seen your load script and the load output, I should be able
> to figure out what needs to be done to get you moving in the right
> direction ...
>
> I know that there are other folks out there that use variations of the
> cacheRepository/repositoryOverrides scripts for their production needs
> and it would be helpful if they were to chime in with examples of the
> scripts that they use as well
>
> Dale
>
> On 12/27/17 10:43 AM, Iwan Vosloo via Glass wrote:
>> Hi Dale,
>>
>>
>> On 27/12/2017 19:13, Dale Henrichs via Glass wrote:
>>> Not sure what you mean by "scripting API"?
>>
>> I mean as per the heading of [1]. I thought that referred to working
>> with via a Metacello instance as explained on that page instead of
>> via projects or versions of a configuration.
>>
>> > Perhaps you could be more specific about exactly what you are trying
>> > to achieve
>>
>> Thats complicated (by details). But, I have inherited the job of
>> upgrading our systems. They, and the scripts that manage them are
>> dependent on Metacello for loading things and for expressing
>> dependencies between things....and have evolved over time which means
>> I need to understand Metacello and our intricate context and make
>> things work again on Gs3.4. (Expect lots more strange questions.)
>>
>> What the home-grown scripts do is basically: build a CI extent,
>> starting from the seaside extent, using pretty much only a local
>> BaselineOf that pulls a bunch of 3rd party projects in. Some of these
>> are ConfigurationOfs with mcz files, some are just mcz projects with
>> no Metacello, some are cloned repositories etc. There are a fair
>> number of packages being pulled in, and we won't live in a
>> git-repo-only world any time soon.
>>
>> The same scripts also upgrade production systems.
>>
>> The CI extents can be built using an option that fetches things via
>> the network, but once that is done they typically just use what we
>> fetched before. Also, the CI builds run continuously so we dont want
>> them to hit the network all the time.
>>
>> It is handy to be able to say "go fetch all you need and save it on
>> disk"... it is more cumbersome to clone several git repos... you have
>> to know which ones, which branches and you have to distribute these
>> to the build machines too. (We use git submodules for that.) If I can
>> just say Metacello fetch, and save the resultant cache repository I
>> dont need to even know which dependencies are all pulled in.
>>
>> We also depend on exact versions of things and use that info to make
>> decisions when there is a conflict.
>>
>> I am still trying to figure Metacello out, but what would be nice is
>> if I can, in my BaselineOf point the various bits to github repos,
>> and still fetch and save the fetched result in a cacheRepository
>> inside my own project's git repository. Then I can handle all the
>> different kinds of packages the same way.
>>
>>
>> [1]
>> https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md
>>
>>
>> Thanks
>> Iwan
>>
>

> Hi Dale,
>
> Sorry, the new year crept up on me (skeleton staff need to multitask).
>
> Hope you have a good one!
>
> On 27/12/2017 21:55, Dale Henrichs via Glass wrote:
> >
> > Sooooo, please provide me with the set of Metacello load scripts that
> > you use to load your application without involving any caching at all
> > ... and the Transcript output for the full load. This will allow me to
> > understand when and where the http repositories get hit as well as
> > whether or not github repos with zip file downloads are involved or not
> > (my suspicion is that things might not work as expected when gthub
> > downloads are involved --- a use case for which I have not tests) ...
> > rather than guess, I'd rather know ...
>
> Extracting all our scripts is very tricky because they rely on a
> particular setup, managed by ruby and chef.
>
> I have extracted the BaselineOfWonkaRuntime package though[1], which
> contains all the relevant smalltalk bits, and I added a bash shell
> script that invokes that smalltalk code via topaz just like our ruby
> scripts do. It also contains a copy of the transcript it currently
> yields [2].
>
> What is missing is other stuff our scripts do before invoking the
> smalltalk, like fixing up monticello packages so they dont have
> non-ascii chars in their metadata.
>
> I have been trying to just play with this outside of our scripts
> because I'd want to remove most of the code there and basically just
> rely on Metacello and GsUpgrader to get the same job done.
>
> Ideally, I'd love to get something like the snippet below to work:
>
> On 27/12/2017 21:55, Dale Henrichs via Glass wrote:
> > So I would like to start over ... I still believe that you should be
> > doing something like the following:
> >
> >    | version repo |
> >    repo := MCDirectoryRepository new.
> >    repo directory: FileDirectory on: '<path to directory>'.
> >    Metacello new
> >      configuration: 'XXX';
> >      version: '1.0';
> >      cacheRepository: repo;
> >      fetch.
> >    Metacello new
> >      configuration: 'XXX';
> >      version: '1.0';
> >      repositoryOverrides: (OrderedCollection with: repo);
> >      load
> >
> > But it sounds like you don't have a single master configuration or
> > baseline to do your loads ... right?
>
> No, we do have a single baseline (BaselineOfWonkaRuntime).
>
> Your code snippet above confuses me somewhat though... because
> according to the docs [3]:
>
> a) I should be able to just pass a string to cacheRepository:
> b) I should specify cacheRepository: to the second call as well
>
> Although I have not played with repositoryOverrides: outside of our
> scripts, it looked like that is what is needed to truly force it to
> read the local repo when it loads the BaselineOf/ConfigurationOfs
> themselves. But, I limited my experiments to just get what is in [3]
> to work as a first step.
>
>
> I feel bad about spamming the list so perhaps chat in the git repo?
> The git repository is public.
>
>
> [1] https://github.com/finworks/BaselineOfWonkaRuntime
>
> [2]
> https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.2018-01-01.log
>
> [3]
> https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching
>
> Regards
> Iwan
>
>
>