Another metacello loading puzzle: xml

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Another metacello loading puzzle: xml

GLASS mailing list
Hi all,

I am trying to load XMLParser (from
http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main) into a new
gemstone 3.4 image as a dependency of another BaselineOf.

My reproducing baseline contains only that single dependency:

baseline: spec
   <baseline>
   spec
     for: #common do: [
         spec
             blessing: #baseline;
             description: 'Example project';
             configuration: 'XMLParser' with: [
               spec
repository: 'http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main';
                  versionString: '3.2.8'.
            ];
             group: 'Core' with: #('XMLParser');
             group: 'default' with: #('Core')
     ].


.. which I try to load with:

Metacello new
   baseline: 'Example';
   repository: 'filetree:///tmp/examplerepository';
   cacheRepository: '/tmp/github-cache';
   load


This seems to load fine, but breaks when it tries to run (as post load
directive) ConfigurationOfXMLParser>>initializeClassesForGSBaseline321
because the classes referred to in that method seem not to be loaded at
this point. (The actual exception is "Set doesNotUnderstand
#entityCache" - perhaps I understand that incorrectly...)



If I just load it myself, it works:

(Metacello new
   configuration: 'XMLParser';
   repository: 'http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main';
   version: '3.2.8';
   load)


Any ideas? How can I debug the issue?

Regards
- Iwan



--

Reahl, the Python only web framework / www.reahl.org
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
On 19/12/2017 16:36, Iwan Vosloo via Glass wrote:
> I am trying to load XMLParser (from
> http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main) into a new
> gemstone 3.4 image as a dependency of another BaselineOf.
>
[deleted]
>
> .. which I try to load with:
>
> Metacello new
>    baseline: 'Example';
>    repository: 'filetree:///tmp/examplerepository';
>    cacheRepository: '/tmp/github-cache';
>    load

After some more experiments I found that I can load it successfully, but
only if the cacheRepository is empty/is not specified.


Regards
- Iwan

--

Reahl, the Python only web framework / www.reahl.org
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
In reply to this post by GLASS mailing list
Iwan,

For some reason google put this email into the spam folder and I just
saw it ... I'll try to take a look at this later this afternoon ...

Dale

On 12/19/17 6:36 AM, Iwan Vosloo via Glass wrote:

> Hi all,
>
> I am trying to load XMLParser (from
> http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main) into a new
> gemstone 3.4 image as a dependency of another BaselineOf.
>
> My reproducing baseline contains only that single dependency:
>
> baseline: spec
>   <baseline>
>   spec
>     for: #common do: [
>         spec
>             blessing: #baseline;
>             description: 'Example project';
>             configuration: 'XMLParser' with: [
>            spec
> repository: 'http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main';
>                  versionString: '3.2.8'.
>         ];
>             group: 'Core' with: #('XMLParser');
>             group: 'default' with: #('Core')
>     ].
>
>
> .. which I try to load with:
>
> Metacello new
>   baseline: 'Example';
>   repository: 'filetree:///tmp/examplerepository';
>   cacheRepository: '/tmp/github-cache';
>   load
>
>
> This seems to load fine, but breaks when it tries to run (as post load
> directive) ConfigurationOfXMLParser>>initializeClassesForGSBaseline321
> because the classes referred to in that method seem not to be loaded
> at this point. (The actual exception is "Set doesNotUnderstand
> #entityCache" - perhaps I understand that incorrectly...)
>
>
>
> If I just load it myself, it works:
>
> (Metacello new
>   configuration: 'XMLParser';
>   repository:
> 'http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main';
>   version: '3.2.8';
>   load)
>
>
> Any ideas? How can I debug the issue?
>
> Regards
> - Iwan
>
>
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
In reply to this post by GLASS mailing list
What are you using the cacheRepository: for? It's not something that you
should use except when you are trying to explicitly load mcz files from
disk instead of the http:// repos...

Dale


On 12/20/17 1:53 AM, Iwan Vosloo via Glass wrote:

> On 19/12/2017 16:36, Iwan Vosloo via Glass wrote:
>> I am trying to load XMLParser (from
>> http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main) into a new
>> gemstone 3.4 image as a dependency of another BaselineOf.
>>
> [deleted]
>>
>> .. which I try to load with:
>>
>> Metacello new
>>    baseline: 'Example';
>>    repository: 'filetree:///tmp/examplerepository';
>>    cacheRepository: '/tmp/github-cache';
>>    load
>
> After some more experiments I found that I can load it successfully,
> but only if the cacheRepository is empty/is not specified.
>
>
> Regards
> - Iwan
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Hi Dale,

Thanks for checking...

On 22/12/2017 20:44, Dale Henrichs via Glass wrote:
> What are you using the cacheRepository: for? It's not something that you
> should use except when you are trying to explicitly load mcz files from
> disk instead of the http:// repos...

We fetch what we need once, and keep it thereafter locally (in the cache
repository) so that we don't need to keep hitting the network during CI
builds etc.

Regards
Iwan


--

Reahl, the Python only web framework / www.reahl.org
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list


On 12/22/17 12:38 PM, Iwan Vosloo via Glass wrote:

> Hi Dale,
>
> Thanks for checking...
>
> On 22/12/2017 20:44, Dale Henrichs via Glass wrote:
>> What are you using the cacheRepository: for? It's not something that
>> you should use except when you are trying to explicitly load mcz
>> files from disk instead of the http:// repos...
>
> We fetch what we need once, and keep it thereafter locally (in the
> cache repository) so that we don't need to keep hitting the network
> during CI builds etc.
okay ... perhaps the fetch fails when attempting to download the xml
files? There shouldn't be anything special about he xml project to make
it fail, unless you haven't pre-fetched the XML mcz files ///

Dale
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Hi Dale,

On 23/12/2017 00:11, Dale Henrichs via Glass wrote:
>> We fetch what we need once, and keep it thereafter locally (in the
>> cache repository) so that we don't need to keep hitting the network
>> during CI builds etc.
 >
> okay ... perhaps the fetch fails when attempting to download the xml
> files? There shouldn't be anything special about he xml project to make
> it fail, unless you haven't pre-fetched the XML mcz files ///

Do you mean I should always first do :

Metacello new
   ....
   cacheDirectory: '...';
   fetch: 'ALL'.

And THEN only a load: 'ALL'?

I was under the impression doing just a load would fetch, thus populate
the cache and then load; and on subsequent loads of the same thing would
hit the cache. And indeed if I clean the cache out, and then just do a
load I see that happening.

Thanks
Iwan




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Well I think that if you want to avoid hitting the network all of the
time, then you do a separate fetch into the cacheRepository[1] and then
when you are doing a load, you use repositoryOverrides[2]. The links are
broken in the urls below but the code is and explanations are still
valid ...

Dale


[1]
https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_create_a_local_cache_repository_for_a_config
[2]
https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_override_the_repository_for_a_config
On 12/23/17 2:12 AM, Iwan Vosloo via Glass wrote:

> Hi Dale,
>
> On 23/12/2017 00:11, Dale Henrichs via Glass wrote:
>>> We fetch what we need once, and keep it thereafter locally (in the
>>> cache repository) so that we don't need to keep hitting the network
>>> during CI builds etc.
> >
>> okay ... perhaps the fetch fails when attempting to download the xml
>> files? There shouldn't be anything special about he xml project to
>> make it fail, unless you haven't pre-fetched the XML mcz files ///
>
> Do you mean I should always first do :
>
> Metacello new
>   ....
>   cacheDirectory: '...';
>   fetch: 'ALL'.
>
> And THEN only a load: 'ALL'?
>
> I was under the impression doing just a load would fetch, thus
> populate the cache and then load; and on subsequent loads of the same
> thing would hit the cache. And indeed if I clean the cache out, and
> then just do a load I see that happening.
>
> Thanks
> Iwan
>
>
>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
On 26/12/2017 19:38, Dale Henrichs via Glass wrote:

> Well I think that if you want to avoid hitting the network all of the
> time, then you do a separate fetch into the cacheRepository[1] and then
> when you are doing a load, you use repositoryOverrides[2]. The links are
> broken in the urls below but the code is and explanations are still
> valid ...
>
> Dale
>
>
> [1]
> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_create_a_local_cache_repository_for_a_config 
>
> [2]
> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_override_the_repository_for_a_config 


Thanks Dale,

I have to use the Metacello scripting API though... and according to
its docs[1] that means to simply fetch (with cacheRepository set) and
thereafter a load (also with cacheRepository set).

I have been trying to get just this bit to work as per the docs... Are
those docs correct? Should this work via the scripting API?

I figured that repositoryOverrides will just help when it comes to the
ConfigurationOf/BaselineOf classes themselves. And I dont see anything
about repositoryOverrides in those scripting API docs.

I also struggle to understand how docs of the one API apply to the
scripting API etc


[1]
https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching

Regards
Iwan

--
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
In reply to this post by GLASS mailing list
On 22/12/2017 20:42, Dale Henrichs via Glass wrote:
> On 12/19/17 6:36 AM, Iwan Vosloo via Glass wrote:
>> I am trying to load XMLParser (from
>> http://www.smalltalkhub.com/mc/PharoExtras/XMLParser/main) into a new
>> gemstone 3.4 image as a dependency of another BaselineOf.

[snip]

>> .. which I try to load with:
>>
>> Metacello new
>>   baseline: 'Example';
>>   repository: 'filetree:///tmp/examplerepository';
>>   cacheRepository: '/tmp/github-cache';
>>   load
>>
>>
>> This seems to load fine, but breaks when it tries to run (as post load
>> directive) ConfigurationOfXMLParser>>initializeClassesForGSBaseline321
>> because the classes referred to in that method seem not to be loaded
>> at this point. (The actual exception is "Set doesNotUnderstand
>> #entityCache" - perhaps I understand that incorrectly...)
I got this to work if I specifically use 'filetree:///tmp/github-cache'
instead of just '/tmp/github-cache' (the latter being what our older
code currently does)

Regards
Iwan

--


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
On 27/12/2017 08:37, Iwan Vosloo via Glass wrote:
 >
> I got this to work if I specifically use 'filetree:///tmp/github-cache'
> instead of just '/tmp/github-cache' (the latter being what our older
> code currently does)

Oh, no, sorry... I spoke too soon. I just hit a different unrelated snag
that masked the same issue.

Regards
Iwan


--
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
In reply to this post by GLASS mailing list


On 12/26/17 10:03 PM, Iwan Vosloo via Glass wrote:

> On 26/12/2017 19:38, Dale Henrichs via Glass wrote:
>> Well I think that if you want to avoid hitting the network all of the
>> time, then you do a separate fetch into the cacheRepository[1] and
>> then when you are doing a load, you use repositoryOverrides[2]. The
>> links are broken in the urls below but the code is and explanations
>> are still valid ...
>>
>> Dale
>>
>>
>> [1]
>> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_create_a_local_cache_repository_for_a_config 
>>
>> [2]
>> https://code.google.com/archive/p/metacello/wikis/FAQ.wiki#How_do_I_override_the_repository_for_a_config 
>
>
>
> Thanks Dale,
>
> I have to use the Metacello scripting API though... and according to
> its docs[1] that means to simply fetch (with cacheRepository set) and
> thereafter a load (also with cacheRepository set).
>
> I have been trying to get just this bit to work as per the docs... Are
> those docs correct? Should this work via the scripting API?
Not sure what you mean by "scripting API"?
>
> I figured that repositoryOverrides will just help when it comes to the
> ConfigurationOf/BaselineOf classes themselves. And I dont see anything
> about repositoryOverrides in those scripting API docs.
if you are using local clones of git repositories, there is no real
advantage to using cacheRepositories and repositoryOverrides ... they
were invented to avoid hitting the network for http-based mcz repositories.

The GLASS1 bootstrapping script and the GLASS1 baseline itself still
uses one or two http mcz repositories ... I am currently working on
completely eliminating the use of http mcz repositories for the
bootstrapping scripts and tODE ...
>
> I also struggle to understand how docs of the one API apply to the
> scripting API etc
Note that the examples in[1] are using configurations and configurations
are using http-based mcz repositories ... These days it makes more sense
to create a github repository and use local clones for your projects
than to fiddle with cacheRepsitories and repositoryOverrides ...

Perhaps you could be more specific about exactly what you are trying to
achieve
>
>
> [1]
> https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching
>
Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Hi Dale,


On 27/12/2017 19:13, Dale Henrichs via Glass wrote:
> Not sure what you mean by "scripting API"?

I mean as per the heading of [1]. I thought that referred to working
with via a Metacello instance as explained on that page instead of via
projects or versions of a configuration.

 > Perhaps you could be more specific about exactly what you are trying
 > to achieve

Thats complicated (by details). But, I have inherited the job of
upgrading our systems. They, and the scripts that manage them are
dependent on Metacello for loading things and for expressing
dependencies between things....and have evolved over time which means I
need to understand Metacello and our intricate context and make things
work again on Gs3.4. (Expect lots more strange questions.)

What the home-grown scripts do is basically: build a CI extent, starting
from the seaside extent, using pretty much only a local BaselineOf that
pulls a bunch of 3rd party projects in. Some of these are
ConfigurationOfs with mcz files, some are just mcz projects with no
Metacello, some are cloned repositories etc. There are a fair number of
packages being pulled in, and we won't live in a git-repo-only world any
time soon.

The same scripts also upgrade production systems.

The CI extents can be built using an option that fetches things via the
network, but once that is done they typically just use what we fetched
before. Also, the CI builds run continuously so we dont want them to hit
the network all the time.

It is handy to be able to say "go fetch all you need and save it on
disk"... it is more cumbersome to clone several git repos... you have to
know which ones, which branches and you have to distribute these to the
build machines too. (We use git submodules for that.) If I can just say
Metacello fetch, and save the resultant cache repository I dont need to
even know which dependencies are all pulled in.

We also depend on exact versions of things and use that info to make
decisions when there is a conflict.

I am still trying to figure Metacello out, but what would be nice is if
I can, in my BaselineOf point the various bits to github repos, and
still fetch and save the fetched result in a cacheRepository inside my
own project's git repository. Then I can handle all the different kinds
of packages the same way.


[1]
https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md


Thanks
Iwan

--
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Okay ...

Now I understand what you are trying to do, so now I would like to
understand the exact problems you are having ...

So I would like to start over ... I still believe that you should be
doing something like the following:

   | version repo |
   repo := MCDirectoryRepository new.
   repo directory: FileDirectory on: '<path to directory>'.
   Metacello new
     configuration: 'XXX';
     version: '1.0';
     cacheRepository: repo;
     fetch.
   Metacello new
     configuration: 'XXX';
     version: '1.0';
     repositoryOverrides: (OrderedCollection with: repo);
     load

But it sounds like you don't have a single master configuration or
baseline to do your loads ... right? if this is the case, then the
problem is just slightly more complicated ...

Sooooo, please provide me with the set of Metacello load scripts that
you use to load your application without involving any caching at all
... and the Transcript output for the full load. This will allow me to
understand when and where the http repositories get hit as well as
whether or not github repos with zip file downloads are involved or not
(my suspicion is that things might not work as expected when gthub
downloads are involved --- a use case for which I have not tests) ...
rather than guess, I'd rather know ...

Once I've seen your load script and the load output, I should be able to
figure out what needs to be done to get you moving in the right
direction ...

I know that there are other folks out there that use variations of the
cacheRepository/repositoryOverrides scripts for their production needs
and it would be helpful if they were to chime in with examples of the
scripts that they use as well

Dale

On 12/27/17 10:43 AM, Iwan Vosloo via Glass wrote:

> Hi Dale,
>
>
> On 27/12/2017 19:13, Dale Henrichs via Glass wrote:
>> Not sure what you mean by "scripting API"?
>
> I mean as per the heading of [1]. I thought that referred to working
> with via a Metacello instance as explained on that page instead of via
> projects or versions of a configuration.
>
> > Perhaps you could be more specific about exactly what you are trying
> > to achieve
>
> Thats complicated (by details). But, I have inherited the job of
> upgrading our systems. They, and the scripts that manage them are
> dependent on Metacello for loading things and for expressing
> dependencies between things....and have evolved over time which means
> I need to understand Metacello and our intricate context and make
> things work again on Gs3.4. (Expect lots more strange questions.)
>
> What the home-grown scripts do is basically: build a CI extent,
> starting from the seaside extent, using pretty much only a local
> BaselineOf that pulls a bunch of 3rd party projects in. Some of these
> are ConfigurationOfs with mcz files, some are just mcz projects with
> no Metacello, some are cloned repositories etc. There are a fair
> number of packages being pulled in, and we won't live in a
> git-repo-only world any time soon.
>
> The same scripts also upgrade production systems.
>
> The CI extents can be built using an option that fetches things via
> the network, but once that is done they typically just use what we
> fetched before. Also, the CI builds run continuously so we dont want
> them to hit the network all the time.
>
> It is handy to be able to say "go fetch all you need and save it on
> disk"... it is more cumbersome to clone several git repos... you have
> to know which ones, which branches and you have to distribute these to
> the build machines too. (We use git submodules for that.) If I can
> just say Metacello fetch, and save the resultant cache repository I
> dont need to even know which dependencies are all pulled in.
>
> We also depend on exact versions of things and use that info to make
> decisions when there is a conflict.
>
> I am still trying to figure Metacello out, but what would be nice is
> if I can, in my BaselineOf point the various bits to github repos, and
> still fetch and save the fetched result in a cacheRepository inside my
> own project's git repository. Then I can handle all the different
> kinds of packages the same way.
>
>
> [1]
> https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md
>
>
> Thanks
> Iwan
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Iwan,

If you don't want to publicly share you load scripts and transcrips,
then we can take this conversation to private email ... and post a
summary conclusion when we're done for the benefit of others who may be
facing similar problems ...

Dale


On 12/27/17 11:55 AM, Dale Henrichs wrote:

> Okay ...
>
> Now I understand what you are trying to do, so now I would like to
> understand the exact problems you are having ...
>
> So I would like to start over ... I still believe that you should be
> doing something like the following:
>
>   | version repo |
>   repo := MCDirectoryRepository new.
>   repo directory: FileDirectory on: '<path to directory>'.
>   Metacello new
>     configuration: 'XXX';
>     version: '1.0';
>     cacheRepository: repo;
>     fetch.
>   Metacello new
>     configuration: 'XXX';
>     version: '1.0';
>     repositoryOverrides: (OrderedCollection with: repo);
>     load
>
> But it sounds like you don't have a single master configuration or
> baseline to do your loads ... right? if this is the case, then the
> problem is just slightly more complicated ...
>
> Sooooo, please provide me with the set of Metacello load scripts that
> you use to load your application without involving any caching at all
> ... and the Transcript output for the full load. This will allow me to
> understand when and where the http repositories get hit as well as
> whether or not github repos with zip file downloads are involved or
> not (my suspicion is that things might not work as expected when gthub
> downloads are involved --- a use case for which I have not tests) ...
> rather than guess, I'd rather know ...
>
> Once I've seen your load script and the load output, I should be able
> to figure out what needs to be done to get you moving in the right
> direction ...
>
> I know that there are other folks out there that use variations of the
> cacheRepository/repositoryOverrides scripts for their production needs
> and it would be helpful if they were to chime in with examples of the
> scripts that they use as well
>
> Dale
>
> On 12/27/17 10:43 AM, Iwan Vosloo via Glass wrote:
>> Hi Dale,
>>
>>
>> On 27/12/2017 19:13, Dale Henrichs via Glass wrote:
>>> Not sure what you mean by "scripting API"?
>>
>> I mean as per the heading of [1]. I thought that referred to working
>> with via a Metacello instance as explained on that page instead of
>> via projects or versions of a configuration.
>>
>> > Perhaps you could be more specific about exactly what you are trying
>> > to achieve
>>
>> Thats complicated (by details). But, I have inherited the job of
>> upgrading our systems. They, and the scripts that manage them are
>> dependent on Metacello for loading things and for expressing
>> dependencies between things....and have evolved over time which means
>> I need to understand Metacello and our intricate context and make
>> things work again on Gs3.4. (Expect lots more strange questions.)
>>
>> What the home-grown scripts do is basically: build a CI extent,
>> starting from the seaside extent, using pretty much only a local
>> BaselineOf that pulls a bunch of 3rd party projects in. Some of these
>> are ConfigurationOfs with mcz files, some are just mcz projects with
>> no Metacello, some are cloned repositories etc. There are a fair
>> number of packages being pulled in, and we won't live in a
>> git-repo-only world any time soon.
>>
>> The same scripts also upgrade production systems.
>>
>> The CI extents can be built using an option that fetches things via
>> the network, but once that is done they typically just use what we
>> fetched before. Also, the CI builds run continuously so we dont want
>> them to hit the network all the time.
>>
>> It is handy to be able to say "go fetch all you need and save it on
>> disk"... it is more cumbersome to clone several git repos... you have
>> to know which ones, which branches and you have to distribute these
>> to the build machines too. (We use git submodules for that.) If I can
>> just say Metacello fetch, and save the resultant cache repository I
>> dont need to even know which dependencies are all pulled in.
>>
>> We also depend on exact versions of things and use that info to make
>> decisions when there is a conflict.
>>
>> I am still trying to figure Metacello out, but what would be nice is
>> if I can, in my BaselineOf point the various bits to github repos,
>> and still fetch and save the fetched result in a cacheRepository
>> inside my own project's git repository. Then I can handle all the
>> different kinds of packages the same way.
>>
>>
>> [1]
>> https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md
>>
>>
>> Thanks
>> Iwan
>>
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
In reply to this post by GLASS mailing list
Hi Dale,

On 27/12/2017 21:55, Dale Henrichs via Glass wrote:
> Sooooo, please provide me with the set of Metacello load scripts that
> you use to load your application without involving any caching at all

I am trying to pull out enough of the scripts to give you a working
example (or a non-working one at this stage...). I will put it in a
github repo and post here.

I have to run now, and could not quite finish it, but I hope to do so
tomorrow.

Thanks
Iwan


--
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
In reply to this post by GLASS mailing list
Hi Dale,

Sorry, the new year crept up on me (skeleton staff need to multitask).

Hope you have a good one!

On 27/12/2017 21:55, Dale Henrichs via Glass wrote:
 >
 > Sooooo, please provide me with the set of Metacello load scripts that
 > you use to load your application without involving any caching at all
 > ... and the Transcript output for the full load. This will allow me to
 > understand when and where the http repositories get hit as well as
 > whether or not github repos with zip file downloads are involved or not
 > (my suspicion is that things might not work as expected when gthub
 > downloads are involved --- a use case for which I have not tests) ...
 > rather than guess, I'd rather know ...

Extracting all our scripts is very tricky because they rely on a
particular setup, managed by ruby and chef.

I have extracted the BaselineOfWonkaRuntime package though[1], which
contains all the relevant smalltalk bits, and I added a bash shell
script that invokes that smalltalk code via topaz just like our ruby
scripts do. It also contains a copy of the transcript it currently
yields [2].

What is missing is other stuff our scripts do before invoking the
smalltalk, like fixing up monticello packages so they dont have
non-ascii chars in their metadata.

I have been trying to just play with this outside of our scripts because
I'd want to remove most of the code there and basically just rely on
Metacello and GsUpgrader to get the same job done.

Ideally, I'd love to get something like the snippet below to work:

On 27/12/2017 21:55, Dale Henrichs via Glass wrote:
 > So I would like to start over ... I still believe that you should be
 > doing something like the following:
 >
 >    | version repo |
 >    repo := MCDirectoryRepository new.
 >    repo directory: FileDirectory on: '<path to directory>'.
 >    Metacello new
 >      configuration: 'XXX';
 >      version: '1.0';
 >      cacheRepository: repo;
 >      fetch.
 >    Metacello new
 >      configuration: 'XXX';
 >      version: '1.0';
 >      repositoryOverrides: (OrderedCollection with: repo);
 >      load
 >
 > But it sounds like you don't have a single master configuration or
 > baseline to do your loads ... right?

No, we do have a single baseline (BaselineOfWonkaRuntime).

Your code snippet above confuses me somewhat though... because according
to the docs [3]:

a) I should be able to just pass a string to cacheRepository:
b) I should specify cacheRepository: to the second call as well

Although I have not played with repositoryOverrides: outside of our
scripts, it looked like that is what is needed to truly force it to read
the local repo when it loads the BaselineOf/ConfigurationOfs themselves.
But, I limited my experiments to just get what is in [3] to work as a
first step.


I feel bad about spamming the list so perhaps chat in the git repo? The
git repository is public.


[1] https://github.com/finworks/BaselineOfWonkaRuntime

[2]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.2018-01-01.log

[3]
https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching

Regards
Iwan



--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Iwan,

Sorry I got involved in some internal development work that needed to
get done pronto and now I've got a bit of breathing room:)

Thanks for putting all of the information up on github.

Now I need to get my bearings a bit ... the transcript that you put up
on github fails with:

   DataStream does not understand #'writeWordLike:'

and it looks like you are running with a pure fetch load and you are
doing of fetch of the not only GLASS/GsDevKit, but your entire application.

a) I would like to see the stack from #'writeWordLike:'. This method is
not in the current GemStone code base, and may never have been.

b) it seems that you are in the process of fetching Seaside at this
point ...

My first thought is that the way you want to structure your load is to
first get the basic GLASS classes bootstrapped into the image and then
load the rest of your application.  So that you would have two load
phases and two cache repositories. the first load phase bootstraps
GLASS/GsDevKit.

My second thought is that I've recently been working on getting
GsDevKit/GLASS to be bootstrapped using only git repositories (i.e., no
network) and this script[2] (within the context of GsDevKit_home) does
just that. It should be straightforward to adapt the script to a
non-GsDevKit_home

So if you are struggling with getting the networkless load to work it
may make sense for you to use the bootstrapGsDevKit script ... and then
work on getting all of the other monticello-based projects that you are
using up on github, so that your entire application load can be done
without hitting the network ...

If you take this route then we can ignore the #'writeWordLike:' and also
ignore the cacheRepository:/repositoryOverrides issues.

I am headed in this direction anyway and would be willing to spend time
helping you get the remaining monticello-based projects moved to github...

As a side note. My current work is not on the master branch yet, because
the bootstrapGsDevKit is still a bit rough and I have yet to solve the
upgrade requiring method recompilation problem...but if you are coming
from a 3.3 or later upgrade will not be an immediate issue.

I highly recommend to the bootstrapGsDevKit approach, but if you don't
want to take that route, then you should try splitting your load into
two phases 1) fetch and load GLASS 2) fetch and load your application.

I'm sure you will have a few more questions and we might as well stay on
this list for the time being, since I think this information is
generally useful.

Dale

[1]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.2018-01-01.log#L996

[2]
https://github.com/GsDevKit/GsDevKit_home/blob/issue_200/bin/bootstrapGsDevKit

On 1/1/18 1:02 AM, Iwan Vosloo via Glass wrote:

> Hi Dale,
>
> Sorry, the new year crept up on me (skeleton staff need to multitask).
>
> Hope you have a good one!
>
> On 27/12/2017 21:55, Dale Henrichs via Glass wrote:
> >
> > Sooooo, please provide me with the set of Metacello load scripts that
> > you use to load your application without involving any caching at all
> > ... and the Transcript output for the full load. This will allow me to
> > understand when and where the http repositories get hit as well as
> > whether or not github repos with zip file downloads are involved or not
> > (my suspicion is that things might not work as expected when gthub
> > downloads are involved --- a use case for which I have not tests) ...
> > rather than guess, I'd rather know ...
>
> Extracting all our scripts is very tricky because they rely on a
> particular setup, managed by ruby and chef.
>
> I have extracted the BaselineOfWonkaRuntime package though[1], which
> contains all the relevant smalltalk bits, and I added a bash shell
> script that invokes that smalltalk code via topaz just like our ruby
> scripts do. It also contains a copy of the transcript it currently
> yields [2].
>
> What is missing is other stuff our scripts do before invoking the
> smalltalk, like fixing up monticello packages so they dont have
> non-ascii chars in their metadata.
>
> I have been trying to just play with this outside of our scripts
> because I'd want to remove most of the code there and basically just
> rely on Metacello and GsUpgrader to get the same job done.
>
> Ideally, I'd love to get something like the snippet below to work:
>
> On 27/12/2017 21:55, Dale Henrichs via Glass wrote:
> > So I would like to start over ... I still believe that you should be
> > doing something like the following:
> >
> >    | version repo |
> >    repo := MCDirectoryRepository new.
> >    repo directory: FileDirectory on: '<path to directory>'.
> >    Metacello new
> >      configuration: 'XXX';
> >      version: '1.0';
> >      cacheRepository: repo;
> >      fetch.
> >    Metacello new
> >      configuration: 'XXX';
> >      version: '1.0';
> >      repositoryOverrides: (OrderedCollection with: repo);
> >      load
> >
> > But it sounds like you don't have a single master configuration or
> > baseline to do your loads ... right?
>
> No, we do have a single baseline (BaselineOfWonkaRuntime).
>
> Your code snippet above confuses me somewhat though... because
> according to the docs [3]:
>
> a) I should be able to just pass a string to cacheRepository:
> b) I should specify cacheRepository: to the second call as well
>
> Although I have not played with repositoryOverrides: outside of our
> scripts, it looked like that is what is needed to truly force it to
> read the local repo when it loads the BaselineOf/ConfigurationOfs
> themselves. But, I limited my experiments to just get what is in [3]
> to work as a first step.
>
>
> I feel bad about spamming the list so perhaps chat in the git repo?
> The git repository is public.
>
>
> [1] https://github.com/finworks/BaselineOfWonkaRuntime
>
> [2]
> https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.2018-01-01.log
>
> [3]
> https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching
>
> Regards
> Iwan
>
>
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list

On Mon, Jan 8, 2018 at 11:24 AM, Dale Henrichs via Glass <[hidden email]> wrote:
Iwan,

Sorry I got involved in some internal development work that needed to get done pronto and now I've got a bit of breathing room:)

Thanks for putting all of the information up on github.

Now I need to get my bearings a bit ... the transcript that you put up on github fails with:

  DataStream does not understand #'writeWordLike:'


Just a small comment from the outside: that problem seems to be known: http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html
Maybe Iwan can check if he has instances of QuadByteString too... that might be the reason...

Cheers, 

--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list

Good catch Mariano.

Getting rid of the QuadByteStrings in the mcz file is one solution the other solution is to save the package in Filetree format, since Filetree uses utf8.

Dale


On 1/8/18 6:35 AM, Mariano Martinez Peck wrote:

On Mon, Jan 8, 2018 at 11:24 AM, Dale Henrichs via Glass <[hidden email]> wrote:
Iwan,

Sorry I got involved in some internal development work that needed to get done pronto and now I've got a bit of breathing room:)

Thanks for putting all of the information up on github.

Now I need to get my bearings a bit ... the transcript that you put up on github fails with:

  DataStream does not understand #'writeWordLike:'


Just a small comment from the outside: that problem seems to be known: http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html
Maybe Iwan can check if he has instances of QuadByteString too... that might be the reason...

Cheers, 

--


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
123