Caching and offline use with both Git and non-Git style projects

We're trying to cache everything needed to load Seaside 3.1/Magritte 3.1
on GemStone, so that we can run Metacello offline (not have it go over the
network, not have CI and deployments fail randomly due to network
glitches).

We can't use cacheRepository: anymore, since that writes MCZs and they
can't handle non-ASCII characters that occur in our dependencies.

Dale said we should cache by cloning projects' GitHub repositories
locally, but that only works for those projects which are using Git, and
I'm not yet clear on how to efficiently build up such a set of clones - is
there something more elegant than just reading the transcript after a
succesful load and cloning manually?

So I tried caching to a filetree repository, under Pharo where the
non-ASCII issue does not manifest, and that fails for a very interesting
reason.

If I look at the MCZs left behind in the cacheRepository after loading our
dependencies, is see there are quite a few packages for which there are
multiple versions cached: i.e. OSProcess-Base-dtl.27.mcz and
OSProcess-Base-dtl.45.mcz or XML-Parser-GuillermoPolito.16.mcz and
XML-Parser-NorbertHartl.141.mcz.

The problem is, using filetree format the various versions of packages
each share one directory, so only one of them can be kept on disk at a
time. This causes resolve errors when trying to loading from the cache
repository later.

And to deepen the mystery, there is absolutely no reference in the
Transcript to Metacello fetching or loading OSProcess-Base-dtl.27.mcz or
XML-Parser-GuillermoPolito.16.mcz. But it does.

So back to the original question: how to build up a cache projects for
offline use when some of the projects need to be git cloned and others
need to be cached as MCZs?

--
You received this message because you are subscribed to the Google Groups "Metacello" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

On 2/2/15 6:02 AM, Pieter Nagel wrote:
> We're trying to cache everything needed to load Seaside 3.1/Magritte 3.1
> on GemStone, so that we can run Metacello offline (not have it go over the
> network, not have CI and deployments fail randomly due to network
> glitches).
>
> We can't use cacheRepository: anymore, since that writes MCZs and they
> can't handle non-ASCII characters that occur in our dependencies.
Before we go too far, have you explored the possibility of created a new
version of the troublesome mcz file? There can't be that many mcz files
with non-ASCII characters in them....
>
> Dale said we should cache by cloning projects' GitHub repositories
> locally, but that only works for those projects which are using Git, and
> I'm not yet clear on how to efficiently build up such a set of clones - is
> there something more elegant than just reading the transcript after a
> succesful load and cloning manually?
I have code that I use in tODE that can be used to extract the loaded
version information or github and non-github repositores... In tODE I
have a menu item that clones the github repo based on the version that
you've loaded into the image. So it is indeed possible "efficiently
build up such a set of clones".

Git/github requires a different tool set from those used with mcz files
and I've been working on that tool set for several years ...

I've architected the  tODE toolset so that it is possible to write
scripts to trigger the same code triggered by all menu items, so the
tODE code base could be leveraged by you without having to run the tODE
GUI... If you wanted to go that route...
>
> So I tried caching to a filetree repository, under Pharo where the
> non-ASCII issue does not manifest, and that fails for a very interesting
> reason.
Yeah, FileTree repos are very different from mcz repos ... a FileTree
repos is only supposed to store one version of a package at one time ...
git  manages the package versions .... I think you have two options:

   1. repair the one or two mcz files that have non-ASCII characters in
       them and continue with your current scheme
   2. Create your own local git repos for those projects that are currently
       based on git and use Metacello locks.

1. should be the easiest to do and presumably requires the least amount
of work on your part.

2. introduces a fair amount of change on your part in both scripting and
work flow. I personally think that using git for local caching is
superior to downloading from mcz repositories, but that's mainly because
I hate surprises and prefer to explicitly control when I get new
packages and I like to do so under controlled conditions ...

But that's just my opinion. There is a lot to be said for minimizing
change as well ...

Dale

--
You received this message because you are subscribed to the Google Groups "Metacello" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

> Before we go too far, have you explored the possibility of created a new
version of the troublesome mcz file? There can't be that many mcz files
with non-ASCII characters in them....

This would work in the immediate term, but be a maintenance headache.

Since the problem is an offending character in the version history, every
future version of this MCZ is going to have that same offending character
in its version file.

So we would need to manually re-create a fixed MCZ every time we upgrade
to a Seaside with a new version of that package, or a dependency pulls it
in. Worse, this will be implicit knowledge not captured in the code: some
future load will fail, and someone will need to realise "oh, we need to
manually tweak that MCZ again".

From a maintenance perspective, the github cloning is less onerous in this
one specific regard.

> I have code that I use in tODE that can be used to extract the loaded
version information or github and non-github repositores... In tODE I
have a menu item that clones the github repo based on the version that
you've loaded into the image. So it is indeed possible "efficiently
build up such a set of clones".

All the more reason for us to get round to evaluate moving from GemTools
to tODE.




--
You received this message because you are subscribed to the Google Groups "Metacello" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

On 2/3/15 12:49 AM, Pieter Nagel wrote: Make sense. You can patch on the short term and buy time for moving to
the new model.
>
>> I have code that I use in tODE that can be used to extract the loaded
> version information or github and non-github repositores... In tODE I
> have a menu item that clones the github repo based on the version that
> you've loaded into the image. So it is indeed possible "efficiently
> build up such a set of clones".
>
> All the more reason for us to get round to evaluate moving from GemTools
> to tODE.
I'm in the final throes of some structural changes for tODE that I think
is critical for managing the "local git clone" model. I expect to be
done in the next few days (if nothing unforeseen pops up:). When I
finish I plan to make a couple of short screencasts illustrating the use
of git and tode ... so this is good timing.

Dale

--
You received this message because you are subscribed to the Google Groups "Metacello" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Pieter,

If you are going to start looking into tODE more seriously, you should
joing the tODE mailing list[1] and possibly the gitter chat room[2].

I use the tode mailing list for release announcements and answering tODE
questions ... You can use the GLASS list for tODE questions as well, but
I prefer to discuss tODE on the it's own list until tODE has had a bit
more usage and has stabilized a bit ... the upcoming release will be
making changes to the disk structure used by tODE .. I don't anticipate
that I will be making structural changes like this one in the future,
but you never know:)

Dale


[1] https://groups.google.com/forum/#!forum/tode_st
[2] https://gitter.im/dalehenrich/tode
On 02/03/2015 07:10 AM, Dale Henrichs wrote:
--
You received this message because you are subscribed to the Google Groups "Metacello" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.