Another metacello loading puzzle: xml

Next Topic
 
classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Hi guys,

On 1/8/18 6:35 AM, Mariano Martinez Peck wrote:
 > Just a small comment from the outside: that problem seems to be
 > known: http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html
 > Maybe Iwan can check if he has instances of QuadByteString too...
 > that might be the reason...

On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
> Getting rid of the QuadByteStrings in the mcz file is one solution the
 > other solution is to save the package in Filetree format, since
 > Filetree uses utf8.



Thanks Mariano and Dale,

Yes, I eventually discovered the QuadByteString issue... I tried using
filetree initially, but given the code that's there (that WonkaLoader
thing) it is not straightforward to use filetree (I suspect because it
uses FileTree to fetch FileTree as well, there may be a chicken-and-egg
scenario going on...but I did not investigate).

Instead I figured the QuadByteString issue must be a bug, since
writeWordLike: is being called from code present in Gemstone, yet it is
not implemented in Gemstone.

I provided an implementation (should I log a bug somewhere?), and using
that it actually loads using our scripts. Which is nice because it gives
me a starting point of sorts from where I can refactor and simplify.

Of course there's strong incentive to just leave it at that from the
people who have to pay the $$ for this... But I would have liked to be
able to load it all (via/not via network) using only what is provided by
GsUpgrader and Metacello without all the various things in our
WonkaLoader class....

It is painful to have to get all the right branches of all the right
projects from all the right github repositories locally just to be able
to load...being able to fetch the correct dependencies at their correct
versions, from wherever, recursively is exactly the point of using
metacello in the first place. Having to manage that ourselves defeats
the point of using Metacello in the first place... (Besides, many of our
dependencies are just Monticello packages anyways)


If you want to play along, my answers to Dale:


On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
 > I highly recommend to the bootstrapGsDevKit approach, but if you don't
 > want to take that route, then you should try splitting your load into
 > two phases 1) fetch and load GLASS 2) fetch and load your application.

That is actually exactly what our code does. When you call WonkaLoader
class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork: (see
the call in topaz/makeCIImage.topaz [1] ):
  - it first updates/loads GsUpgrader, FileTree, Metacello, Grease,
GLASS1 (in that order, from locally cloned git repositories); and
  - then it loads group 'CI' of BaselineOfWonkaRuntime from
wonkaRoot/monticello repository (this is where our own code sits).

But, all the loading that takes place - whether loading a "bootstrap
dependency" (the GLASS/Metacello stuff) or any other package to load -
is done in the same method: WonkaLoader>>loadGroup [2], using the same
code to create a Metacello [3] and [4].

In the first phase, (loading bootstrap dependencies), it sets up a new
Monticello independently for each of those "bootstrap" projects in turn,
on their own (cloned github) repository. ( See BaselineOfWOnkaRuntime
class >> loadBootstrapBaseline:repository:modes: [5] )

In the second phase, it uses the same underlying method to set up a new
Metacello on our own repository.

What is shared in all of the above is how conflicts etc are handled; and
the given repositoryOverrides are always ignored if you do it
overNetwork. Further, if you DO load overNetwork, it sets the
cacheRepository, and always to the same platform-dependent location
(which we check into our git repo).


What I understand our not-so-pretty code does in short (ignoring
handling of conflicts etc) is:

Metacello new
   baseline: xxx;
   repository: aLocalGitClone;
   cacheRepository: yyy;
   get.

Metacello new
   baseline: xxx;
   repository: aLocalGitClone;
   cacheRepository: yyy;
   load: 'default'.

For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that order)

Then it does:

Metacello new
   baseline: xxx;
   repository: ourOwnFileTreeRepo;
   cacheRepository: yyy;
   get.

Metacello new
   baseline: xxx;
   repository: ourOwnFileTreeRepo;
   cacheRepository: yyy;
   load: 'CI'.


Either way, this example (working, if done using all the complications
in our scripts) contradicts the documentation [6] if my reasoning is
correct.


I have hastily put a version of the above (without any WonkaLoader
stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github, with
its transcript in logs/transcript.2018-01-09.log  but have not applied
my mind to that yet. Obviously this naive version still fails.




> and it looks like you are running with a pure fetch load and you are
> doing of fetch of the not only GLASS/GsDevKit, but your entire application.

Not quite. WonkaLoader is used to load a group from
BaselineOfWonkaRuntime, and the baseline only contains dependencies to
stuff we need, not any of our actual own code (besides what is contained
in BaselineOfWonkaRuntime package itself).

...and the two-phase load does happen, as I explained above.


(I added some detail about our project layout at the bottom of the
README.rst file in github)




> My second thought is that I've recently been working on getting
> GsDevKit/GLASS to be bootstrapped using only git repositories (i.e., no
> network) and this script[2] (within the context of GsDevKit_home) does
> just that. It should be straightforward to adapt the script to a
> non-GsDevKit_home
>
> So if you are struggling with getting the networkless load to work it
> may make sense for you to use the bootstrapGsDevKit script ... and then
> work on getting all of the other monticello-based projects that you are
> using up on github, so that your entire application load can be done
> without hitting the network ...
>
> If you take this route then we can ignore the #'writeWordLike:' and also
> ignore the cacheRepository:/repositoryOverrides issues.

We are currently using git repos, at least for the GLASS bits, but we
would prefer not to. We'd prefer going with
cacheRepository:/repositoryOverrides and rely on Metacello to fetch
whatever is needed without us having to know the details....as I
explained above. Managing all those dependencies is what we understand
Metacello's purpose in life is... Cant we help bash it until it works
for this use case?

There is a bit more to it: the way we handle load conflicts... but I'll
leave that for a later discussion perhaps.


[1]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImage.topaz

[2]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/loadGroup..st

[3]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/initializeBaseName.repository.modes.repositoryOverrides..st

[4]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/instance/metacelloForLoading.st

[5]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/class/loadBootstrapBaseline.repository.modes..st

[6]
https://github.com/Metacello/metacello/blob/master/docs/MetacelloScriptingAPI.md#fetching

[7]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImageUsingStandardTools.topaz



--

Regards
- Iwan
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list


On 1/9/18 4:02 AM, Iwan Vosloo via Glass wrote:

> Hi guys,
>
> On 1/8/18 6:35 AM, Mariano Martinez Peck wrote:
> > Just a small comment from the outside: that problem seems to be
> > known:
> http://forum.world.st/DataStream-dnu-writeWordLike-td4953239.html
> > Maybe Iwan can check if he has instances of QuadByteString too...
> > that might be the reason...
>
> On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
>> Getting rid of the QuadByteStrings in the mcz file is one solution the
> > other solution is to save the package in Filetree format, since
> > Filetree uses utf8.
>
>
>
> Thanks Mariano and Dale,
>
> Yes, I eventually discovered the QuadByteString issue... I tried using
> filetree initially, but given the code that's there (that WonkaLoader
> thing) it is not straightforward to use filetree (I suspect because it
> uses FileTree to fetch FileTree as well, there may be a
> chicken-and-egg scenario going on...but I did not investigate).
>
> Instead I figured the QuadByteString issue must be a bug, since
> writeWordLike: is being called from code present in Gemstone, yet it
> is not implemented in Gemstone.
>
> I provided an implementation (should I log a bug somewhere?), and
> using that it actually loads using our scripts. Which is nice because
> it gives me a starting point of sorts from where I can refactor and
> simplify.
This is not a correct solution ... the byte patterns differ between
WideString and QuadByteString ... if it is working it is likely to be
luck ... or the corruption is only showing up in a comment ... or the
code compiles without error, but will fail when run ... really the only
solution in your case is to "edit the Monticello .mcz file" and remove
your patch ... it is much safer to have the load fail than debug the
potential downstream problems...

To edit a Monticello .mcz file with WideStrings, you should load the mcz
file into a pharo (or squeak) image ... from there your best bet would
seem to be to inspect the package structure and get the snapshot
definitions and find look for MCMethodDefinitions whose #source is a
kind of WideString also look for a class definition #comment that might
contain a WideString ... if you find a comment or method with a
WideString, then edit the method(s) and comment(s) to eliminate the
WideString character(s) save a new version of the .mcz file and then
edit the appropriate ConfigurationOf to point to the new .mcz version
and try the load again ...

If you are still getting an error or you didn't find any methods or
comments with WideStrings, then it is likely that a commit comment in
the .mcz contains the offending character (and there may be more than
one comment with the problem) or the problem might be another of the
fields ... all it takes is one WideString to spoil the entire package.

Here's some sample code:

| basePackageName wc comments methods versions badDefs |
basePackageName := 'Metacello-Base'.
wc := MCWorkingCopy allManagers detect: [:each | each packageName =
basePackageName].
comments := OrderedCollection new..
methods := OrderedCollection new.
wc snapshot definitions do: [:def |
     def isClassDefinition
         ifTrue: [ (def comment isKindOf: WideString) ifTrue: [ comments
add: def]].
     def isMethodDefinition
         ifTrue: [ (def source isKindOf: WideString) ifTrue: [ comments
add: def]]].
badDefs := OrderedCollection new.
comments isEmpty ifFalse: [ badDefs add: comments ].
methods isEmpty ifFalse: [ badDefs add: methods ].
badDefs isEmpty ifFalse: [ ^ badDefs ].
"now scan wc ancestry for WideStrings"
versions := OrderedCollection new.
wc ancestry allAncestorsDo: [:aMCVersionInfo |
     (aMCVersionInfo message isKindOf: WideString) ifTrue: [ versions
add: aMCVersionInfo ] ].
versions
>
> Of course there's strong incentive to just leave it at that from the
> people who have to pay the $$ for this... But I would have liked to be
> able to load it all (via/not via network) using only what is provided
> by GsUpgrader and Metacello without all the various things in our
> WonkaLoader class....
Yeah the FinWorks WonkaLoader likely predates Metacello let alone
GsUpgrader and GsDevKit_home...
>
> It is painful to have to get all the right branches of all the right
> projects from all the right github repositories locally just to be
> able to load...
Hmmm, it seems that there is some pain associated with the current
scheme and once you do clone the projects and get them to the right
version then you will be able to load without network access all of the
time and when it is a good time for you to update the version of a
project you are using, you can do it on your own schedule when you are
prepared for the porting effort ...

A quick scan o f the http: projects that you are using several different
http: repositories ... Other than Seaside which turns into a github
download for GemStone anyway and Metacello (which doesn't need http:
access if you use bootstrapGsDevKit[2]) , the other http: repositories
I've identified are:

   http://seaside.gemtalksystems.com/ss/hyper
   http://smalltalkhub.com/mc/hernan/XMLPullParser/main
   http://smalltalkhub.com/mc/PharoExtras/XMLParser/main
http://smalltalkhub.com/mc/PharoExtras/OrderPreservingDictionary/main
   http://www.smalltalkhub.com/mc/PharoExtras/BitmapCharacterSet/main
   http://smalltalkhub.com/mc/Swazoo/Swazoo/main
   http://seaside.gemtalksystems.com/ss/swazoo2

hyper also uses a local clone for bootstrapGsDevKit[2], but the others
have not been moved to Github yet ... are you using Swazoo for your web
server? If not then you are down to only 4 different projects based on
http: and I'm thinking that the XML projects need to move to GitHub soon
and have already heard from Monty that he would be willing to do that
... The other two OrderPreservingDictionary and  BitmapCharacterSet
presumably aren't changing that frequently to become difficult to manage
as a git clone ....

  As for the github: projects that aren't already covered by
bootstrapGsDevKit[2] it seems that only SeasideSt/Seaside is being used ...

I don't know, but from where I am sitting, it would be easier to manage
git repositories than try to figure out why your scripts aren't working:)

Oh well...

[1]
https://raw.githubusercontent.com/finworks/BaselineOfWonkaRuntime/master/logs/transcript.2018-01-01.log
[2]
https://github.com/GsDevKit/GsDevKit_home/blob/issue_200/bin/bootstrapGsDevKit 

> being able to fetch the correct dependencies at their correct
> versions, from wherever, recursively is exactly the point of using
> metacello in the first place. Having to manage that ourselves defeats
> the point of using Metacello in the first place..
> (Besides, many of our dependencies are just Monticello packages anyways)
Things are not quite that simple ... and I would say that if you
downloading individual package versions, then I would claim that a git
pull is much simpler than hand editting ConfigurationOf files to change
the version of the package you are using ...

If you are not specifically managing the versions of packages and
projects (i.e., not hand editing ConfigurationOf files) then you are at
the mercy of the project developers as to which version of the project
you pick up when you update and have no control (other than hand
editting ConfigurationOf files) over which version you are actually
using in your product ...

I would argue that with git clones you know exactly which version of
each of the projects you are using ... and you are in complete control
over when you want to use a new version ...

With all of the said, you are working with a legacy production loading
scheme and if you are going to pay the price of converting the load
system, I would recommend that you wait until GemStone 3.5 (planned, not
committed) comes out with Rowan support ... at which time package
loading, Metacello and git support will be incorporated in the GemStone
kernel and be supported by GemTalk Systems instead of on community
support, while still maintaining open source status for those tools...

>
> If you want to play along, my answers to Dale:
>
>
> On 08/01/2018 16:24, Dale Henrichs via Glass wrote:
> > I highly recommend to the bootstrapGsDevKit approach, but if you don't
> > want to take that route, then you should try splitting your load into
> > two phases 1) fetch and load GLASS 2) fetch and load your application.
>
> That is actually exactly what our code does. When you call WonkaLoader
> class >> loadWonkaRuntimeGroup:fromWonkaRoot:platform:overNetwork:
> (see the call in topaz/makeCIImage.topaz [1] ):
>  - it first updates/loads GsUpgrader, FileTree, Metacello, Grease,
> GLASS1 (in that order, from locally cloned git repositories); and
>  - then it loads group 'CI' of BaselineOfWonkaRuntime from
> wonkaRoot/monticello repository (this is where our own code sits).
Okay that makes sense, but I can't tell that from the loading Transcript
... dumping into the Transcript the steps that are performed by Wonka
(and the Metacello or other load expression used) would help me
understand where the transitions are rather than trying to interpret
things from a pure Metacello log ...
>
> But, all the loading that takes place - whether loading a "bootstrap
> dependency" (the GLASS/Metacello stuff) or any other package to load -
> is done in the same method: WonkaLoader>>loadGroup [2], using the same
> code to create a Metacello [3] and [4].
Okay ... skipping ahead a bit I don't see a Smalltalk stack and without
the wonka loader markers in the new log[3], I don't really know when or
what happened so save reading the various code snippets until I can get
my hands on a stack ...

[3]
https://github.com/finworks/BaselineOfWonkaRuntime/commit/6316b77d63189f5c47407403d6a9f7f1e37f048e

>
> In the first phase, (loading bootstrap dependencies), it sets up a new
> Monticello independently for each of those "bootstrap" projects in
> turn, on their own (cloned github) repository. ( See
> BaselineOfWOnkaRuntime class >>
> loadBootstrapBaseline:repository:modes: [5] )
>
> In the second phase, it uses the same underlying method to set up a
> new Metacello on our own repository.
>
> What is shared in all of the above is how conflicts etc are handled;
> and the given repositoryOverrides are always ignored if you do it
> overNetwork. Further, if you DO load overNetwork, it sets the
> cacheRepository, and always to the same platform-dependent location
> (which we check into our git repo).
>
>
> What I understand our not-so-pretty code does in short (ignoring
> handling of conflicts etc) is:
>
> Metacello new
>   baseline: xxx;
>   repository: aLocalGitClone;
>   cacheRepository: yyy;
>   get.
>
> Metacello new
>   baseline: xxx;
>   repository: aLocalGitClone;
>   cacheRepository: yyy;
>   load: 'default'.
>
> For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that
> order)
>
> Then it does:
>
> Metacello new
>   baseline: xxx;
>   repository: ourOwnFileTreeRepo;
>   cacheRepository: yyy;
>   get.
>
> Metacello new
>   baseline: xxx;
>   repository: ourOwnFileTreeRepo;
>   cacheRepository: yyy;
>   load: 'CI'.
>
>
> Either way, this example (working, if done using all the complications
> in our scripts) contradicts the documentation [6] if my reasoning is
> correct.
I guess I'm confused right now ... the log new log does not look like
"working" to me as there is an error at the very end and I have not seen
a stack ...
>
>
> I have hastily put a version of the above (without any WonkaLoader
> stuff) in topaz/makeCIImageUsingStandardTools.topaz [7] on github,
> with its transcript in logs/transcript.2018-01-09.log  but have not
> applied my mind to that yet. Obviously this naive version still fails.
it's not obvious to me and without a stack I guess I cannot comment with
any authority.

If the load process is actually working, then I'm not exactly sure why I
spent an hour or so on my weekend answering this email message only to
find out that you are apparently saying "never mind at the end" ...

Dale....

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Hi Dale,

>> Either way, this example (working, if done using all the
>> complications in our scripts) contradicts the documentation [6] if my
>> reasoning is correct.
> I guess I'm confused right now ... the log new log does not look like
> "working" to me as there is an error at the very end and I have not
> seen a stack ...
[...]
> If the load process is actually working, then I'm not exactly sure why
> I spent an hour or so on my weekend answering this email message only
> to find out that you are apparently saying "never mind at the end" ...

Sorry, no, that's not what I meant, I did not express it clearly. (And I
do appreciate your help!)

What is working is: using the WonkaLoader, ie, in that code, running
./bootstrap_image.sh -n (a)

The way I understand WonkaLoader code is that *in principle* it does do
a version of my "almost-pseudocode" below (b), except (a) adds ways to
deal with conflicts and a bunch of other stuff.  (And, in (b) I am
trying to use a filetree repo)

>> Metacello new
>>   baseline: xxx;
>>   repository: aLocalGitClone;
>>   cacheRepository: yyy;
>>   get.
>>
>> Metacello new
>>   baseline: xxx;
>>   repository: aLocalGitClone;
>>   cacheRepository: yyy;
>>   load: 'default'.
>>
>> For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that
>> order)
>>
>> Then it does:
>>
>> Metacello new
>>   baseline: xxx;
>>   repository: ourOwnFileTreeRepo;
>>   cacheRepository: yyy;
>>   get.
>>
>> Metacello new
>>   baseline: xxx;
>>   repository: ourOwnFileTreeRepo;
>>   cacheRepository: yyy;
>>   load: 'CI'.


My reasoning is that, if (a) works, and I understand that it really does
(b) with some complications added, I should be able to do (b).

But just doing (b) straight up does not work (that's the log you saw)
and I have not exposed its breakage for you properly last week - I ran
out of time. I am busy delving deeper into it today and will report what
I manage to get going.




About the WideString issue and missing  writeWordLike: you say:
> This is not a correct solution ... the byte patterns differ between  > WideString and QuadByteString ... if it is working it is likely to be
 > luck ... or the corruption is only showing up in a comment ... or the
 > code compiles without error, but will fail when run ... really the
only > solution in your case is to "edit the Monticello .mcz file" and
remove > your patch ... it is much safer to have the load fail than
debug the > potential downstream problems...

I understand, I did not try to build a proper solution - I just copied a
Pharo implementation over to see what happens. Point is: is it not
possible to provide a proper implementation for writeWordLike: (since
there is code in GS that calls it)?

This also means that whatever I *think* is working using our existing
stuff might be breaking in weird ways, so I have to actually go back and
sort this out better.


> Here's some sample code:
(thanks)


> ... are you using Swazoo for your web server?

I think I can ditch Swazoo, it apparently used to be used in some
scenarios, but if I hit those I'll switch them to zinc.


>> being able to fetch the correct dependencies at their correct
>> versions, from wherever, recursively is exactly the point of using
>> metacello in the first place. Having to manage that ourselves defeats
>> the point of using Metacello in the first place..
>> (Besides, many of our dependencies are just Monticello packages anyways)
> Things are not quite that simple ... and I would say that if you
> downloading individual package versions, then I would claim that a git
> pull is much simpler than hand editting ConfigurationOf files to change
> the version of the package you are using ...
>
> If you are not specifically managing the versions of packages and
> projects (i.e., not hand editing ConfigurationOf files) then you are at
> the mercy of the project developers as to which version of the project
> you pick up when you update and have no control (other than hand
> editting ConfigurationOf files) over which version you are actually
> using in your product ...
>
> I would argue that with git clones you know exactly which version of
> each of the projects you are using ... and you are in complete control
> over when you want to use a new version ...

I understand your motivation for git clones. I suspect there's stuff I
do not understand about "being at the mercy of project developers".  I
am also just used to similar tools in other ecosystems that do not rely
on me having to clone git repos (apt, npm, maven, pip) - these do the
recursive fetch (although some also have issues regarding conflicting
dependencies). My "being used to" these tools, and from Metacello docs I
thought this is what Metacello does too.

What do you mean by hand-editing ConfigurationOf files? Do you mean
editing our own, which includes project versions? If we are specific in
that, won't we always pick up the same version files? Isn't this just
like changing the debian/rules of your own deb package source?

>
> With all of the said, you are working with a legacy production loading
> scheme and if you are going to pay the price of converting the load
> system, I would recommend that you wait until GemStone 3.5 (planned, not
> committed) comes out with Rowan support ... at which time package
> loading, Metacello and git support will be incorporated in the GemStone
> kernel and be supported by GemTalk Systems instead of on community
> support, while still maintaining open source status for those tools...

Thanks, I hear you. I am trying to understand the tools and my options
at the moment, so this certainly is one option to keep in mind.


Regarding the rest of your comments: I am working on this today again. I
am hoping to delve deeper into it and will check in better logs and
report back...

Thank you
-Iwan

--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list


On 1/15/18 3:11 AM, Iwan Vosloo via Glass wrote:

> Hi Dale,
>
>>> Either way, this example (working, if done using all the
>>> complications in our scripts) contradicts the documentation [6] if
>>> my reasoning is correct.
>> I guess I'm confused right now ... the log new log does not look like
>> "working" to me as there is an error at the very end and I have not
>> seen a stack ...
> [...]
>> If the load process is actually working, then I'm not exactly sure
>> why I spent an hour or so on my weekend answering this email message
>> only to find out that you are apparently saying "never mind at the
>> end" ...
>
> Sorry, no, that's not what I meant, I did not express it clearly. (And
> I do appreciate your help!)
Okay ... I understand ... email is often imperfect especially for
complicated problems ..

>
> What is working is: using the WonkaLoader, ie, in that code, running
> ./bootstrap_image.sh -n (a)
>
> The way I understand WonkaLoader code is that *in principle* it does
> do a version of my "almost-pseudocode" below (b), except (a) adds ways
> to deal with conflicts and a bunch of other stuff. (And, in (b) I am
> trying to use a filetree repo)
>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: aLocalGitClone;
>>>   cacheRepository: yyy;
>>>   get.
>>>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: aLocalGitClone;
>>>   cacheRepository: yyy;
>>>   load: 'default'.
>>>
>>> For all of: GsUpgrader, FileTree, Metacello, Grease, GLASS1 (in that
>>> order)
>>>
>>> Then it does:
>>>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: ourOwnFileTreeRepo;
>>>   cacheRepository: yyy;
>>>   get.
>>>
>>> Metacello new
>>>   baseline: xxx;
>>>   repository: ourOwnFileTreeRepo;
>>>   cacheRepository: yyy;
>>>   load: 'CI'.
>
>
> My reasoning is that, if (a) works, and I understand that it really
> does (b) with some complications added, I should be able to do (b).
>
> But just doing (b) straight up does not work (that's the log you saw)
> and I have not exposed its breakage for you properly last week - I ran
> out of time. I am busy delving deeper into it today and will report
> what I manage to get going.
Probably the best way for me to help is for me to start with a stack and
a log file ... from the stack I tell where in the code the error is
coming from ... from the log file I should be able to tell which part of
the process is being run ... and then that will connect with the scripts .

I don't expect to be able to be an expert on how the overall WonkaLoad
goes ... but with a stack I will be able to pinpoint where things are
going wrong.

>
>
>
>
> About the WideString issue and missing  writeWordLike: you say:
>> This is not a correct solution ... the byte patterns differ between 
>> > WideString and QuadByteString ... if it is working it is likely to be
> > luck ... or the corruption is only showing up in a comment ... or
> the > code compiles without error, but will fail when run ... really
> the only > solution in your case is to "edit the Monticello .mcz file"
> and remove > your patch ... it is much safer to have the load fail
> than debug the > potential downstream problems...
>
> I understand, I did not try to build a proper solution - I just copied
> a Pharo implementation over to see what happens. Point is: is it not
> possible to provide a proper implementation for writeWordLike: (since
> there is code in GS that calls it)?
Monticello is based on serializing a collection of MCDefinitions using
DataStream which serializes objects to a stream. Like most serializers
the objects are copied to a byteStream recording class names and fields
in a platform-specific fashion. With some luck and a little bit of
hacking that made it possible for GemStone to be able to deserialize
Monticello packages.

The String implementation is common across all platforms, but when it
comes to multi-byte characters each dialect does things a bit
differently and in the case of Squeak/Pharo the WideString
implementation is completely different from the GemStone multibyte
implementations (DoubleTypeString, QuadByteString, Unicode16, and
Unicode32) and I've never even attempted to create a proper converter.

By the time multi-byte characters became common in pacakges, we had
moved to FileTree which has proper Utf8 support and a cross-dialect
serialization model ...

It is probably possible to provide a converter between WideString and
the GemStone character collection hierarchy, but I do not intend to
spend a lot of time on Monticello ... it will be supported "forever" but
I hope to make it an optional addon in GsDevKit.

Over time, the kernel of GsDevKit will rely less and less on the pharo
compatibility layer as the chunks of functionality are moved into the
kernel. Monticello itself requires quite a bit of pharo compatibility
and I have no interest in rewriting Monticello to run in the kernel --
tonel and filetree are better disk-based package formats as they are SCM
neutral and Utf8 aware and we have already ported tonel, filetree and
Metacello to the kernel so Monticello will be supported for folks who
still rely on http: .mcz repositories and Monticello will run continue
to run on top of the pharo compatibility layer (which will of course be
kept and expanded --- support for specific pharo versions should be
possible then ...
>
> This also means that whatever I *think* is working using our existing
> stuff might be breaking in weird ways, so I have to actually go back
> and sort this out better.
You only have to worry about code that is loaded with writeWordLike: and
if the issue is in the ancestry, then the corruption would affect the
loaded code ... however, you may run into trouble saving packages that
end up being non-String objects ...

>
>
>> Here's some sample code:
> (thanks)
>
>
>> ... are you using Swazoo for your web server?
>
> I think I can ditch Swazoo, it apparently used to be used in some
> scenarios, but if I hit those I'll switch them to zinc.
>
>
>>> being able to fetch the correct dependencies at their correct
>>> versions, from wherever, recursively is exactly the point of using
>>> metacello in the first place. Having to manage that ourselves
>>> defeats the point of using Metacello in the first place..
>>> (Besides, many of our dependencies are just Monticello packages
>>> anyways)
>> Things are not quite that simple ... and I would say that if you
>> downloading individual package versions, then I would claim that a
>> git pull is much simpler than hand editting ConfigurationOf files to
>> change the version of the package you are using ...
>>
>> If you are not specifically managing the versions of packages and
>> projects (i.e., not hand editing ConfigurationOf files) then you are
>> at the mercy of the project developers as to which version of the
>> project you pick up when you update and have no control (other than
>> hand editting ConfigurationOf files) over which version you are
>> actually using in your product ...
>>
>> I would argue that with git clones you know exactly which version of
>> each of the projects you are using ... and you are in complete
>> control over when you want to use a new version ...
>
> I understand your motivation for git clones. I suspect there's stuff I
> do not understand about "being at the mercy of project developers".  I
> am also just used to similar tools in other ecosystems that do not
> rely on me having to clone git repos (apt, npm, maven, pip) - these do
> the recursive fetch (although some also have issues regarding
> conflicting dependencies). My "being used to" these tools, and from
> Metacello docs I thought this is what Metacello does too.
Automatically cloning git repos is something that is coming ... it will
be a replacing for the old github: to tar file download ... my concern
is that if local git clones are not used, then contributing to a project
that is managed by git on github becomes much more difficult ... if the
git clones are loaded into the image from the start, then it is natural
to be able to push any changes/bugfixes back to the originating project ...

Isn't it true that for apt, npm, maven, pit, etc., that you are talking
about deployment when you say that you don't need git clones? If you
want to contribute to any of those projects you would need to use a git
clone, right?

With Smalltalk the development environment is always part of the
deployment there isn't really a distinction between deployment and
development ... we often develop in our deployed app ..
>
> What do you mean by hand-editing ConfigurationOf files? Do you mean
> editing our own, which includes project versions? If we are specific
> in that, won't we always pick up the same version files? Isn't this
> just like changing the debian/rules of your own deb package source?
If you are specific, you always get the same version files. If you want
to move to a different version then you have to edit a ConfigurationOf
file to change the version of each project ... and I actually think that
pulling a different branch/tag/sha is actually easier than the process
required to change the ConfigurationOf file ...

>
>>
>> With all of the said, you are working with a legacy production
>> loading scheme and if you are going to pay the price of converting
>> the load system, I would recommend that you wait until GemStone 3.5
>> (planned, not committed) comes out with Rowan support ... at which
>> time package loading, Metacello and git support will be incorporated
>> in the GemStone kernel and be supported by GemTalk Systems instead of
>> on community support, while still maintaining open source status for
>> those tools...
>
> Thanks, I hear you. I am trying to understand the tools and my options
> at the moment, so this certainly is one option to keep in mind.
>
>
> Regarding the rest of your comments: I am working on this today again.
> I am hoping to delve deeper into it and will check in better logs and
> report back...
Don't forget to include stacks ... the stack is just as important as the
error message ... at least for me;)

Dale
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Hi Dale.

I tried getting a version going that does kinda what WonkaLoader does,
but not using the WonkaLoader class. That was tricky because it contains
logic in onUpgrade/onDowngrade/onConflict which is not trivial [1].
Doing this, I managed to get it to load overNetwork using filetree repos
and using GsUprader, GsDeployer etc. But, I could not get it to work
without the network.

I am starting to feel doing this is not productive....

So, I switched back to getting WonkaLoader to work. The issue there was
that I had to make it use a filetree cache repository in order to bypass
the character encoding issue with mcz files.

This is now on github, but it breaks with log [2]. (Stack is included
there, but not the WHOLE stack.) It would be great if you can advise on
this...

Just to recap (in case it helps with navigating the log output):

This thing works in 2 phases as you also suggested. First phase it loads
the basic projects, in this order:

GsUpgrader (from a local git clone)

MetacelloPreviewBootstrap (from a local mcz repo, possibly old [3])
MetacelloPreview (from a local git clone [4])

FileTree (from a local git clone)
Metacello (from a local git clone)
Grease (from a local git clone)
GLASS1 (from a local git clone)

These are prefixed with '*** Loading' in the log.

The breakage happens while loading GLASS AFAIK. It never gets to the
second phase, which is to load BaselineOfWonkaRuntime.

[1]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/topaz/makeCIImageUsingStandardTools.topaz

[2]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.overNetwork.2018-01-16.log

[3]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/class/loadGemStoneMetacelloPreviewBootstrap.st

[4]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/monticello/BaselineOfWonkaRuntime.package/WonkaLoader.class/class/loadGemStoneMetacelloPreview.st


Thanks
- Iwan

--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list


On 16/01/2018 14:34, Iwan Vosloo via Glass wrote:
> This is now on github, but it breaks with log [2]. (Stack is included
> there, but not the WHOLE stack.) It would be great if you can advise on
> this...

Hang on... as Murphy would have it I found this problem shortly after
sending my email. It was also trying to read all the ConfigurationOfs
but tried to read the (not filetree) repository as an
MCDirectoryRepository. I'll report wat I achieve going past this...

Thanks
-Iwan


--

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
On 16/01/2018 14:49, Iwan Vosloo via Glass wrote:
>
> I'll report wat I achieve going past this...
>

I have now managed to get it to load (barring a few warnings) using
WonkaLoader via the network using filetree on GS3.4 [1].  When I load it
bypassing the network though, things break [2] with:

"Class category name 'Sport' for the class 'SpAbstractError' is
inconsistent with the package name 'Sport3.010.v3'"



[1]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.overNetwork.2018-01-16.2.log

[2]
https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.NOToverNetwork.2018-01-16.2.log


Thanks
- Iwan

--


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Another metacello loading puzzle: xml

GLASS mailing list
Iwan,

I see that you are using a filetree cache now, which is clever, but
because .mcz files does not have a consistent file/package naming
convention, when the Sport package is converted to filtree, which _does_
have an enforced package naming convention it runs into the consistency
check error for filetree:( Between the "corrupt" .mcz files and
improperly named .mcz files you are caught between a rock and a hard place.

Now it turns out I've already moved Sport to a github project[1] and the
github project for Sport does not have this inconsistency[2]. It looks
like I renamed the Sport3.010.v3 package to Sport.v3 and you might be
able to pull off the same trick by renaming the package in the hyper
project[1] and whatever ConfigurationOf that references it - but this is
wandering into complicated territory that is only becoming a problem
because you are trying to use a filetree cache ... If this is the very
last problem, then it might be worth it ...

I think it would take an afternoon to clone all of the projects from
github and create local git repos for the projects that you use that are
not on github and I would recommend that you use GsDevKit[4] instead of
GLASS[5], because I've already done the work of converting all of the
dependent projects from .mcz to github projects that are loaded in a
standard install ... the actual code in GsDevKit and GLASS is
"identical" (GLASS is a fork of the GsDevKit project and pull requests
are shared between the two) so it should be that difficult to switch out
and pretty quickly I think that you would be networkless and loading all
of your source from disk-based git repositories and you would be heading
in the right direction.

I have been doing builds using the GsDevKit repository since October and
for the last month I've been doing active development in a GsDevKit (not
GLASS) stone on a daily basis..

So I strongly suggest that you reconsider your decision to not follow my
recommendation to use local git repos as a solution to your networkless
load problems.

Dale

[1] https://github.com/GsDevKit/Sport
[2]
https://github.com/GsDevKit/Sport/blob/master/src/Sport.v3.package/SpAbstractError.class/properties.json
[3] http://seaside.gemtalksystems.com/ss/hyper.html
[4] https://github.com/GsDevKit/GsDevKit/tree/gs_port_200
[5] https://github.com/glassdb/glass

On 1/16/18 5:38 AM, Iwan Vosloo via Glass wrote:

> On 16/01/2018 14:49, Iwan Vosloo via Glass wrote:
>>
>> I'll report wat I achieve going past this...
>>
>
> I have now managed to get it to load (barring a few warnings) using
> WonkaLoader via the network using filetree on GS3.4 [1]. When I load
> it bypassing the network though, things break [2] with:
>
> "Class category name 'Sport' for the class 'SpAbstractError' is
> inconsistent with the package name 'Sport3.010.v3'"
>
>
>
> [1]
> https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.overNetwork.2018-01-16.2.log
>
> [2]
> https://github.com/finworks/BaselineOfWonkaRuntime/blob/master/logs/transcript.WonkaLoader.NOToverNetwork.2018-01-16.2.log
>
>
> Thanks
> - Iwan
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
12