Adding collection classes to core GLASS?

Hi,

I'm using the XMLWriter project (squeaksource/XMLWriter) in a codebase that is being kept compatible between Pharo and GemStone.
I noticed that GemStone does not have the CharacterSet class (which is required for XMLWriter to work).

Just to be able to quickly make things load and work, I added a package containing the class to the XMLWriter repository and updated the XMLWriter metacello config to load the package in gemstone.

However, there are still some things to adapt to make it work completely (which I intend to address) but a general question come to my mind: since the CharacterSet class is a part of Pharo core, would it make sense to have this class part of the GLASS code too (given that the code is adapted apropriately). And who/what/how would be required to make that happen?

cheers
Johan

Johan,

IN GLASS 1.0-betya.8.5, I added a trio of configurations:

   ConfigurationOfGsSqueakCommon
   ConfigurationOfGsSqueak
   ConfigurationOfGsPharo

for the purpose of managing these types of extensions ... The repository:

   http://seaside.gemstone.com/ss/PharoCompat

was created to house the compatibility code.

If I'm not mistaken, the class CharacterSet is in Squeak4.x as well, so
it would be appropriate to add the class to
ConfigurationOfGsSqueakCommon... If the actual implementation of
CharacterSet is diverging between Pharo and Squeak then it would be
appropriate to implement the common protocol in a package in
ConfigurationOfGsSqueakCommon and the rest of the methods in two other
packages ...

ConfigurationOfGsSqueakCommon is included by default in GLASS (on 2.x
right now, but 3.x in the future) so to the extent that the code used by
XMLWriter is the common set of methods between Squeak and Pharo,
XMLWriter could be dependent on ConfigurationOfGsSqueakCommon ...

I am hoping that you guys could contribute by creating the new
packages/tests, and updating configurations as needed ... mail could be
used to coordinate, or we could dedicate pages on the wiki or use the
issue tracking system for coordination ...

I am willing to support the process (helping with configuration issues,
creating repositories, etc.) but I'd like it so that I wasn't a
_required_ piece of the puzzle:)

Dale

On 01/31/2011 12:21 PM, Johan Brichau wrote:

Hi,

I've started with something as this is bothering me too.

I'm just finding it difficult to figure some things out. As I'm
writing, I get more nervous about writing big emails. Sorry, but I'm
summarising a morning's battle. Can you please help?

1. I was thinking that since the CharacterSet, WideCharacterSet and
CharacterSetComplement classes work together, I should put it in a
package resembling Collections-Support (the MC package loaded in
Pharo). I also saw Collections-OrderPreservingDictionary and thought
well, is this not rather a port of more monticello packages to
GemStone?
2. So I created Collection-Support.g, with a GemStone version of the
code in it. This is because I see packages like XML-Parser.g. But I am
a bit confused when to use the .g version and when to use the MC
convention of something like <name>-<platform>-Core. I see though that
the .g name is mangled a bit because I just have to specify the
dependency as Collections-Support and then in a #gemstone directive
say it is the .g one. So what is the approach I should follow here?
3. The Collections-Support package has nice unrelated stuff in it, for
example classes that support Collections-Weak. This is too daunting to
port now, so I removed the classes I don't want to port from
Collections-Support.g. I suppose the better way is to create a new
package named something like Collections-CharacterSets and move the
CharacterSet related stuff in there. How do I go about doing this? Is
this sensible?
4. There would be a lot of common code between
Collections-CharacterSet and the GemStone specific version except for
a few issues: WideCharacterSet | #bitmap:do: uses the ByteArray syntax
#[1 2 3], which does not compile into GemStone because it uses #[1, 2,
3] for dynamically constructed arrays (same as {1. 2. 3}). If I want
to preserve a lot of code in Collections-CharacterSets and override
some stuff like this for GemStone, how do I go about it? Or is the
better answer to just change the code in WideCharacterSet to be more
compatible?
5. Another compatibility problem was that #become: is called if adding
a "Wide" character (ascii >= 256) to a CharacterSet: it tries to
become a WideCharacterSet. This code will not work in GS as the
receiver is on the stack. I changed the "constructor" method to check
for wide characters first and then to decide what class to
instantiate. (It does leave a gap for a bug though). How do I resolve
this?
6. I extracted some tests from CollectionsTests(-Support) and got them
to pass in GS. This was after adding a few missing methods to
Core(-DaleHenrichs.35). It's difficult to figure out where methods
should go. What's the best approach? I looked for "similar" methods
and found the *core-squeak protocol to make the most sense. And I
created Collections-Tests-Support. Should probably also be
Collections-Tests-CharacterSet.
7. I changed ConfigurationOfXMLWriter to depend on Collections-Support.
8. When changing things, I created a lot of versions. I built them in
Pharo and loaded them into GemStone to see how it works. I am trying
to get our configuration to load cleanly. When I'm reasonably happy,
how to I "publish" the version I created, without a bunch of
unnecessary versions and rubbish inbetween? How do you generally work
re. this kind of thing?

Enough for now. I've got the mcz files, if you'd like to see them.

Cheers
Otto

On Mon, Jan 31, 2011 at 10:43 PM, Dale Henrichs <[hidden email]> wrote:

On 02/01/2011 02:51 AM, Otto Behrens wrote:
In the long term, I am hoping to get to the point where the pharo
packages can be loaded into GemStone ... we have a ways to go to make
that possible so incrememntal ports are the best way to go.

I'm not sure what you are trying to say about
Collections-OrderPreservingDictionary?

> 2. So I created Collection-Support.g, with a GemStone version of the
> code in it. This is because I see packages like XML-Parser.g. But I am
> a bit confused when to use the .g version and when to use the MC
> convention of something like<name>-<platform>-Core. I see though that
> the .g name is mangled a bit because I just have to specify the
> dependency as Collections-Support and then in a #gemstone directive
> say it is the .g one. So what is the approach I should follow here?

When I first started using Monticello I asked Avi what naming convention
I should use for GemStone branches of packages and he suggested using
'.g' and I started doing that ... At some point in time I decided that
'.gemstone' was a better name to use for gemstone branches, so I started
using that ...

Then I wrote Metacello. Metacello makes it easier to manage a
multi-package project, so it isn't necessary to put everything into once
XXX-Core-author.version.mcz file and then make platform-specific
branches and naming the branch XXX-Core.platform-author.version.mcz.

With Metacello it is easy to create a XXX-Core-author.version.mcz file
for the code that is intended to be common across all platforms and then
create XXX-Platform-Core-author.version.mcz with the platform-specific
code in it (one for each supported platform). So this is my inclination
today.

With Seaside I have found it necessary to make platform-specific
branches for some of the core packages like Seaside-Core, so I've used
the .gemstone branch naming convention.

So for this project I have the following suggestion:

     Treat this as a gemstone project from the start, so there's no need
     to preserve the pharo/squeak package structure. The package name
     should be something like GsCharacterSet-SqueakCommon,
     GsCharacterSet-Squeak, and GsCharacterSet-Pharo (assuming that
     there are differences between Squeak and Pharo.

Personally I prefer smaller targeted sets of packages so that meaningful
dependencies from other projects can be created that minimize the amount
of cruft that gets brought in...

> 3. The Collections-Support package has nice unrelated stuff in it, for
> example classes that support Collections-Weak. This is too daunting to
> port now, so I removed the classes I don't want to port from
> Collections-Support.g. I suppose the better way is to create a new
> package named something like Collections-CharacterSets and move the
> CharacterSet related stuff in there. How do I go about doing this? Is
> this sensible?

I partially answered this above ... Yes it makes sense to break the port
up into pieces.

When I've done a port like this in the past, I've take a Pharo image and
then duplicated the basic package structure in the Pharo image itself,
moving classes and extension methods into the new organization. Then I
write out the mcz files, suck them into GemStone and commence getting
the tests to pass in GemStone.

> 4. There would be a lot of common code between
> Collections-CharacterSet and the GemStone specific version except for
> a few issues: WideCharacterSet | #bitmap:do: uses the ByteArray syntax
> #[1 2 3], which does not compile into GemStone because it uses #[1, 2,
> 3] for dynamically constructed arrays (same as {1. 2. 3}). If I want
> to preserve a lot of code in Collections-CharacterSets and override
> some stuff like this for GemStone, how do I go about it? Or is the
> better answer to just change the code in WideCharacterSet to be more
> compatible?

My first inclination when it comes to WideCharacterSet is to not try to
port it at all. GemStone has MutiByte characters and the underlying
implementation is quite different from the WideString implementation in
Pharo/Squeak. I would be inclined to create a MultiByteCharacterSet that
was designed to work with GemStones model.

I would hope that the actual class references are embedded in extension
methods for the String/WideString classes themselves and not from the
XML code.

If there _are_ direct references from the XML code, then this idea may
have to be revised.

> 5. Another compatibility problem was that #become: is called if adding
> a "Wide" character (ascii>= 256) to a CharacterSet: it tries to
> become a WideCharacterSet. This code will not work in GS as the
> receiver is on the stack. I changed the "constructor" method to check
> for wide characters first and then to decide what class to
> instantiate. (It does leave a gap for a bug though). How do I resolve
> this?

There is a good workaround for doing become: on an object on the stack
... it involves forking off a thread to do the become: and the forked
off thread can be designed to avoid having the instance being #becomed:
on the stack.

Take a look at Class>>migrateInstances:to: for the trick.

> 6. I extracted some tests from CollectionsTests(-Support) and got them
> to pass in GS. This was after adding a few missing methods to
> Core(-DaleHenrichs.35). It's difficult to figure out where methods
> should go. What's the best approach? I looked for "similar" methods
> and found the *core-squeak protocol to make the most sense. And I
> created Collections-Tests-Support. Should probably also be
> Collections-Tests-CharacterSet.

Yes the package structure is a bit messy in that area of GLASS, to make
it possible to bootstrap Monticello into an image ... The bootstrap
comes in in three pieces:

   1. a fileIn to support MczInstaller
   2. a set of mcz file loads (using MczInstaller) to get Monticello
      loaded
   3. reloading all code using Monticello

So, the *bootstrap-squeak code tended to be organized to support the
MczInstaller, the *core-squeak code tended to support the basic
functionality for Monticello and then Squeak for the rest.

This bootstrap process is used for GemStone2.x, but for GemStone3.x
Monticello is already ported to the base, so the distinction between
bootstrap/core/squeak are less important.

Historically, I've added extension methods that weren't needed by
bootstrap or Monticello to the Squeak package.

Today I'd be inclined to add those methods to GsSqueakCommon as long as
the Squeak and Pharo implementations are compatible.

With that said, Core works just as well, since GsSqueakCommon and Core
are always loaded.

> 7. I changed ConfigurationOfXMLWriter to depend on Collections-Support.
> 8. When changing things, I created a lot of versions. I built them in
> Pharo and loaded them into GemStone to see how it works. I am trying
> to get our configuration to load cleanly. When I'm reasonably happy,
> how to I "publish" the version I created, without a bunch of
> unnecessary versions and rubbish inbetween? How do you generally work
> re. this kind of thing?

I would publish the mcz files and configuration in:

   http://seaside.gemstone.com/ss/PharoCompat

and then copy the Configuration to:

   http://seaside.gemstone.com/ss/MetacelloRepository

>
> Enough for now. I've got the mcz files, if you'd like to see them.

I guess I have suggested some name changes and perhaps we should discuss
the names a bit more, I have opinions, but not strong opinions:)

I would like to see your configuration for Collections-Support, since
that will give me a better idea of the existing package structure.

Dale

Thanks for the effort to got through the detail.

Yes, agree.

>
> I'm not sure what you are trying to say about
> Collections-OrderPreservingDictionary?

I just had the idea that it is sensible to try to keep the
Collections-X packages in the same structure as we are aiming to load
them in GS as is. ConfigurationOfXMLWriter loads
Collections-OrderPreservingDictionary, so I thought it a good idea to
do that too by creating Collections-Support, altough the port is not
easy.

OK, will do. Agree, smaller is better as long as we don't end up with
one class per package :-)

> When I've done a port like this in the past, I've take a Pharo image and
> then duplicated the basic package structure in the Pharo image itself,
> moving classes and extension methods into the new organization. Then I write
> out the mcz files, suck them into GemStone and commence getting the tests to
> pass in GemStone.

Do you then publish the new packages as a replacement onto
squeaksource, for example?

> There is a good workaround for doing become: on an object on the stack ...
> it involves forking off a thread to do the become: and the forked off thread
> can be designed to avoid having the instance being #becomed: on the stack.
>
> Take a look at Class>>migrateInstances:to: for the trick.

Eeew, trickery scares me, but OK, can try.

OK. A newbie kind of question: How do I do this publish and how do I
copy the configuration? I have
ConfigurationOfXMLWriter-OttoBehrens.12.mcz. If I publish from Pharo
to gemsource, I create no. 13. With 10, 11, 12 missing. Is this OK?

> I guess I have suggested some name changes and perhaps we should discuss the
> names a bit more, I have opinions, but not strong opinions:)
>
> I would like to see your configuration for Collections-Support, since that
> will give me a better idea of the existing package structure.

I don't have one. I just included the package in
ConfigurationOfXMLWriter. Is it necessary to create a Configuration
for one package?

I know this is an old debate, but I really think that we should be
using dependency information in packages. A package should know what
other packages it depends upon (like subclasses and direct
references). And then perhaps what versions. It would make the job of
Metacello a lot cleaner. I still don't get the argument why this
cannot be done.

Cheers
Otto

On Feb 2, 2011, at 9:11 AM, Otto Behrens wrote:

No, I usually create a GemSource repository for the gemstone packages...we're talking about Pharo/Squeak compatibility here, so this sort of thing involves monkeying with the fundamental packages that the core images are built from, so anything that is done in these packages are going to be very GemStone-specific...

>> There is a good workaround for doing become: on an object on the stack ...
>> it involves forking off a thread to do the become: and the forked off thread
>> can be designed to avoid having the instance being #becomed: on the stack.
>>
>> Take a look at Class>>migrateInstances:to: for the trick.
>
> Eeew, trickery scares me, but OK, can try.

GemStone itself will automatically convert a String instance into a MultiByteString without using become: when a multibyte character is added to a String and the same sort of thing could be done for characterset support, but until the vm support is added, this kind of trick is needed ... in the end, though it is a good technique ... the vm cannot do become for instances that are on the stack, because there is a 'special' copy of the object on the C stack ... when the process is swapped out , the 'special' copy no longer exists and normal become: activity is allowed...

Yeah this will work fine ... holes in the version numbering sequence aren't a problem...

In my comment above though I was talking about the configuration for the collections-support packages.

Metacello configurations are designed for multi-platform support so if you are adding GemStone support to an existing metacello configuration, I would not publish a new version in PharoCompat.

Also, keep in mind that you can copy and mcz file from one repository to another without creating a new version of the mcz file ... that's what I would do in this case ...


It is not necessary to create a configuration for one package, however, if an mcz file is going to be referenced from multiple metacello configurations, then a configuration is preferred.

>
> I know this is an old debate, but I really think that we should be
> using dependency information in packages. A package should know what
> other packages it depends upon (like subclasses and direct
> references). And then perhaps what versions. It would make the job of
> Metacello a lot cleaner. I still don't get the argument why this
> cannot be done.

Metacello was created because it was much easier to add the dependencies external to Monticello than change the fundamental structure of Monticello packages ...

I was hoping that MC2 would deal with dependencies since it involved brand new package structure, but as soon as it became clear that MC2 wasn't going to address dependency/configuration issues, I decided to write Metacello. Keep in mind that we're have to allow for conditional dependencies ... classes and individual methods may have to have different implementation depending upon the platform that the system is running on ....

In my opinion, the issue isn't technical, but social/practical. I think it is a big job to introduce a new source code management system ... not only will the file formats change but all of the tools will have to be updated ... just look at MC2 to see how hard this is...

I agree that something should be done and I assume that Pharo will eventually tackle the problem ...

Dale

Hi Otto,