[ANN] Another namespaces approach.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] Another namespaces approach.

Michael van der Gulik-2
Hi all.

Now by this stage you're probably all thoroughly sick of namespaces.

I've put together a more complex namespaces architecture. It's now available on the PackageUniverses for 3.10. Note that this release is still very buggy and for curious developers only.

To install: Go to Tools/NamespaceBrowser and install that.

Then browse to PasteUpMorph>>isSafeToServe, modify and save the source there. (I noticed this bug just now).

Then evaluate: "NamespaceExamples openPackageManager."

This converts existing code in the image and eventually pops up a PackageManager with four packages: Kernel, Collections, Morphic and Legacy (which contains the SystemDictionary). Right-click on these and click "Browse" and play around some. Then close Squeak, chuck the image away and wait for the next announcement :-). Currently there's a NamespaceBrowser, NamespaceWorkspace, and a package management tool where all the action starts from.

Some documentation for my approach is at http://gulik.pbwiki.com/Namespaces.

Some advantages of this approach:

- It doesn't need the SystemDictionary, but will work happily alongside it. It makes a couple of changes to the compiler etc, but these should not affect the workings of the rest of Squeak. Classes in namespaces are not stored in the SystemDictionary but are stored in packages (which again aren't stored in the SystemDictionary either).

- You can load two (or more) versions of the same package into the same image at the same time and make objects from either. Loading a package does not modify any other classes currently in the image.

- Packages depends on other packages by UUID, and if you modify a package it gets a new UUID. This means that your code will never suffer from dependency problems and will always run in exactly the same environment you wrote and tested it in. Bit-rot shouldn't happen unless you start changing the VM.

- You don't need to use haltOnce: any more. The system can run the code in a read-only package (not yet supported...) and you can do your work on a working copy of that package.

- Naming collisions are impossible (or at least they should be... not yet tested).

- It doesn't need to be integrated into the official image because it can be loaded as a Monticello package. I plan to work out a way to make .mcz or .st files that use namespaces automatically load namespace support into the image.

- It's probably compatible with Göran's approach :-). You can have two namespace implementations in your image at the same time... if you really want to.

Gulik.

--
http://people.squeakfoundation.org/person/mikevdg
http://gulik.pbwiki.com/

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Another namespaces approach.

keith1y
Hi Michael,

I think your approach sounds great and I will definitely try it out.

I can help integrate this with Monticello.

I can see this potentially help in moving the kernel forward, since new
kernel module implementations
can co-exist with the old ones in the transition. I like the idea that
when no one uses the old package it can simply disappear. (bye bye
FileDirectory!)

For simple kernel class extensions and overrides, it is probably
simplest to copy the whole class into the package. Lets call this the
"copy and modify" approach.

However this could get difficult, since a package like Magma adds
serialization methods to cover a lot of classes in the system, and these
serialization methods do need access to all instVars. To support this
there would have to be some notion of "Package A which is trusted to
make extensions to Package B".... but what if I load Magma with the
intention of extending all users of String, but actually users of
My.String will miss out on those extensions. Also if you look at magma,
you will find some classes which have method contributions from
*magma-server, and contributions from *magma-client. There is a
dovetailing of packages to make a whole. Perhaps this could be solved by
having a specialized form of a class with is not a complete copy, but a
wrapper around the original, with the extension methods. I call this the
"link-to and extend" idea. Perhaps the additional methods could be
defined in the form of a deltastream.

So it looks like we need a new paradigm for class extensions. If we
could use deltastreams then we can have the idea of class extensions as
a patch queue.

I also thought that if there is a  nominated "master" repository for
each namespace, then the master repository could be updated with EVERY
users adaptation.

So if I were the owner of Kernel.Foo and other people include Kernel.Foo
in their namespaces, whenever they "copy and modify" Kernel.Foo within
their namespace, my Kernel.Foo repository gets a copy of the changed
version! That way I can make an effort to integrate any additions.  If
users "link to and extend" my Kernel.Foo, then my repository would get a
copy of the delta-stream that defines the extension.

I think that some social tools like this could help avoid the
fragmentation that Göran hopes to avoid by only having one  
non-hierarchical namespace

Keith








Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Another namespaces approach.

Igor Stasenko
On 22/09/2007, Keith Hodges <[hidden email]> wrote:

> Hi Michael,
>
> I think your approach sounds great and I will definitely try it out.
>
> I can help integrate this with Monticello.
>
> I can see this potentially help in moving the kernel forward, since new
> kernel module implementations
> can co-exist with the old ones in the transition. I like the idea that
> when no one uses the old package it can simply disappear. (bye bye
> FileDirectory!)
>
> For simple kernel class extensions and overrides, it is probably
> simplest to copy the whole class into the package. Lets call this the
> "copy and modify" approach.
>
> However this could get difficult, since a package like Magma adds
> serialization methods to cover a lot of classes in the system, and these
> serialization methods do need access to all instVars. To support this
> there would have to be some notion of "Package A which is trusted to
> make extensions to Package B".... but what if I load Magma with the
> intention of extending all users of String, but actually users of
> My.String will miss out on those extensions. Also if you look at magma,
> you will find some classes which have method contributions from
> *magma-server, and contributions from *magma-client. There is a
> dovetailing of packages to make a whole. Perhaps this could be solved by
> having a specialized form of a class with is not a complete copy, but a
> wrapper around the original, with the extension methods. I call this the
> "link-to and extend" idea. Perhaps the additional methods could be
> defined in the form of a deltastream.
>
You pointed to interesting problem.
Lets consider, how should be Magma loaded into image with namespaces enabled:
- any extensions to existing packages (like Kernel) should be done
with 'copy and modify' approach.
'Copy and modify' can be done in two ways:
- we can create a new version of base package and add new behavoirs
there. Then Magma imports these modified packages, which in same turn
importing old non-modified packages.
Like following:
Magma -> Kernel(MagmaMod) -> Kernel

Another way, we can simply copy and modify behaviors to Magma package
itself and any non-modified behaviors are still visible via imports.
So, magma will contain own version of, say Collection class, and don't
need to change imports list.

And now most interesting thing: serializing objects.
First, to make use of Magma, you need to add it to imports list in
your project package. And thus, you could see all changes to base
packages done by Magma.

Second, I don't think that we need to support serialization of every
object in image! Personally, i think its more useful and safe to
enable serialization only of those objects which used by Magma and
which you making serializeable by importing Magma package into your
own.
In fact, since only your package knowns how to work with magma (by
importing it), only this package should be granted to use
serialization.
Any attempt so serialize objects, which not belong to package(s) Magma
imports branch should be considered as erroneous (and in fact this
will lead to errors due to DNU's, because of use non-modified Kernel
behaviors which installed by Magma).

Maybe this is wrong or impossible? Time will say. But i think things
should be done in that way. I can see how this approach helping to
prevent from appearing any objects, unrelated to your project, in
Magma DB and thus could indicate design flaws.

I'm very interesting in what Michael will say. I can presume, that he
strongly against enabling image-wide serialization because its
obviously a security issue. And in Secure Squeak such things should be
impossible.

> So it looks like we need a new paradigm for class extensions. If we
> could use deltastreams then we can have the idea of class extensions as
> a patch queue.
>
> I also thought that if there is a  nominated "master" repository for
> each namespace, then the master repository could be updated with EVERY
> users adaptation.
>
> So if I were the owner of Kernel.Foo and other people include Kernel.Foo
> in their namespaces, whenever they "copy and modify" Kernel.Foo within
> their namespace, my Kernel.Foo repository gets a copy of the changed
> version! That way I can make an effort to integrate any additions.  If
> users "link to and extend" my Kernel.Foo, then my repository would get a
> copy of the delta-stream that defines the extension.
>
> I think that some social tools like this could help avoid the
> fragmentation that Göran hopes to avoid by only having one
> non-hierarchical namespace
>
> Keith
>

--
Best regards,
Igor Stasenko AKA sig.


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Another namespaces approach.

Michael van der Gulik-2


On 9/23/07, Igor Stasenko <[hidden email]> wrote:
On 22/09/2007, Keith Hodges <[hidden email]> wrote:
> Hi Michael,
>
> I think your approach sounds great and I will definitely try it out.
>
> I can help integrate this with Monticello.


I tried and gave up - Monticello seems designed with a global SystemDictionary in mind and it feels like a kludge to try to make it compatible with namespaces.

Matthew Fullmer gave me a great idea on IRC yesterday; I could manually put together .mcz files. I've already got code that files out a package in chunk format, so with a bit of extra work it's possible to make a Monticello file out of it.

<snip - Keith pointing out that my approach doesn't do methods overrides>

You pointed to interesting problem.


It's interesting that you find the biggest weakness of my approach with such speed. I really don't know how to solve this, so I'm ignoring this problem until some inspiration occurs.

The main issue is that one package should not be able to directly modify another package, for security reasons, and because you lose the guarantee that packages are deployed into the same environment they were coded in.

I've thought about having some fancy technique similar to ClassBoxes, where an object has one behaviour when appearing to one client, and another behaviour when appearing to another client, but that approach is likely to open a big can of Pandora's boxes.

You could use wrapper objects. In fact, wrappers will be a common and important pattern that you would use when writing secure code.


Lets consider, how should be Magma loaded into image with namespaces enabled:
- any extensions to existing packages (like Kernel) should be done
with 'copy and modify' approach.
'Copy and modify' can be done in two ways:
- we can create a new version of base package and add new behavoirs
there. Then Magma imports these modified packages, which in same turn
importing old non-modified packages.
Like following:
Magma -> Kernel(MagmaMod) -> Kernel


This is a possibility, but it has a big weakness. Often you want to use instances that are passed to your package from somewhere else, and these would not be instances of your overridden class.



> I also thought that if there is a  nominated "master" repository for
> each namespace, then the master repository could be updated with EVERY
> users adaptation.


This would likely happen but would be done manually by developers. 

> So if I were the owner of Kernel.Foo and other people include Kernel.Foo
> in their namespaces, whenever they "copy and modify" Kernel.Foo within
> their namespace, my Kernel.Foo repository gets a copy of the changed
> version! That way I can make an effort to integrate any additions.  If
> users "link to and extend" my Kernel.Foo, then my repository would get a
> copy of the delta-stream that defines the extension.


I'm not quite this far ahead yet. This could certainly be implemented by tools if you want. 

Gulik.


--
http://people.squeakfoundation.org/person/mikevdg
http://gulik.pbwiki.com/

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Another namespaces approach.

Lex Spoon-3
In reply to this post by Michael van der Gulik-2
"Michael van der Gulik" <[hidden email]> writes:
> - Packages depends on other packages by UUID, and if you modify a package it
> gets a new UUID. This means that your code will never suffer from dependency
> problems and will always run in exactly the same environment you wrote and
> tested it in. Bit-rot shouldn't happen unless you start changing the VM.

All dependencies work this way?

If so, then how do you handle situations where two packages depend on
different versions of some third packages?  Let's make it concrete and
say packages A and B both depend on package Morphic.  A creates a
morph, hands it to B, and then B tries to add the morph to a window.
How does this work?  Does B's Morphic really have to cope with a morph
created by A's Morphic?  That would seem to mean Morphic has to be
written very defensively, thus defeating the point of the exercise.

My gut is that loading multiple live versions should not be the
default.  By default, packages are not supposed to hum along in their
own private worlds, completely immune to anything else that is
happening in the image.  Instead, packages are supposed to interact.
They should interact with the user, with the image, and with each
other.

Yes, they should interact carefully, and they should avoid making too
many assumptions about each other.  Still, interact they should.

Now, as a non-default option, it's a different matter.  It would be
cool to be able to create a sub-image that has its own versions of
packages in it....

Lex


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Another namespaces approach.

Michael van der Gulik-2


On 23 Sep 2007 14:32:40 +0200, Lex Spoon <[hidden email]> wrote:
"Michael van der Gulik" <[hidden email]> writes:
> - Packages depends on other packages by UUID, and if you modify a package it
> gets a new UUID. This means that your code will never suffer from dependency
> problems and will always run in exactly the same environment you wrote and
> tested it in. Bit-rot shouldn't happen unless you start changing the VM.

All dependencies work this way?

If so, then how do you handle situations where two packages depend on
different versions of some third packages?  Let's make it concrete and
say packages A and B both depend on package Morphic.  A creates a
morph, hands it to B, and then B tries to add the morph to a window.
How does this work?  Does B's Morphic really have to cope with a morph
created by A's Morphic?  That would seem to mean Morphic has to be
written very defensively, thus defeating the point of the exercise.


Yes, that would break, and this is something I hadn't thought of. Thanks for bringing it up. I guess the programmer would need to be aware of this and design code accordingly.

This makes my claim about code running in exactly the same environment false. Instead, the code just runs in a very similar environment.


My gut is that loading multiple live versions should not be the
default.  By default, packages are not supposed to hum along in their
own private worlds, completely immune to anything else that is
happening in the image.  Instead, packages are supposed to interact.
They should interact with the user, with the image, and with each
other.

Yes, they should interact carefully, and they should avoid making too
many assumptions about each other.  Still, interact they should.


The interaction is the problem -- if a package and its dependencies stays the same but its environment changes, the package may become buggy. It depends on how the package interacts with and depends on its environment. In practise I'm hoping that it won't be much of an issue. There should be a core set of stable library packages that won't change much that packages can rely on.

On another note, I've also just looked into classes in the VM's special objects array. These classes cannot be modified without causing side-effects in the rest of the VM, and you can't have multiple versions of them. I might have to add the caveat that these classes can't be modified. (These classes are: "Smalltalk specialObjectsArray select: [ :each | each isKindOf: Class ].")

Gulik.

--
<a href="http://people.squeakfoundation.org/person/mikevdg" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)"> http://people.squeakfoundation.org/person/mikevdg
<a href="http://gulik.pbwiki.com/" target="_blank" onclick="return top.js.OpenExtLink(window,event,this)">http://gulik.pbwiki.com/

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Another namespaces approach.

Igor Stasenko
In reply to this post by Lex Spoon-3
On 23 Sep 2007 14:32:40 +0200, Lex Spoon <[hidden email]> wrote:

> "Michael van der Gulik" <[hidden email]> writes:
> > - Packages depends on other packages by UUID, and if you modify a package it
> > gets a new UUID. This means that your code will never suffer from dependency
> > problems and will always run in exactly the same environment you wrote and
> > tested it in. Bit-rot shouldn't happen unless you start changing the VM.
>
> All dependencies work this way?
>
> If so, then how do you handle situations where two packages depend on
> different versions of some third packages?  Let's make it concrete and
> say packages A and B both depend on package Morphic.  A creates a
> morph, hands it to B, and then B tries to add the morph to a window.
> How does this work?  Does B's Morphic really have to cope with a morph
> created by A's Morphic?  That would seem to mean Morphic has to be
> written very defensively, thus defeating the point of the exercise.
>

This is a decision which must be handled by developer. If i , lets
say, creating a package and adding imports to it, then dev tools
should warn me, that i have multiple versions of same package(s)
imported by mine.
There are two choices for developer:
- update imports of those packages, so they will use same version of
some base package. This is a good choice, btw, forcing developers to
sync versions of different packages and update code to work fine with
latest updates.
- second choice is to keep things intact, then it's a developer's
responsibility to write code in his package in such way, that there
will no conflicts occur, like using instances of classes, created by
package A, but thinking that these instances have behavior denoted in
package B.

> My gut is that loading multiple live versions should not be the
> default.  By default, packages are not supposed to hum along in their
> own private worlds, completely immune to anything else that is
> happening in the image.  Instead, packages are supposed to interact.
> They should interact with the user, with the image, and with each
> other.
>
> Yes, they should interact carefully, and they should avoid making too
> many assumptions about each other.  Still, interact they should.
>
> Now, as a non-default option, it's a different matter.  It would be
> cool to be able to create a sub-image that has its own versions of
> packages in it....
>
> Lex
>
>
>


--
Best regards,
Igor Stasenko AKA sig.