Eager Import

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Eager Import

Colin Putney-3
Hi folks,

I've just uploaded to the Inbox a series of packages that implement a new way of handling export and import of bindings between packages. The protocol for working with environments remains the same, but internally, it's quite different. 

The idea here is to make imports eager, rather than lazy. Instead of the "references" dictionary, which was a cache of the bindings that had been imported during method compilation, an environment now has a "bindings" dictionary, containing all bindings that are available during compilation. This is much easier to understand, and more similar to the old SystemDictionary approach, which should result in better compatibility. Environments now inform each other when bindings are added and removed, so that the bindings dictionary will always be up to date.

Also, following Nicolas Cellier's ideathe system of imports and exports used to configure an environment is simpler and more "declarative" (to the extent that Smalltalk could ever be considered declarative!). This should make tool support easier to implement. 

To try it out, load the packages into a trunk image in this order:

Environments-cwp.41
Environments-cwp.42
ShoutCore-cwp.40
Environments-cwp.43
Tests-cwp.281

In my image, this passes all the Environments tests, and appears to clean up the bindings in the image nicely. This DoIt:

classes := Bag new.
CompiledMethod allInstancesDo:
[:method |
method allLiterals do:
[:literal |
classes add: literal class]].
classes sortedCounts.

answers this:

234856->ByteSymbol
58608->ClassBinding
17788->ByteString
12419->SmallInteger
10495->Association
3422->Array
2350->Float
2328->Global
1853->Character
863->AdditionalMethodState
207->LargePositiveInteger
61->LargeNegativeInteger
50->ByteArray
26->ScaledDecimal
4->WideString
2->Fraction

We have lots of ClassBindings, and some Globals, but no Aliases. AFAICT, the associations in there are class variables. We should probably convert those to Globals at some point. 

Colin


Reply | Threaded
Open this post in threaded view
|

Re: Eager Import

Frank Shearar-3
Thanks, Colin!

On 1 January 2014 19:15, Colin Putney <[hidden email]> wrote:

> Hi folks,
>
> I've just uploaded to the Inbox a series of packages that implement a new
> way of handling export and import of bindings between packages. The protocol
> for working with environments remains the same, but internally, it's quite
> different.
>
> The idea here is to make imports eager, rather than lazy. Instead of the
> "references" dictionary, which was a cache of the bindings that had been
> imported during method compilation, an environment now has a "bindings"
> dictionary, containing all bindings that are available during compilation.
> This is much easier to understand, and more similar to the old
> SystemDictionary approach, which should result in better compatibility.
> Environments now inform each other when bindings are added and removed, so
> that the bindings dictionary will always be up to date.
>
> Also, following Nicolas Cellier's idea, the system of imports and exports
> used to configure an environment is simpler and more "declarative" (to the
> extent that Smalltalk could ever be considered declarative!). This should
> make tool support easier to implement.
>
> To try it out, load the packages into a trunk image in this order:
>
> Environments-cwp.41
> Environments-cwp.42
> ShoutCore-cwp.40

Don't forget this one :)

> Environments-cwp.43
> Tests-cwp.281
>
> In my image, this passes all the Environments tests, and appears to clean up
> the bindings in the image nicely. This DoIt:
>
> classes := Bag new.
> CompiledMethod allInstancesDo:
> [:method |
> method allLiterals do:
> [:literal |
> classes add: literal class]].
> classes sortedCounts.
>
> answers this:
>
> 234856->ByteSymbol
> 58608->ClassBinding
> 17788->ByteString
> 12419->SmallInteger
> 10495->Association
> 3422->Array
> 2350->Float
> 2328->Global
> 1853->Character
> 863->AdditionalMethodState
> 207->LargePositiveInteger
> 61->LargeNegativeInteger
> 50->ByteArray
> 26->ScaledDecimal
> 4->WideString
> 2->Fraction
>
> We have lots of ClassBindings, and some Globals, but no Aliases. AFAICT, the
> associations in there are class variables. We should probably convert those
> to Globals at some point.

Why would we convert class variables to Globals? Do you mean we'd
represent the bindings of class variables as Globals (because the code
referencing the class variable has no way of knowing whether it's
referencing a class variable or a Really True global? If so, OK. I
guess "Global" might need a different name then. It's really
SomethingThatsNotAClassOrTraitButAvailableToMe.

frank

> Colin

Reply | Threaded
Open this post in threaded view
|

Re: Eager Import

Colin Putney-3



On Wed, Jan 1, 2014 at 3:59 PM, Frank Shearar <[hidden email]> wrote:
 
Why would we convert class variables to Globals? Do you mean we'd
represent the bindings of class variables as Globals (because the code
referencing the class variable has no way of knowing whether it's
referencing a class variable or a Really True global? If so, OK.

Yeah, I just mean use Global instance for the bindings instead of Associations. 
 
I guess "Global" might need a different name then. It's really
SomethingThatsNotAClassOrTraitButAvailableToMe.

It's true the Global isn't strictly accurate, but that's true of bindings in an environment too. The effect of Environments is that nothing is global any more. VisualWorks uses the term SharedVariable for everything that's accessed via a binding (i.e., all variables that are not  temps or instance variables). That's accurate, I guess, but kind of awkward. 

Colin



Reply | Threaded
Open this post in threaded view
|

Re: Eager Import

Levente Uzonyi-2
In reply to this post by Colin Putney-3
I took a quick look, and found two issues so far.

Environment >> #bindingOf:ifAbsent: returns bindings from undeclared:

bindingOf: aSymbol ifAbsent: aBlock
  ^ bindings bindingOf: aSymbol ifAbsent:
  [undeclared bindingOf: aSymbol ifAbsent: aBlock]

This fools Shout and the rest of the system, making them "think" that
any missing instance variable which has a method referencing it exists.
A good example is Environment's exports variable, which was deleted, but
the #exports accessor is still in the image, and it's referencing it. So I
can compile any method using the exports variable, and neither Shout, nor
the compiler will complain. This is the reason why ReleaseTest >>
#testUndeclared is still failing.

Is there a reason to return bindings from Undeclared from
#bindingOf:ifAbsent:?


The other issue is related to the handling of Undeclared. Dictionary >>
#unreferencedKeys - which is used during #cleanOutUndeclared - ignores the
current Environment and uses the classes of the global environment.
The Environment itself should provide a method to remove the
unreferenced bindings from its own Undeclared.


Levente

On Wed, 1 Jan 2014, Colin Putney wrote:

> Hi folks,
> I've just uploaded to the Inbox a series of packages that implement a new way of handling export and import of bindings between packages. The protocol for working with environments remains the same, but
> internally, it's quite different. 
>
> The idea here is to make imports eager, rather than lazy. Instead of the "references" dictionary, which was a cache of the bindings that had been imported during method compilation, an environment now has a
> "bindings" dictionary, containing all bindings that are available during compilation. This is much easier to understand, and more similar to the old SystemDictionary approach, which should result in better
> compatibility. Environments now inform each other when bindings are added and removed, so that the bindings dictionary will always be up to date.
>
> Also, following Nicolas Cellier's idea, the system of imports and exports used to configure an environment is simpler and more "declarative" (to the extent that Smalltalk could ever be considered
> declarative!). This should make tool support easier to implement. 
>
> To try it out, load the packages into a trunk image in this order:
>
> Environments-cwp.41
> Environments-cwp.42
> ShoutCore-cwp.40
> Environments-cwp.43
> Tests-cwp.281
>
> In my image, this passes all the Environments tests, and appears to clean up the bindings in the image nicely. This DoIt:
>
> classes := Bag new.
> CompiledMethod allInstancesDo:
> [:method |
> method allLiterals do:
> [:literal |
> classes add: literal class]].
> classes sortedCounts.
>
> answers this:
>
> 234856->ByteSymbol
> 58608->ClassBinding
> 17788->ByteString
> 12419->SmallInteger
> 10495->Association
> 3422->Array
> 2350->Float
> 2328->Global
> 1853->Character
> 863->AdditionalMethodState
> 207->LargePositiveInteger
> 61->LargeNegativeInteger
> 50->ByteArray
> 26->ScaledDecimal
> 4->WideString
> 2->Fraction
>
> We have lots of ClassBindings, and some Globals, but no Aliases. AFAICT, the associations in there are class variables. We should probably convert those to Globals at some point. 
>
> Colin
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Eager Import

Colin Putney-3



On Wed, Jan 1, 2014 at 5:15 PM, Levente Uzonyi <[hidden email]> wrote:
I took a quick look, and found two issues so far.

Environment >> #bindingOf:ifAbsent: returns bindings from undeclared:

bindingOf: aSymbol ifAbsent: aBlock
        ^ bindings bindingOf: aSymbol ifAbsent:
                [undeclared bindingOf: aSymbol ifAbsent: aBlock]

This fools Shout and the rest of the system, making them "think" that any missing instance variable which has a method referencing it exists.
A good example is Environment's exports variable, which was deleted, but the #exports accessor is still in the image, and it's referencing it. So I can compile any method using the exports variable, and neither Shout, nor the compiler will complain. This is the reason why ReleaseTest >> #testUndeclared is still failing.


Argh. I keep making this mistake, because that's obviously how it should work. Of course, despite being obvious, this intuition is just wrong. I've added some tests that should catch this sort of mistake in the future. 
 
Is there a reason to return bindings from Undeclared from #bindingOf:ifAbsent:?


The other issue is related to the handling of Undeclared. Dictionary >> #unreferencedKeys - which is used during #cleanOutUndeclared - ignores the current Environment and uses the classes of the global environment.
The Environment itself should provide a method to remove the unreferenced bindings from its own Undeclared.

It does—#purgeUndeclared. We just need to send that message to the right environment. I suspect that senders of #cleanOutUndeclared should deprecated, and all the existing senders should be rewritten to send #purgeUndeclared to the appropriate environment.