Root bit on active context? Closure VMs and thoughts on flushExternalPrimitives.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Root bit on active context? Closure VMs and thoughts on flushExternalPrimitives.

johnmci
 
Ok, I'm confused about the comments in initialCleanup, since in  
testing with a Pharo image from feb it DOES call the  
flushExternalPrimitives, but the comments imply a 3.8 or later image  
shouldn't have the root bit set.

Also then the flushExternalPrimitiveOf: seems to be invoked by the  
snapshotCleanUp.

So is the comment in initialCleanup wrong?

Could in the Closure VM could we skip flushExternalPrimitives for the  
new image type, since we *know* any closure image would have been  
processed by snapshotCleanUp, versus being a 3.2 image?



initialCleanup
        "Images written by VMs earlier than 3.6/3.7 will wrongly have the  
root bit set on the active context. Besides clearing the root bit, we  
treat this as a marker that these images also lack a cleanup of  
external primitives (which has been introduced at the same time when  
the root bit problem was fixed). In this case, we merely flush them  
from here."

        ((self longAt: activeContext) bitAnd: RootBit) = 0 ifTrue:[^nil].  
"root bit is clean"
        "Clean root bit of activeContext"
        self longAt: activeContext put: ((self longAt: activeContext) bitAnd:  
AllButRootBit).
        "Clean external primitives"
        self flushExternalPrimitives.


then


flushExternalPrimitives
        "Flush the references to external functions from plugin
        primitives. This will force a reload of those primitives when
        accessed next.
        Note: We must flush the method cache here so that any
        failed primitives are looked up again."
        | oop primIdx |
        oop := self firstObject.
        [self oop: oop isLessThan: endOfMemory]
                whileTrue: [(self isFreeObject: oop)
                                ifFalse: [(self isCompiledMethod: oop)
                                                ifTrue: ["This is a compiled method"
                                                        primIdx := self primitiveIndexOf: oop.
                                                        primIdx = PrimitiveExternalCallIndex
                                                                ifTrue: ["It's primitiveExternalCall"
                                                                        self flushExternalPrimitiveOf: oop]]].
                        oop := self objectAfter: oop].
        self flushMethodCache.
        self flushObsoleteIndexedPrimitives.
        self flushExternalPrimitiveTable


snapshotCleanUp
        "Clean up right before saving an image, sweeping memory and:
        * nilling out all fields of contexts above the stack pointer.
        * flushing external primitives
        * clearing the root bit of any object in the root table "
        | oop header fmt sz |
        oop := self firstObject.
        [self oop: oop isLessThan: endOfMemory]
                whileTrue: [(self isFreeObject: oop)
                                ifFalse: [header := self longAt: oop.
                                        fmt := header >> 8 bitAnd: 15.
                                        "Clean out context"
                                        (fmt = 3 and: [self isContextHeader: header])
                                                ifTrue: [sz := self sizeBitsOf: oop.
                                                        (self lastPointerOf: oop) + BytesPerWord
                                                                to: sz - BaseHeaderSize by: BytesPerWord
                                                                do: [:i | self longAt: oop + i put: nilObj]].
                                        "Clean out external functions"
                                        fmt >= 12
                                                ifTrue: ["This is a compiled method"
                                                        (self primitiveIndexOf: oop) = PrimitiveExternalCallIndex
                                                                ifTrue: ["It's primitiveExternalCall"
                                                                        self flushExternalPrimitiveOf: oop]]].
                        oop := self objectAfter: oop].
        self clearRootsTable


--
=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================



Reply | Threaded
Open this post in threaded view
|

Re: Root bit on active context? Closure VMs and thoughts on flushExternalPrimitives.

Eliot Miranda-2
 


On Sat, Apr 11, 2009 at 11:55 PM, John M McIntosh <[hidden email]> wrote:

Ok, I'm confused about the comments in initialCleanup, since in testing with a Pharo image from feb it DOES call the flushExternalPrimitives, but the comments imply a 3.8 or later image shouldn't have the root bit set.

Also then the flushExternalPrimitiveOf: seems to be invoked by the snapshotCleanUp.

So is the comment in initialCleanup wrong?

Could in the Closure VM could we skip flushExternalPrimitives for the new image type, since we *know* any closure image would have been processed by snapshotCleanUp, versus being a 3.2 image?

Good point.  So initialCleanup would then become

flushExternalPrimitives
self flushMethodCache.
self flushAtCache.
self flushExternalPrimitiveTable
 
initialCleanup
       "Images written by VMs earlier than 3.6/3.7 will wrongly have the root bit set on the active context. Besides clearing the root bit, we treat this as a marker that these images also lack a cleanup of external primitives (which has been introduced at the same time when the root bit problem was fixed). In this case, we merely flush them from here."

       ((self longAt: activeContext) bitAnd: RootBit) = 0 ifTrue:[^nil]. "root bit is clean"
       "Clean root bit of activeContext"
       self longAt: activeContext put: ((self longAt: activeContext) bitAnd: AllButRootBit).
       "Clean external primitives"
       self flushExternalPrimitives.


then


flushExternalPrimitives
       "Flush the references to external functions from plugin
       primitives. This will force a reload of those primitives when
       accessed next.
       Note: We must flush the method cache here so that any
       failed primitives are looked up again."
       | oop primIdx |
       oop := self firstObject.
       [self oop: oop isLessThan: endOfMemory]
               whileTrue: [(self isFreeObject: oop)
                               ifFalse: [(self isCompiledMethod: oop)
                                               ifTrue: ["This is a compiled method"
                                                       primIdx := self primitiveIndexOf: oop.
                                                       primIdx = PrimitiveExternalCallIndex
                                                               ifTrue: ["It's primitiveExternalCall"
                                                                       self flushExternalPrimitiveOf: oop]]].
                       oop := self objectAfter: oop].
       self flushMethodCache.
       self flushObsoleteIndexedPrimitives.
       self flushExternalPrimitiveTable


snapshotCleanUp
       "Clean up right before saving an image, sweeping memory and:
       * nilling out all fields of contexts above the stack pointer.
       * flushing external primitives
       * clearing the root bit of any object in the root table "
       | oop header fmt sz |
       oop := self firstObject.
       [self oop: oop isLessThan: endOfMemory]
               whileTrue: [(self isFreeObject: oop)
                               ifFalse: [header := self longAt: oop.
                                       fmt := header >> 8 bitAnd: 15.
                                       "Clean out context"
                                       (fmt = 3 and: [self isContextHeader: header])
                                               ifTrue: [sz := self sizeBitsOf: oop.
                                                       (self lastPointerOf: oop) + BytesPerWord
                                                               to: sz - BaseHeaderSize by: BytesPerWord
                                                               do: [:i | self longAt: oop + i put: nilObj]].
                                       "Clean out external functions"
                                       fmt >= 12
                                               ifTrue: ["This is a compiled method"
                                                       (self primitiveIndexOf: oop) = PrimitiveExternalCallIndex
                                                               ifTrue: ["It's primitiveExternalCall"
                                                                       self flushExternalPrimitiveOf: oop]]].
                       oop := self objectAfter: oop].
       self clearRootsTable


--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================




Reply | Threaded
Open this post in threaded view
|

Re: Root bit on active context? Closure VMs and thoughts on flushExternalPrimitives.

Eliot Miranda-2
 


On Mon, Apr 13, 2009 at 2:08 PM, Eliot Miranda <[hidden email]> wrote:


On Sat, Apr 11, 2009 at 11:55 PM, John M McIntosh <[hidden email]> wrote:

Ok, I'm confused about the comments in initialCleanup, since in testing with a Pharo image from feb it DOES call the flushExternalPrimitives, but the comments imply a 3.8 or later image shouldn't have the root bit set.

Also then the flushExternalPrimitiveOf: seems to be invoked by the snapshotCleanUp.

So is the comment in initialCleanup wrong?

Could in the Closure VM could we skip flushExternalPrimitives for the new image type, since we *know* any closure image would have been processed by snapshotCleanUp, versus being a 3.2 image?

Good point.  So initialCleanup would then become

flushExternalPrimitives
self flushMethodCache.
self flushAtCache.
self flushExternalPrimitiveTable

Oops; copy paste error.  I meant of course 

initialCleanup
self flushMethodCache.
self flushAtCache.
self flushExternalPrimitiveTable
 
 
initialCleanup
       "Images written by VMs earlier than 3.6/3.7 will wrongly have the root bit set on the active context. Besides clearing the root bit, we treat this as a marker that these images also lack a cleanup of external primitives (which has been introduced at the same time when the root bit problem was fixed). In this case, we merely flush them from here."

       ((self longAt: activeContext) bitAnd: RootBit) = 0 ifTrue:[^nil]. "root bit is clean"
       "Clean root bit of activeContext"
       self longAt: activeContext put: ((self longAt: activeContext) bitAnd: AllButRootBit).
       "Clean external primitives"
       self flushExternalPrimitives.


then


flushExternalPrimitives
       "Flush the references to external functions from plugin
       primitives. This will force a reload of those primitives when
       accessed next.
       Note: We must flush the method cache here so that any
       failed primitives are looked up again."
       | oop primIdx |
       oop := self firstObject.
       [self oop: oop isLessThan: endOfMemory]
               whileTrue: [(self isFreeObject: oop)
                               ifFalse: [(self isCompiledMethod: oop)
                                               ifTrue: ["This is a compiled method"
                                                       primIdx := self primitiveIndexOf: oop.
                                                       primIdx = PrimitiveExternalCallIndex
                                                               ifTrue: ["It's primitiveExternalCall"
                                                                       self flushExternalPrimitiveOf: oop]]].
                       oop := self objectAfter: oop].
       self flushMethodCache.
       self flushObsoleteIndexedPrimitives.
       self flushExternalPrimitiveTable


snapshotCleanUp
       "Clean up right before saving an image, sweeping memory and:
       * nilling out all fields of contexts above the stack pointer.
       * flushing external primitives
       * clearing the root bit of any object in the root table "
       | oop header fmt sz |
       oop := self firstObject.
       [self oop: oop isLessThan: endOfMemory]
               whileTrue: [(self isFreeObject: oop)
                               ifFalse: [header := self longAt: oop.
                                       fmt := header >> 8 bitAnd: 15.
                                       "Clean out context"
                                       (fmt = 3 and: [self isContextHeader: header])
                                               ifTrue: [sz := self sizeBitsOf: oop.
                                                       (self lastPointerOf: oop) + BytesPerWord
                                                               to: sz - BaseHeaderSize by: BytesPerWord
                                                               do: [:i | self longAt: oop + i put: nilObj]].
                                       "Clean out external functions"
                                       fmt >= 12
                                               ifTrue: ["This is a compiled method"
                                                       (self primitiveIndexOf: oop) = PrimitiveExternalCallIndex
                                                               ifTrue: ["It's primitiveExternalCall"
                                                                       self flushExternalPrimitiveOf: oop]]].
                       oop := self objectAfter: oop].
       self clearRootsTable


--
===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================





Reply | Threaded
Open this post in threaded view
|

Re: Root bit on active context? Closure VMs and thoughts on flushExternalPrimitives.

johnmci
 
Er I think my question was why is

        ((self longAt: activeContext) bitAnd: RootBit) = 0 ifTrue:[

always false if the comment implies 1 3.8 shouldn't  have the RootBit  
set?

and

flushExternalPrimitives then grinds thru all the objects to do self  
flushExternalPrimitiveOf: oop
but snapshotCleanUp  does the same work (or so it appears), which  
would mean the flushExternalPrimitiveOf: would be pointless at startup  
time.

On 13-Apr-09, at 2:09 PM, Eliot Miranda wrote:

> --

=
=
=
========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
=
=
=
========================================================================



Reply | Threaded
Open this post in threaded view
|

Re: Root bit on active context? Closure VMs and thoughts on flushExternalPrimitives.

Eliot Miranda-2
 


On Mon, Apr 13, 2009 at 2:22 PM, John M McIntosh <[hidden email]> wrote:
Er I think my question was why is


      ((self longAt: activeContext) bitAnd: RootBit) = 0 ifTrue:[

always false if the comment implies 1 3.8 shouldn't  have the RootBit set?

Since the initial context shouldn't have the root bit set all the context manipulation in initialClanup is redundant if this is a new closure image.  If it isn't you need to keep it for loading older images.  So in our sources the root bit code is still there in the ordinary closure interpreter, but gone in the Stack VM (which only runs closure images).


and

flushExternalPrimitives then grinds thru all the objects to do self flushExternalPrimitiveOf: oop
but snapshotCleanUp  does the same work (or so it appears), which would mean the flushExternalPrimitiveOf: would be pointless at startup time.

Right, which was why I said "Good point".  In the Stack VM therefore initialCleanup is just

initialCleanup
"This used to cope with issues with images written by VMs earlier than 3.6/3.7.
Since we won't be loading such images (being a closure only VM) we only have to
deal with external primitives.  Since references to external plugins in methods are
cleaned up in snapshotCleanUp only initialize the tables, not visit each method."
self flushMethodCache.
self flushAtCache.
self flushExternalPrimitiveTable



On 13-Apr-09, at 2:09 PM, Eliot Miranda wrote:

--

===========================================================================
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================