strange array instance

Hi, 

there is a strange array of size 1024 in the Pharo image that is not eaten by garbage collector even if no object points to it. I guess it is related to Spur.

((Array allInstances select: [ :i | i size = 1024 ])

  select: [ :a | a second = SmallInteger  ]) pointersTo

What is it and is it accessible somehow?

Cheers,

-- Pavel

Hi Pavel,

On Tue, Mar 8, 2016 at 1:40 AM, Pavel Krivanek <[hidden email]> wrote:

Hi, 

there is a strange array of size 1024 in the Pharo image that is not eaten by garbage collector even if no object points to it. I guess it is related to Spur.

((Array allInstances select: [ :i | i size = 1024 ])

  select: [ :a | a second = SmallInteger  ]) pointersTo

What is it and is it accessible somehow?

It is indeed the first class table page.  It looks like an old bug.  The oldest Spur image I can start up has the problem.   The way that class table pages are hidden is that they have a class index which is not that of Array.  Array is at index 52 in the class table (51 zero-relative, Array identityHash = 51), but it is also at index 17 (16 zero-relative).  Array allInstances looks for objects whose class index is 51, hence not seeing the class table pages, which all (but the first) have class index 16.

Luckily we can fix the problem easily, and we can fix another problem at the same time, which is that SmallFloat64 has the wrong hash.

The class table is two-level; a (hidden) array of Arrays.  Classes are in the class table at their identityHashes (so that to instantiate a class one copies the class's identity hash into the classIndex of the new instance instead of having to search the class table).  But when SmallFloat64 was created in the 32-bit image I forgot to give it the right hash and store it in the class table.  (there are no SmallFloat64 instances in a 32-bit image, only in 64-bits).

So we can first change SmallFloat64's hash to 3, which is what the Spur 64-bit VM and image require, and enter it into the first class table page, and then we can make the first class table page disappear.

| firstClassTablePage clone |

"The first class table page is the first Array instance. Of course this is a bug."

firstClassTablePage := Array someInstance.

"It should contain only nil and classes."

self assert: (firstClassTablePage allSatisfy: [:e| e == nil or: [e isBehavior]]).

"And its first 17 elements should only be the immediate classes and Array, since Array is used as the class pun for class table pages, with index 17 (16 zero-relative)"

self assert: (firstClassTablePage first: 17) asSet = {nil. SmallInteger. Character. Array} asSet.

"It just so happens that the second Array is the specialSelectors"

self assert: Array someInstance nextInstance == Smalltalk specialSelectors.

"Store SmallFloat64 in the first class table page at index 4 (3 zero-relative)."

firstClassTablePage at: 4 put: SmallFloat64.

"Use the secret set identity hash primitive to set SmallFloat64's identityHash to 3."

SmallFloat64 tryPrimitive: 161 withArgs: #(3).

"Now create an object that looks like Array class, but has identityHash 16."

"Take a shallow copy of Array class."

clone := Array shallowCopy.

"Change it into an Array."

Array adoptInstance: clone.

"Set its identityHash to 16."

clone tryPrimitive: 161 withArgs: #(16).

"Use the adoptInstance: primitive to ``set the class of the firstClassTablePage to the cone''.

 or, more accurately, to set the firstClassTablePage's class index to 16."

clone tryPrimitive: 160 withArgs: {firstClassTablePage}

With the above done, we can check that everything is ok."

self assert: SmallFloat64 identityHash = 3.

self assert: Array someInstance == Smalltalk specialSelectors.

"A class table page is 1024 slots, contains only nil or behaviours, and contains at least one behaviour (is not all nils).  There shouldn't be any that we can find."

self assert: (self systemNavigation allObjects select: [:o| o isArray and: [o size = 1024 and: [(o allSatisfy: [:e| e == nil or: [e isBehavior]]) and: [o anySatisfy: [:e| e isBehavior]]]]]) size = 0

I recommend you create an update map.  Then create a version of kernel with a post-load action, written something like this:

(Array someInstance size = 1014

 and: [(Array someInstance allSatisfy: [:e| e == nil or: [e isBehavior]])

 and: [(firstClassTablePage first: 17) asSet = {nil. SmallInteger. Character. Array} asSet]]) ifTrue:

[| firstClassTablePage clone |

firstClassTablePage := Array someInstance.

self assert: (firstClassTablePage allSatisfy: [:e| e == nil or: [e isBehavior]]).

self assert: (firstClassTablePage first: 17) asSet = {nil. SmallInteger. Character. Array} asSet.

firstClassTablePage at: 4 put: SmallFloat64.

SmallFloat64 tryPrimitive: 161 withArgs: #(3).

clone := Array shallowCopy.

Array adoptInstance: clone.

clone tryPrimitive: 161 withArgs: #(16).

clone tryPrimitive: 160 withArgs: {Array someInstance}

self assert: SmallFloat64 identityHash = 3.

self assert: (Array someInstance first: 4) = {nil. SmallInteger. Character. SmallFloat64}]

Then create a second update map to ensure that that version of Kernel is loaded.

I will do this for Squeak immediately.

Apologies.

P.S. Interestingly enough, it shows what a horrible security hole the debugger primitives tryPrimitive:withArgs: and tryNamedPrimitive:withArgs: are.

Cheers,

-- Pavel

_,,,^..^,,,_

best, Eliot

Hi Pavel,

On Tue, Mar 8, 2016 at 1:40 AM, Pavel Krivanek <[hidden email]> wrote:

Hi, 

there is a strange array of size 1024 in the Pharo image that is not eaten by garbage collector even if no object points to it. I guess it is related to Spur.

((Array allInstances select: [ :i | i size = 1024 ])

  select: [ :a | a second = SmallInteger  ]) pointersTo

What is it and is it accessible somehow?

It is indeed the first class table page.  It looks like an old bug.  The oldest Spur image I can start up has the problem.   The way that class table pages are hidden is that they have a class index which is not that of Array.  Array is at index 52 in the class table (51 zero-relative, Array identityHash = 51), but it is also at index 17 (16 zero-relative).  Array allInstances looks for objects whose class index is 51, hence not seeing the class table pages, which all (but the first) have class index 16.

Luckily we can fix the problem easily, and we can fix another problem at the same time, which is that SmallFloat64 has the wrong hash.

The class table is two-level; a (hidden) array of Arrays.  Classes are in the class table at their identityHashes (so that to instantiate a class one copies the class's identity hash into the classIndex of the new instance instead of having to search the class table).  But when SmallFloat64 was created in the 32-bit image I forgot to give it the right hash and store it in the class table.  (there are no SmallFloat64 instances in a 32-bit image, only in 64-bits).

So we can first change SmallFloat64's hash to 3, which is what the Spur 64-bit VM and image require, and enter it into the first class table page, and then we can make the first class table page disappear.

| firstClassTablePage clone |

"The first class table page is the first Array instance. Of course this is a bug."

firstClassTablePage := Array someInstance.

"It should contain only nil and classes."

self assert: (firstClassTablePage allSatisfy: [:e| e == nil or: [e isBehavior]]).

"And its first 17 elements should only be the immediate classes and Array, since Array is used as the class pun for class table pages, with index 17 (16 zero-relative)"

self assert: (firstClassTablePage first: 17) asSet = {nil. SmallInteger. Character. Array} asSet.

"It just so happens that the second Array is the specialSelectors"

self assert: Array someInstance nextInstance == Smalltalk specialSelectors.

"Store SmallFloat64 in the first class table page at index 4 (3 zero-relative)."

firstClassTablePage at: 4 put: SmallFloat64.

"Use the secret set identity hash primitive to set SmallFloat64's identityHash to 3."

SmallFloat64 tryPrimitive: 161 withArgs: #(3).

"Now create an object that looks like Array class, but has identityHash 16."

"Take a shallow copy of Array class."

clone := Array shallowCopy.

"Change it into an Array."

Array adoptInstance: clone.

"Set its identityHash to 16."

clone tryPrimitive: 161 withArgs: #(16).

"Use the adoptInstance: primitive to ``set the class of the firstClassTablePage to the cone''.

 or, more accurately, to set the firstClassTablePage's class index to 16."

clone tryPrimitive: 160 withArgs: {firstClassTablePage}

With the above done, we can check that everything is ok."

self assert: SmallFloat64 identityHash = 3.

self assert: Array someInstance == Smalltalk specialSelectors.

"A class table page is 1024 slots, contains only nil or behaviours, and contains at least one behaviour (is not all nils).  There shouldn't be any that we can find."

self assert: (self systemNavigation allObjects select: [:o| o isArray and: [o size = 1024 and: [(o allSatisfy: [:e| e == nil or: [e isBehavior]]) and: [o anySatisfy: [:e| e isBehavior]]]]]) size = 0

I recommend you create an update map.  Then create a version of kernel with a post-load action, written something like this:

(Array someInstance size = 1014

 and: [(Array someInstance allSatisfy: [:e| e == nil or: [e isBehavior]])

 and: [(firstClassTablePage first: 17) asSet = {nil. SmallInteger. Character. Array} asSet]]) ifTrue:

[| firstClassTablePage clone |

firstClassTablePage := Array someInstance.

self assert: (firstClassTablePage allSatisfy: [:e| e == nil or: [e isBehavior]]).

self assert: (firstClassTablePage first: 17) asSet = {nil. SmallInteger. Character. Array} asSet.

firstClassTablePage at: 4 put: SmallFloat64.

SmallFloat64 tryPrimitive: 161 withArgs: #(3).

clone := Array shallowCopy.

Array adoptInstance: clone.

clone tryPrimitive: 161 withArgs: #(16).

clone tryPrimitive: 160 withArgs: {Array someInstance}

self assert: SmallFloat64 identityHash = 3.

self assert: (Array someInstance first: 4) = {nil. SmallInteger. Character. SmallFloat64}]

Then create a second update map to ensure that that version of Kernel is loaded.

I will do this for Squeak immediately.

Apologies.

P.S. Interestingly enough, it shows what a horrible security hole the debugger primitives tryPrimitive:withArgs: and tryNamedPrimitive:withArgs: are.

Cheers,

-- Pavel

_,,,^..^,,,_

best, Eliot