Smalltalk › Squeak › Squeak VM

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

10 messages Options

Eliot Miranda-2

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

On Thu, Sep 22, 2011 at 11:06 AM, Eliot Miranda <[hidden email]> wrote:

Hi Igor,

On Thu, Sep 22, 2011 at 10:53 AM, Igor Stasenko <[hidden email]> wrote:

On 22 September 2011 19:16, Eliot Miranda <[hidden email]> wrote:
> (apologies for the duplicate reply; someone needs to sort out their
> threading for the benefit of the community ;) )
>
> On Thu, Sep 22, 2011 at 2:36 AM, Marcus Denker <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> There are two changesets waiting for integrating in 1.4 that have serious
>> consequences:
>>
>> - Ephemerons. The VM level changes are in the Cog VMs build on Jenkins,
>> but have not
>> been integrated in the VMMaker codebase.
>>
>> http://code.google.com/p/pharo/issues/detail?id=4265
>
> I would *really* like to back out these changes. The Ephemeron
> implementation is very much a prototype, requiring a hack to determine
> whether an object is an ephemeron (the presence of a marker class in the
> first inst var) that I'm not at all happy with. There is a neater
> implementation available via using an unused instSpec which IMO has
> significant advantages (much simpler & faster, instSpec is valid at all
> times, including during compaction, less overhead, doesn't require a marker
> class), and is the route I'm taking with the new GC/object-representation
> I'm working on now. Note that other than determining whether an object is
> an ephemeron (instSpec/format vs inst var test) the rest of Igor's code
> remains the same. I'd like to avoid too much VM forking. Would you all
> consider putting these changes on hold for now?
> If so, I'll make the effort to produce prototype changes (in the area of
> ClassBuilder and class definition; no VM code necessary as yet) to allow
> defining Ephemerons via the int spec route by next week at the latest.
>

i agree that in my implementation this is a weak point. But its hard
to do anything without
making changes to object format to identify these special objects.

The main story behind this is can we afford to change the internals of
VM without being beaten hard
by "backwards compatibility" party? :)

I don't think we get stuck in this at all. The instSpec/format field has an unused value (5 i believe) and this can easily be used for Ephemerons. All that is needed is a little image work on these methods:

Behavior>>typeOfClass
needs to answer e.g. #ephemeron for ephemeron classes

ClassBuilder>>computeFormat:instSize:forSuper:ccIndex:
needs to accept e.g. #ephemeron for type and pass variable: false and weak: true for ephemerons to format:variable:words:pointers:weak:.

ClassBuilder>>format:variable:words:pointers:weak:

needs to respond to variable: false and weak: true by computing the ephemeron instSpec.

Class>>weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

ClassBuilder>>superclass:weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:
need siblings, e.g.
ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category

superclass:ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

Right? This is easy. Then in the VM there are a few places where pointer indexability (formats 3 and 4) need to be firmed up to exclude 5, but nothing difficult. We talked about this in email last week.

Here's the format field (Behavior>instSpec at the image level) as currently populated:

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

6 = 32-bit indexable objects (Float, Bitmap ert al)

8 = 8-bit indexable objects (ByteString, ByteArray et al)

12 = CompiledMethod

N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.

For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)

and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.

9 (?) 64-bit indexable

10 - 11 32-bit indexable

12 - 15 16-bit indexable

16 - 23 byte indexable

24 - 31 compiled method

In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.

Ephemerons are versatile way to get notifications of objects which are
about to die,
and there are certain parts in language which is hard (or even
impossible) to implement without ephemerons.

I got stuck with it earlier, when realized that we cannot afford to
have weak subscriptions in announcement framework
for blocks (which is most convenient and most easy way to define
subscriptions) without having ephemerons.
And of course, by having ephemerons we can completely review the weak
finalization scheme and make it
much simpler, and faster.

I think we should do something in this regard, even at cost of
backward compatibility.
Because as to me it blocks us from moving forward.

I wanted to remind to people, that it took me around a day to
implement ephemerons in VM. And then few more days
to actually make a correct implementation and write tests to cover it.

Unfortunately, we yet don't have a well established process, which
could make VM + language side changes to go in sync,
when its required, and go much faster and don't fear to
introduce/change functionality.
One of the reasons for having a continuous integration setup for VM
was exactly for that:
having new VMs every day (comparing to having new VMs every year).

>>
>> - Finalization code checks for #hasNewFinalization
>> This is true in the current VMs build in Jenkins, but in older VMs this
>> is not in.
>>
>> http://code.google.com/p/pharo/issues/detail?id=4483
>>
>> There are two options:
>>
>> a) integrate in
>> b) not integrate it
>>
>> a) means that the image runs on older VMs, too.
>> b) means we accept that we can never improve anything for real.
>>
>> There will be more changes coming... e.g. imagine we have a Vector
>> Graphics Canvas
>> as some point next year... what will we do? use it or not use it to stay
>> compatible?
>>
>> Marcus
>>
>>
>> --
>> Marcus Denker -- http://marcusdenker.de
>>
>>
>
>
>
> --
> best,
> Eliot
>

--
Best regards,
Igor Stasenko.

--
best,
Eliot

--
best,

Eliot

Eliot Miranda-2

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

On Thu, Sep 22, 2011 at 11:28 AM, Igor Stasenko <[hidden email]> wrote:

On 22 September 2011 20:06, Eliot Miranda <[hidden email]> wrote:
> Hi Igor,
>
> On Thu, Sep 22, 2011 at 10:53 AM, Igor Stasenko <[hidden email]> wrote:
>>
>> On 22 September 2011 19:16, Eliot Miranda <[hidden email]> wrote:
>> > (apologies for the duplicate reply; someone needs to sort out their
>> > threading for the benefit of the community ;) )
>> >
>> > On Thu, Sep 22, 2011 at 2:36 AM, Marcus Denker <[hidden email]>
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> There are two changesets waiting for integrating in 1.4 that have
>> >> serious
>> >> consequences:
>> >>
>> >> - Ephemerons. The VM level changes are in the Cog VMs build on Jenkins,
>> >> but have not
>> >> been integrated in the VMMaker codebase.
>> >>
>> >> http://code.google.com/p/pharo/issues/detail?id=4265
>> >
>> > I would *really* like to back out these changes. The Ephemeron
>> > implementation is very much a prototype, requiring a hack to determine
>> > whether an object is an ephemeron (the presence of a marker class in
>> > the
>> > first inst var) that I'm not at all happy with. There is a neater
>> > implementation available via using an unused instSpec which IMO has
>> > significant advantages (much simpler & faster, instSpec is valid at all
>> > times, including during compaction, less overhead, doesn't require a
>> > marker
>> > class), and is the route I'm taking with the new
>> > GC/object-representation
>> > I'm working on now. Note that other than determining whether an object
>> > is
>> > an ephemeron (instSpec/format vs inst var test) the rest of Igor's code
>> > remains the same. I'd like to avoid too much VM forking. Would you all
>> > consider putting these changes on hold for now?
>> > If so, I'll make the effort to produce prototype changes (in the area of
>> > ClassBuilder and class definition; no VM code necessary as yet) to allow
>> > defining Ephemerons via the int spec route by next week at the latest.
>> >
>>
>> i agree that in my implementation this is a weak point. But its hard
>> to do anything without
>> making changes to object format to identify these special objects.
>>
>> The main story behind this is can we afford to change the internals of
>> VM without being beaten hard
>> by "backwards compatibility" party? :)
>
> I don't think we get stuck in this at all. The instSpec/format field has an
> unused value (5 i believe) and this can easily be used for Ephemerons. All
> that is needed is a little image work on these methods:
> Behavior>>typeOfClass
> needs to answer e.g. #ephemeron for ephemeron classes
> ClassBuilder>>computeFormat:instSize:forSuper:ccIndex:
> needs to accept e.g. #ephemeron for type and pass variable: false
> and weak: true for ephemerons to format:variable:words:pointers:weak:.
> ClassBuilder>>format:variable:words:pointers:weak:
> needs to respond to variable: false and weak: true by computing the
> ephemeron instSpec.
>
> Class>>weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:
>
> ClassBuilder>>superclass:weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:
> need siblings, e.g.
>
> ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category
>
> superclass:ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:
> Right? This is easy. Then in the VM there are a few places where pointer
> indexability (formats 3 and 4) need to be firmed up to exclude 5, but
> nothing difficult. We talked about this in email last week.
>

Do you think this will require boosting an image format version number?

That would make sense. Boosting it so that the new VMs can run older images and newer, ephemeron images, but that ephemeron images won't open on older VMs. Yes, this makes perfect sense.

I think it is, because clearly, an images which expect ephemerons to
function cannot work with older VMs properly without
support of ephemerons.

And will lead us to numerous reports "cannot open your f*king image"
with my VM :)
Internally, discussing with Marcus and Stef we came to agreement, that
for each Pharo release we should ship
own version of VM (signed appropriately), so then there will be less
confusion. We also thinking that VM versioning
in future should follow the image versioning, again to make things
simpler and to avoid confusion.

--
Best regards,
Igor Stasenko.

--
best,

Eliot

Henrik Sperre Johansen

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

In reply to this post by Eliot Miranda-2

On 22.09.2011 20:20, Eliot Miranda wrote:

 
On Thu, Sep 22, 2011 at 11:06 AM, Eliot Miranda <[hidden email]> wrote:

Hi Igor,

On Thu, Sep 22, 2011 at 10:53 AM, Igor Stasenko <[hidden email]> wrote:

On 22 September 2011 19:16, Eliot Miranda <[hidden email]> wrote:
> (apologies for the duplicate reply; someone needs to sort out their
> threading for the benefit of the community ;) )
>
> On Thu, Sep 22, 2011 at 2:36 AM, Marcus Denker <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> There are two changesets waiting for integrating in 1.4 that have serious
>> consequences:
>>
>> - Ephemerons. The VM level changes are in the Cog VMs build on Jenkins,
>> but have not
>> been integrated in the VMMaker codebase.
>>
>> http://code.google.com/p/pharo/issues/detail?id=4265
>
> I would *really* like to back out these changes. The Ephemeron
> implementation is very much a prototype, requiring a hack to determine
> whether an object is an ephemeron (the presence of a marker class in the
> first inst var) that I'm not at all happy with. There is a neater
> implementation available via using an unused instSpec which IMO has
> significant advantages (much simpler & faster, instSpec is valid at all
> times, including during compaction, less overhead, doesn't require a marker
> class), and is the route I'm taking with the new GC/object-representation
> I'm working on now. Note that other than determining whether an object is
> an ephemeron (instSpec/format vs inst var test) the rest of Igor's code
> remains the same. I'd like to avoid too much VM forking. Would you all
> consider putting these changes on hold for now?
> If so, I'll make the effort to produce prototype changes (in the area of
> ClassBuilder and class definition; no VM code necessary as yet) to allow
> defining Ephemerons via the int spec route by next week at the latest.
>

i agree that in my implementation this is a weak point. But its hard
to do anything without
making changes to object format to identify these special objects.

The main story behind this is can we afford to change the internals of
VM without being beaten hard
by "backwards compatibility" party? :)

I don't think we get stuck in this at all. The instSpec/format field has an unused value (5 i believe) and this can easily be used for Ephemerons. All that is needed is a little image work on these methods:

Behavior>>typeOfClass

needs to answer e.g. #ephemeron for ephemeron classes

ClassBuilder>>computeFormat:instSize:forSuper:ccIndex:

needs to accept e.g. #ephemeron for type and pass variable: false and weak: true for ephemerons to format:variable:words:pointers:weak:.

ClassBuilder>>format:variable:words:pointers:weak:

needs to respond to variable: false and weak: true by computing the ephemeron instSpec.

Class>>weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

ClassBuilder>>superclass:weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

need siblings, e.g.

ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category

superclass:ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

Right? This is easy. Then in the VM there are a few places where pointer indexability (formats 3 and 4) need to be firmed up to exclude 5, but nothing difficult. We talked about this in email last week.

Here's the format field (Behavior>instSpec at the image level) as currently populated:

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

6 = 32-bit indexable objects (Float, Bitmap ert al)

8 = 8-bit indexable objects (ByteString, ByteArray et al)

12 = CompiledMethod

N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.

For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)

and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.

9 (?) 64-bit indexable

10 - 11 32-bit indexable

12 - 15 16-bit indexable

16 - 23 byte indexable

24 - 31 compiled method

In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.

If we are changing the format for 64bit images anyways, why not simplify it/ be more consistent by spending a full byte?

Bit: 8 7 6 5 4 3 2 1
| 64bit | 32bit |16bit | 8bit |compiled | weak | indexable | instVars |
(Odd number encoded in remaining indexable bit fields)

Could get away with 7 if you put f.ex. the unused indexable weak combination (6) as compiled method/8bit

Or is the header space in your new 64bit format already quite filled, so this is a bad idea?

Cheers,
Henry

Eliot Miranda-2

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

On Thu, Sep 22, 2011 at 12:29 PM, Henrik Sperre Johansen <[hidden email]> wrote:

On 22.09.2011 20:20, Eliot Miranda wrote:
 
On Thu, Sep 22, 2011 at 11:06 AM, Eliot Miranda <[hidden email]> wrote:

Hi Igor,

On Thu, Sep 22, 2011 at 10:53 AM, Igor Stasenko <[hidden email]> wrote:

On 22 September 2011 19:16, Eliot Miranda <[hidden email]> wrote:
> (apologies for the duplicate reply; someone needs to sort out their
> threading for the benefit of the community ;) )
>
> On Thu, Sep 22, 2011 at 2:36 AM, Marcus Denker <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> There are two changesets waiting for integrating in 1.4 that have serious
>> consequences:
>>
>> - Ephemerons. The VM level changes are in the Cog VMs build on Jenkins,
>> but have not
>> been integrated in the VMMaker codebase.
>>
>> http://code.google.com/p/pharo/issues/detail?id=4265
>
> I would *really* like to back out these changes. The Ephemeron
> implementation is very much a prototype, requiring a hack to determine
> whether an object is an ephemeron (the presence of a marker class in the
> first inst var) that I'm not at all happy with. There is a neater
> implementation available via using an unused instSpec which IMO has
> significant advantages (much simpler & faster, instSpec is valid at all
> times, including during compaction, less overhead, doesn't require a marker
> class), and is the route I'm taking with the new GC/object-representation
> I'm working on now. Note that other than determining whether an object is
> an ephemeron (instSpec/format vs inst var test) the rest of Igor's code
> remains the same. I'd like to avoid too much VM forking. Would you all
> consider putting these changes on hold for now?
> If so, I'll make the effort to produce prototype changes (in the area of
> ClassBuilder and class definition; no VM code necessary as yet) to allow
> defining Ephemerons via the int spec route by next week at the latest.
>

i agree that in my implementation this is a weak point. But its hard
to do anything without
making changes to object format to identify these special objects.

The main story behind this is can we afford to change the internals of
VM without being beaten hard
by "backwards compatibility" party? :)

I don't think we get stuck in this at all. The instSpec/format field has an unused value (5 i believe) and this can easily be used for Ephemerons. All that is needed is a little image work on these methods:

Behavior>>typeOfClass

needs to answer e.g. #ephemeron for ephemeron classes

ClassBuilder>>computeFormat:instSize:forSuper:ccIndex:

needs to accept e.g. #ephemeron for type and pass variable: false and weak: true for ephemerons to format:variable:words:pointers:weak:.

ClassBuilder>>format:variable:words:pointers:weak:

needs to respond to variable: false and weak: true by computing the ephemeron instSpec.

Class>>weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

ClassBuilder>>superclass:weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

need siblings, e.g.

ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category

superclass:ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

Right? This is easy. Then in the VM there are a few places where pointer indexability (formats 3 and 4) need to be firmed up to exclude 5, but nothing difficult. We talked about this in email last week.

Here's the format field (Behavior>instSpec at the image level) as currently populated:

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

6 = 32-bit indexable objects (Float, Bitmap ert al)

8 = 8-bit indexable objects (ByteString, ByteArray et al)

12 = CompiledMethod

N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.

For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)

and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.

9 (?) 64-bit indexable

10 - 11 32-bit indexable

12 - 15 16-bit indexable

16 - 23 byte indexable

24 - 31 compiled method

In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.
If we are changing the format for 64bit images anyways, why not simplify it/ be more consistent by spending a full byte?

Bit: 8 7 6 5 4 3 2 1
| 64bit | 32bit |16bit | 8bit |compiled | weak | indexable | instVars |
(Odd number encoded in remaining indexable bit fields)

I used to prefer this approach but I've realised that the format/instSpec approach (I think Dan came up with) makes better use of bits because so many of the bit combinations are mutually exclusive. For example, pointers excludes all the byte/short/32-bit/64-bit indexability combinations. Also, see below...

Could get away with 7 if you put f.ex. the unused indexable weak combination (6) as compiled method/8bit

Or is the header space in your new 64bit format already quite filled, so this is a bad idea?

Yes, ish. But they're scarce, and very useful for experiments etc. Right now I have

typedef struct {

unsigned short classIndex;

unsigned unused0 : 6;

unsigned isPinned : 1;

unsigned isImmutable : 1;

unsigned format : 5; /* on a byte boundary */

unsigned isMarked : 1;

unsigned isGrey : 1;

unsigned isRemembered : 1;

unsigned objHash : 24; /* on a 32-bit word boundary */

unsigned char slotSize; /* on a byte boundary */

} CogObjectHeader;

Where classIndex is 16-bits simply for efficiency and will grow to 20 or 22 bits as needed. So one could steal one or two bits from unused0 and two bits from objHash, and give these to format, but it would be a waste. Better keep these back for other uses.

Also, can I ask the assembled company exactly how many bits you'd spend on the objHash (identityHash)? Think forward to 64-bits. Is 24 bits about all we can afford or still too generous? Anybody have any data to contribute?

Cheers,
Henry

--
best,

Eliot

David T. Lewis

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

In reply to this post by Eliot Miranda-2

On Thu, Sep 22, 2011 at 11:37:46AM -0700, Eliot Miranda wrote:
>
> On Thu, Sep 22, 2011 at 11:28 AM, Igor Stasenko <[hidden email]> wrote:
> >
> > Do you think this will require boosting an image format version number?
>
> That would make sense. Boosting it so that the new VMs can run older images
> and newer, ephemeron images, but that ephemeron images won't open on older
> VMs. Yes, this makes perfect sense.

Please use image format number 6521, allocating bit 5 as the "ephemeron support bit".

Details:
6505 printStringBase: 2 ==> '1100101101001'
ImageFormat bitsInUse printStringBase: 2 ==> '10001100111101111'

"Next available bit is bit 5"
2r1100101101001 ==> 6505
2r1100101111001 ==> 6521

"A 64 bit image with cog support plus ephemeron support would be 68019"
(ImageFormat wordSize: 4 cog: true) asInteger ==> 6505
(ImageFormat wordSize: 4 cog: true) asInteger bitOr: 2r10000 ==> 6521
(ImageFormat wordSize: 8 cog: true) asInteger ==> 68003
(ImageFormat wordSize: 8 cog: true) asInteger bitOr: 2r10000 ==> 68019

6505 printStringBase: 2 ==> '1100101101001'
6521 printStringBase: 2 ==> '1100101111001'
68003 printStringBase: 2 ==> '10000100110100011'
68019 printStringBase: 2 ==> '10000100110110011'

If you agree, I'll update ImageFormat and ImageFormatTest to document
the new format number assignments.

Dave

stephane ducasse-2

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

In reply to this post by Eliot Miranda-2

Stupid question:

what does it mean?
> we need 8 CompiledMethod values

Stef

>
>
> Here's the format field (Behavior>instSpec at the image level) as currently populated:
>
> 0 = 0 sized objects (UndefinedObject True False et al)
> 1 = non-indexable objects with inst vars (Point et al)
> 2 = indexable objects with no inst vars (Array et al)
> 3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)
> 4 = weak indexable objects with inst vars (WeakArray et al)
> 6 = 32-bit indexable objects (Float, Bitmap ert al)
> 8 = 8-bit indexable objects (ByteString, ByteArray et al)
> 12 = CompiledMethod
>
> N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.
>
>
> For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then
>
> 0 = 0 sized objects (UndefinedObject True False et al)
> 1 = non-indexable objects with inst vars (Point et al)
> 2 = indexable objects with no inst vars (Array et al)
> 3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)
> 4 = weak indexable objects with inst vars (WeakArray et al)
> 5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)
>
> and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.
>
> 9 (?) 64-bit indexable
> 10 - 11 32-bit indexable
> 12 - 15 16-bit indexable
> 16 - 23 byte indexable
> 24 - 31 compiled method
>
> In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.
>
>

Igor Stasenko

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

On 23 September 2011 09:44, stephane ducasse <[hidden email]> wrote:
>
> Stupid question:
>
> what does it mean?
>> we need 8 CompiledMethod values
>
to encode odd size values.
The header encoding size in machine words, which on 64 bit system are
always 8-byte aligned.
But compiled method size could be any size in bytes and to encode it
we need 3 bits somewhere (or 8 different values),
to encode size in bytes i.e.:

sizeinbytes = sizeInWords*8 + (oddSize)

where oddsize is our 0..7 value

> Stef
>
>>
>>
>> Here's the format field (Behavior>instSpec at the image level) as currently populated:
>>
>> 0 = 0 sized objects (UndefinedObject True False et al)
>> 1 = non-indexable objects with inst vars (Point et al)
>> 2 = indexable objects with no inst vars (Array et al)
>> 3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)
>> 4 = weak indexable objects with inst vars (WeakArray et al)
>> 6 = 32-bit indexable objects (Float, Bitmap ert al)
>> 8 = 8-bit indexable objects (ByteString, ByteArray et al)
>> 12 = CompiledMethod
>>
>> N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.
>>
>>
>> For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then
>>
>> 0 = 0 sized objects (UndefinedObject True False et al)
>> 1 = non-indexable objects with inst vars (Point et al)
>> 2 = indexable objects with no inst vars (Array et al)
>> 3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)
>> 4 = weak indexable objects with inst vars (WeakArray et al)
>> 5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)
>>
>> and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.
>>
>> 9 (?) 64-bit indexable
>> 10 - 11 32-bit indexable
>> 12 - 15 16-bit indexable
>> 16 - 23 byte indexable
>> 24 - 31 compiled method
>>
>> In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.
>>
>>
>

--
Best regards,
Igor Stasenko.

Igor Stasenko

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

btw, i am still wondering if it necessary to know exact size in bytes
of compiled method.
I think we could align its size to 8 bytes without much loss. then it
will take single place (value) in format instead of 8.

Of course, for bytearrays we cannot do same trick, since then we will
have not container left which can
hold data with size aligned to single byte.

On 23 September 2011 12:58, Igor Stasenko <[hidden email]> wrote:

> On 23 September 2011 09:44, stephane ducasse <[hidden email]> wrote:
>>
>> Stupid question:
>>
>> what does it mean?
>>> we need 8 CompiledMethod values
>>
> to encode odd size values.
> The header encoding size in machine words, which on 64 bit system are
> always 8-byte aligned.
> But compiled method size could be any size in bytes and to encode it
> we need 3 bits somewhere (or 8 different values),
> to encode size in bytes i.e.:
>
> sizeinbytes = sizeInWords*8 + (oddSize)
>
> where oddsize is our 0..7 value
>
>> Stef
>>
>>>
>>>
>>> Here's the format field (Behavior>instSpec at the image level) as currently populated:
>>>
>>> 0 = 0 sized objects (UndefinedObject True False et al)
>>> 1 = non-indexable objects with inst vars (Point et al)
>>> 2 = indexable objects with no inst vars (Array et al)
>>> 3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)
>>> 4 = weak indexable objects with inst vars (WeakArray et al)
>>> 6 = 32-bit indexable objects (Float, Bitmap ert al)
>>> 8 = 8-bit indexable objects (ByteString, ByteArray et al)
>>> 12 = CompiledMethod
>>>
>>> N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.
>>>
>>>
>>> For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then
>>>
>>> 0 = 0 sized objects (UndefinedObject True False et al)
>>> 1 = non-indexable objects with inst vars (Point et al)
>>> 2 = indexable objects with no inst vars (Array et al)
>>> 3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)
>>> 4 = weak indexable objects with inst vars (WeakArray et al)
>>> 5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)
>>>
>>> and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.
>>>
>>> 9 (?) 64-bit indexable
>>> 10 - 11 32-bit indexable
>>> 12 - 15 16-bit indexable
>>> 16 - 23 byte indexable
>>> 24 - 31 compiled method
>>>
>>> In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.
>>>
>>>
>>
>
>
>
> --
> Best regards,
> Igor Stasenko.
>

--
Best regards,
Igor Stasenko.

Henrik Sperre Johansen

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

In reply to this post by Eliot Miranda-2

On 22.09.2011 21:57, Eliot Miranda wrote:

 
On Thu, Sep 22, 2011 at 12:29 PM, Henrik Sperre Johansen <[hidden email]> wrote:
On 22.09.2011 20:20, Eliot Miranda wrote:
 
On Thu, Sep 22, 2011 at 11:06 AM, Eliot Miranda <[hidden email]> wrote:

Hi Igor,

On Thu, Sep 22, 2011 at 10:53 AM, Igor Stasenko <[hidden email]> wrote:

On 22 September 2011 19:16, Eliot Miranda <[hidden email]> wrote:
> (apologies for the duplicate reply; someone needs to sort out their
> threading for the benefit of the community ;) )
>
> On Thu, Sep 22, 2011 at 2:36 AM, Marcus Denker <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> There are two changesets waiting for integrating in 1.4 that have serious
>> consequences:
>>
>> - Ephemerons. The VM level changes are in the Cog VMs build on Jenkins,
>> but have not
>> been integrated in the VMMaker codebase.
>>
>> http://code.google.com/p/pharo/issues/detail?id=4265
>
> I would *really* like to back out these changes. The Ephemeron
> implementation is very much a prototype, requiring a hack to determine
> whether an object is an ephemeron (the presence of a marker class in the
> first inst var) that I'm not at all happy with. There is a neater
> implementation available via using an unused instSpec which IMO has
> significant advantages (much simpler & faster, instSpec is valid at all
> times, including during compaction, less overhead, doesn't require a marker
> class), and is the route I'm taking with the new GC/object-representation
> I'm working on now. Note that other than determining whether an object is
> an ephemeron (instSpec/format vs inst var test) the rest of Igor's code
> remains the same. I'd like to avoid too much VM forking. Would you all
> consider putting these changes on hold for now?
> If so, I'll make the effort to produce prototype changes (in the area of
> ClassBuilder and class definition; no VM code necessary as yet) to allow
> defining Ephemerons via the int spec route by next week at the latest.
>

i agree that in my implementation this is a weak point. But its hard
to do anything without
making changes to object format to identify these special objects.

The main story behind this is can we afford to change the internals of
VM without being beaten hard
by "backwards compatibility" party? :)

I don't think we get stuck in this at all. The instSpec/format field has an unused value (5 i believe) and this can easily be used for Ephemerons. All that is needed is a little image work on these methods:

Behavior>>typeOfClass

needs to answer e.g. #ephemeron for ephemeron classes

ClassBuilder>>computeFormat:instSize:forSuper:ccIndex:

needs to accept e.g. #ephemeron for type and pass variable: false and weak: true for ephemerons to format:variable:words:pointers:weak:.

ClassBuilder>>format:variable:words:pointers:weak:

needs to respond to variable: false and weak: true by computing the ephemeron instSpec.

Class>>weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

ClassBuilder>>superclass:weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

need siblings, e.g.

ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category

superclass:ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

Right? This is easy. Then in the VM there are a few places where pointer indexability (formats 3 and 4) need to be firmed up to exclude 5, but nothing difficult. We talked about this in email last week.

Here's the format field (Behavior>instSpec at the image level) as currently populated:

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

6 = 32-bit indexable objects (Float, Bitmap ert al)

8 = 8-bit indexable objects (ByteString, ByteArray et al)

12 = CompiledMethod

N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.

For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)

and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.

9 (?) 64-bit indexable

10 - 11 32-bit indexable

12 - 15 16-bit indexable

16 - 23 byte indexable

24 - 31 compiled method

In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.
If we are changing the format for 64bit images anyways, why not simplify it/ be more consistent by spending a full byte?

Bit: 8 7 6 5 4 3 2 1
| 64bit | 32bit |16bit | 8bit |compiled | weak | indexable | instVars |
(Odd number encoded in remaining indexable bit fields)
I used to prefer this approach but I've realised that the format/instSpec approach (I think Dan came up with) makes better use of bits because so many of the bit combinations are mutually exclusive. For example, pointers excludes all the byte/short/32-bit/64-bit indexability combinations. Also, see below...

Could get away with 7 if you put f.ex. the unused indexable weak combination (6) as compiled method/8bit

Or is the header space in your new 64bit format already quite filled, so this is a bad idea?

Yes, ish. But they're scarce, and very useful for experiments etc. Right now I have

typedef struct {

unsigned short classIndex;

unsigned unused0 : 6;

unsigned isPinned : 1;

unsigned isImmutable : 1;

unsigned format : 5; /* on a byte boundary */

unsigned isMarked : 1;

unsigned isGrey : 1;

unsigned isRemembered : 1;

unsigned objHash : 24; /* on a 32-bit word boundary */

unsigned char slotSize; /* on a byte boundary */

} CogObjectHeader;

Where classIndex is 16-bits simply for efficiency and will grow to 20 or 22 bits as needed. So one could steal one or two bits from unused0 and two bits from objHash, and give these to format, but it would be a waste. Better keep these back for other uses.

Also, can I ask the assembled company exactly how many bits you'd spend on the objHash (identityHash)? Think forward to 64-bits. Is 24 bits about all we can afford or still too generous? Anybody have any data to contribute?

This is probably a stupid question, but where is the variable size in words stored?
In a quadword preceding the header like it is in 32bit format?

The reason I'm asking is that to me, the main application of identityHash is for HashedCollection's, and the max size of those thus impact what a reasonable answer would be...

Cheers,
Henry

Eliot Miranda-2

Re: [Pharo-project] Integrating Changes in 1.4 that require a new VM

On Fri, Sep 23, 2011 at 5:01 AM, Henrik Sperre Johansen <[hidden email]> wrote:

On 22.09.2011 21:57, Eliot Miranda wrote:
 
On Thu, Sep 22, 2011 at 12:29 PM, Henrik Sperre Johansen <[hidden email]> wrote:
On 22.09.2011 20:20, Eliot Miranda wrote:
 
On Thu, Sep 22, 2011 at 11:06 AM, Eliot Miranda <[hidden email]> wrote:

Hi Igor,

On Thu, Sep 22, 2011 at 10:53 AM, Igor Stasenko <[hidden email]> wrote:

On 22 September 2011 19:16, Eliot Miranda <[hidden email]> wrote:
> (apologies for the duplicate reply; someone needs to sort out their
> threading for the benefit of the community ;) )
>
> On Thu, Sep 22, 2011 at 2:36 AM, Marcus Denker <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> There are two changesets waiting for integrating in 1.4 that have serious
>> consequences:
>>
>> - Ephemerons. The VM level changes are in the Cog VMs build on Jenkins,
>> but have not
>> been integrated in the VMMaker codebase.
>>
>> http://code.google.com/p/pharo/issues/detail?id=4265
>
> I would *really* like to back out these changes. The Ephemeron
> implementation is very much a prototype, requiring a hack to determine
> whether an object is an ephemeron (the presence of a marker class in the
> first inst var) that I'm not at all happy with. There is a neater
> implementation available via using an unused instSpec which IMO has
> significant advantages (much simpler & faster, instSpec is valid at all
> times, including during compaction, less overhead, doesn't require a marker
> class), and is the route I'm taking with the new GC/object-representation
> I'm working on now. Note that other than determining whether an object is
> an ephemeron (instSpec/format vs inst var test) the rest of Igor's code
> remains the same. I'd like to avoid too much VM forking. Would you all
> consider putting these changes on hold for now?
> If so, I'll make the effort to produce prototype changes (in the area of
> ClassBuilder and class definition; no VM code necessary as yet) to allow
> defining Ephemerons via the int spec route by next week at the latest.
>

i agree that in my implementation this is a weak point. But its hard
to do anything without
making changes to object format to identify these special objects.

The main story behind this is can we afford to change the internals of
VM without being beaten hard
by "backwards compatibility" party? :)

I don't think we get stuck in this at all. The instSpec/format field has an unused value (5 i believe) and this can easily be used for Ephemerons. All that is needed is a little image work on these methods:

Behavior>>typeOfClass

needs to answer e.g. #ephemeron for ephemeron classes

ClassBuilder>>computeFormat:instSize:forSuper:ccIndex:

needs to accept e.g. #ephemeron for type and pass variable: false and weak: true for ephemerons to format:variable:words:pointers:weak:.

ClassBuilder>>format:variable:words:pointers:weak:

needs to respond to variable: false and weak: true by computing the ephemeron instSpec.

Class>>weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

ClassBuilder>>superclass:weakSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

need siblings, e.g.

ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category

superclass:ephemeronSubclass:instanceVariableNames:classVariableNames:poolDictionaries:category:

Right? This is easy. Then in the VM there are a few places where pointer indexability (formats 3 and 4) need to be firmed up to exclude 5, but nothing difficult. We talked about this in email last week.

Here's the format field (Behavior>instSpec at the image level) as currently populated:

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

6 = 32-bit indexable objects (Float, Bitmap ert al)

8 = 8-bit indexable objects (ByteString, ByteArray et al)

12 = CompiledMethod

N.B. in the VM the least two bits of the format/instSpec for byte objects (formats 8 and 12) is used to encode the number of odd bytes in the object, so that a 1 character ByteString has a format of 11, = 8 + 3, size = 1 word - 3 bytes.

For the future (i.e. the new GC/object representation, /not/ for the first implementation of ephemerons which we can do now, for Pharo 1.4 or 1.5) we need to extend format/instSpec to support 64 bits. I think format needs to be made a 5 bit field with room for 4 bits of odd bytes for 64-bit images. [For VMers, the Size4Bit is a horrible hack). So then

0 = 0 sized objects (UndefinedObject True False et al)

1 = non-indexable objects with inst vars (Point et al)

2 = indexable objects with no inst vars (Array et al)

3 = indexable objects with inst vars (MethodContext AdditionalMethodState et al)

4 = weak indexable objects with inst vars (WeakArray et al)

5 = weak non-indexable objects with inst vars (ephemerons) (Ephemeron)

and we need 8 CompiledMethod values, 8 byte values, 4 16-bit values, 2 32-bit values and a 64-bit value, = 23 values, 23 + 5 = 30, so there is room, e.g.

9 (?) 64-bit indexable

10 - 11 32-bit indexable

12 - 15 16-bit indexable

16 - 23 byte indexable

24 - 31 compiled method

In 32-bit images only the least significant 2 bits would be used for formats 16 & 24, and the least significant bit for format 12.
If we are changing the format for 64bit images anyways, why not simplify it/ be more consistent by spending a full byte?

Bit: 8 7 6 5 4 3 2 1
| 64bit | 32bit |16bit | 8bit |compiled | weak | indexable | instVars |
(Odd number encoded in remaining indexable bit fields)
I used to prefer this approach but I've realised that the format/instSpec approach (I think Dan came up with) makes better use of bits because so many of the bit combinations are mutually exclusive. For example, pointers excludes all the byte/short/32-bit/64-bit indexability combinations. Also, see below...

Could get away with 7 if you put f.ex. the unused indexable weak combination (6) as compiled method/8bit

Or is the header space in your new 64bit format already quite filled, so this is a bad idea?

Yes, ish. But they're scarce, and very useful for experiments etc. Right now I have

typedef struct {

unsigned short classIndex;

unsigned unused0 : 6;

unsigned isPinned : 1;

unsigned isImmutable : 1;

unsigned format : 5; /* on a byte boundary */

unsigned isMarked : 1;

unsigned isGrey : 1;

unsigned isRemembered : 1;

unsigned objHash : 24; /* on a 32-bit word boundary */

unsigned char slotSize; /* on a byte boundary */

} CogObjectHeader;

Where classIndex is 16-bits simply for efficiency and will grow to 20 or 22 bits as needed. So one could steal one or two bits from unused0 and two bits from objHash, and give these to format, but it would be a waste. Better keep these back for other uses.

Also, can I ask the assembled company exactly how many bits you'd spend on the objHash (identityHash)? Think forward to 64-bits. Is 24 bits about all we can afford or still too generous? Anybody have any data to contribute?
This is probably a stupid question, but where is the variable size in words stored?

So the size of an object in slots (32-bit or 64-bit) is the byte slotSize field, allowing an object to have 254 slots before it is too big for this representation. If an object has more than 254 slots its slotSize has the value 255 and an additional 64-bit word (64-bit to guarantee 64-bit alignment of all objects) preceeding contains the actual slot size (even if only 32-bits are used). This means that there is a maximum of 2/255 words of overhead for this extra word, or < 1%.

So to compute the size of an indexable object one determines the slot size and subtracts the number of named inst vars. For format 4 there are no named inst vars, so the computation is something like

| ss |

numIndexableFields := (ss := obj slotSize > 254 ifTrue: [obj overflowSlotSize] ifFalse: [ss]) - (obj format = 3 ifTrue: [self numIndexableFields: obj] ifFalse: [0])

In 64-bit VisualWorks I took a different approach, using an 8 bit field to hold either the number of named inst vars or the number of odd bytes. But this meant an 8 bit slot size, an 8 bit nun named inst vars/odd bytes and 8 header flags, leaving only 20 buts each for the class index and obj hash fields. With Squeak this isn't compelling because the collection hierarchy has been reworked to use Arrays instead of the older flat format (e.g. where OrderedCollection was indexable and has firstIndex and lastIndex).

In a quadword preceding the header like it is in 32bit format?

Yes.

The reason I'm asking is that to me, the main application of identityHash is for HashedCollection's, and the max size of those thus impact what a reasonable answer would be...

and what kinds of sizes have you seen?

Cheers,
Henry

--
best,

Eliot