ObjectMemory>>contiguousFixedSpaceAt: seems to report non-contiguous memory

I've been getting occasional crashes when trying to allocate ByteArrays
using newInFixedSpace:.  The system has plenty of free memory, so the VM
can certainly allocate more.  Some investigation has revealed that the
following seems to be happening:

- The primitive in newInFixedSpace: fails
- The failure handling code decides to favour garbage collection over
more allocation and runs a garbage collect.
- The garbage collect frees some memory.
- MemoryPolicy>>makeSpaceFor:ofType: thinks it has freed enough
according to the numbers it gets back from contiguousFixedSpaceAt:, but
in actual fact the memory is fragmented and contiguousFixedSpaceAt: is
not taking this into account.
- newInFixedSpaceNoRetry: has a second attempt but also fails because of
the fragmentation.

Had the failure handling code known the actual contiguous fixed space it
would have known that it was not enough and increased the size of fixed
space.

The following is the output from my test script to reproduce this:

--------------------
Fixed space contiguous sizes: #(204780)
total fixed space size: 204800
Maximum that can be allocated without growing fixed space: 204754

After allocating 150 x 1000 fixed space ByteArrays: #(50580)
total fixed space size: 204800
Maximum that can be allocated without growing fixed space: 50554

After freeing every second allocation: #(126180)
total fixed space size: 204800
Maximum that can be allocated without growing fixed space: 50554
--------------------

Freeing every second allocation should not have increased contiguous
fixed space by more than one or two thousand that may have happened to
be adjacent to the 50k block of free fixed space.  Also, note that the
maximum that could then be allocated (50554) was less than what should
have been the unfragmented size (50580).  This may be due to more
fragmentation not caused my me (although this is a clean image), or due
to some other fixed size not being taken into account.

Following is the script:

--------------------
fragmentedAllocations := nil.
log := String new writeStream.

reportFixedMemoryInfo := [:message |
ObjectMemory globalGarbageCollect.
log cr;
    cr; nextPutAll: message; print: ((1 to: ObjectMemory current
fixedSegments) collect: [:x | ObjectMemory current
contiguousFixedSpaceAt: x]);
    cr; nextPutAll: 'total fixed space size: '; print: ObjectMemory
current fixedBytes.
actuallyAllocated := ObjectMemory current contiguousFixedSpaceAt: 1.
[ByteArray newInFixedSpaceNoRetry: actuallyAllocated. ObjectMemory
globalGarbageCollect.]
    on: ObjectMemory allocationFailedSignal
    do: [:ex |
        actuallyAllocated := actuallyAllocated - 1.
        ex retry].
log cr; nextPutAll: 'Maximum that can be allocated without growing fixed
space: '; print: actuallyAllocated].

reportFixedMemoryInfo value: 'Fixed space contiguous sizes: '.

fragmentedAllocations := OrderedCollection new.
150 timesRepeat: [
    fragmentedAllocations add: (ByteArray newInFixedSpace: 1000)].

reportFixedMemoryInfo value: 'After allocating 150 x 1000 fixed space
ByteArrays: '.

1 to: fragmentedAllocations size by: 2 do: [:x |
    fragmentedAllocations at: x put: nil].

ObjectMemory globalGarbageCollect.
reportFixedMemoryInfo value: 'After freeing every second allocation: '.

log contents
--------------------

David

Hi David,

        you're exactly right.  I just looked at ObjectMemory>>contiguousFixedSpaceAt: and iots clear that waht it answers is the total amount of free space in a fixed-space segment, which has nothing at all to do with the maximum contiguous free space in the segment.  I'll create an AR.  Thanks!

Foer reference here's the erroneous definition, which is correcty only for empty free-space segments:

contiguousFixedSpaceAt: index
        "Answer the number of bytes of contiguous free space in the specified FixedSpace
        segment."
        "ObjectMemory current contiguousFixedSpaceAt: 1"

        ^(self fixedSegmentSizeAt: index)
                - (self ftEntriesToBytes: (self fixedTableSizeAt: index))
                - (self fixedDataSizeAt: index)

David Price <[hidden email]> wrote:

| I've been getting occasional crashes when trying to allocate ByteArrays
| using newInFixedSpace:.  The system has plenty of free memory, so the VM
| can certainly allocate more.  Some investigation has revealed that the
| following seems to be happening:

| - The primitive in newInFixedSpace: fails
| - The failure handling code decides to favour garbage collection over
| more allocation and runs a garbage collect.
| - The garbage collect frees some memory.
| - MemoryPolicy>>makeSpaceFor:ofType: thinks it has freed enough
| according to the numbers it gets back from contiguousFixedSpaceAt:, but
| in actual fact the memory is fragmented and contiguousFixedSpaceAt: is
| not taking this into account.
| - newInFixedSpaceNoRetry: has a second attempt but also fails because of
| the fragmentation.

| Had the failure handling code known the actual contiguous fixed space it
| would have known that it was not enough and increased the size of fixed
| space.

| The following is the output from my test script to reproduce this:

| --------------------
| Fixed space contiguous sizes: #(204780)
| total fixed space size: 204800
| Maximum that can be allocated without growing fixed space: 204754

| After allocating 150 x 1000 fixed space ByteArrays: #(50580)
| total fixed space size: 204800
| Maximum that can be allocated without growing fixed space: 50554

| After freeing every second allocation: #(126180)
| total fixed space size: 204800
| Maximum that can be allocated without growing fixed space: 50554
| --------------------

| Freeing every second allocation should not have increased contiguous
| fixed space by more than one or two thousand that may have happened to
| be adjacent to the 50k block of free fixed space.  Also, note that the
| maximum that could then be allocated (50554) was less than what should
| have been the unfragmented size (50580).  This may be due to more
| fragmentation not caused my me (although this is a clean image), or due
| to some other fixed size not being taken into account.

| Following is the script:

| --------------------
| fragmentedAllocations := nil.
| log := String new writeStream.

| reportFixedMemoryInfo := [:message |
| ObjectMemory globalGarbageCollect.
| log cr;
|     cr; nextPutAll: message; print: ((1 to: ObjectMemory current
| fixedSegments) collect: [:x | ObjectMemory current
| contiguousFixedSpaceAt: x]);
|     cr; nextPutAll: 'total fixed space size: '; print: ObjectMemory
| current fixedBytes.
| actuallyAllocated := ObjectMemory current contiguousFixedSpaceAt: 1.
| [ByteArray newInFixedSpaceNoRetry: actuallyAllocated. ObjectMemory
| globalGarbageCollect.]
|     on: ObjectMemory allocationFailedSignal
|     do: [:ex |
|         actuallyAllocated := actuallyAllocated - 1.
|         ex retry].
| log cr; nextPutAll: 'Maximum that can be allocated without growing fixed
| space: '; print: actuallyAllocated].

| reportFixedMemoryInfo value: 'Fixed space contiguous sizes: '.

| fragmentedAllocations := OrderedCollection new.
| 150 timesRepeat: [
|     fragmentedAllocations add: (ByteArray newInFixedSpace: 1000)].

| reportFixedMemoryInfo value: 'After allocating 150 x 1000 fixed space
| ByteArrays: '.

| 1 to: fragmentedAllocations size by: 2 do: [:x |
|     fragmentedAllocations at: x put: nil].

| ObjectMemory globalGarbageCollect.
| reportFixedMemoryInfo value: 'After freeing every second allocation: '.

| log contents
| --------------------

| David
---
Eliot Miranda                 ,,,^..^,,,                mailto:[hidden email]
VisualWorks Engineering, Cincom  Smalltalk: scene not herd  Tel +1 408 216 4581
3350 Scott Blvd, Bldg 36 Suite B, Santa Clara, CA 95054 USA Fax +1 408 216 4500

Travis Griggs wrote:
> Yay! I'm glad someone else is hitting this and making some headway. I
> posted about issues with this on Feb 16 and Feb 21. And never got
> anywhere. I'd like this to work and work well.
>
I've worked around the problem for now by doing up to three attempts at
allocation, where prior to the third attempt it will always expand free
space, so you can be sure to get enough non-fragmented memory.  This is
kind of ugly, but I think it should always work.  It's in public store
as "FixedSpaceWorkaround".
> I had a hunch I tried with your script actually. And it worked. Change
> your "free every 2nd" to free as 2 to: count by: 2. And then you will
> get the result you are looking for (or close, it's 51582). Because you
> went from 1 to: count by: 2, you got rid of the N - 1 slot, but not
> the Nth one. So no extra contiguous room became available.
>
Ah, yes, of course :)
> I'd love for someone to explain the results I was seeing with the Feb
> 16/21 posts. :)
You mean where it reports 1 fixed space object?  That's certainly odd.

David

[hidden email] wrote: Hi Eliot,

ObjectMemory>>contiguousOldSpaceAt: is implemented similarly.  This I
guess will be correct if old space is never fragmented, which I guess
seems kind of plausible, although if you did a non-compacting GC on old
space then it seems less likely.

Thanks,
David

David Price <[hidden email]> wrote:

| [hidden email] wrote:
| > Hi David,
| >
| > you're exactly right.  I just looked at ObjectMemory>>contiguousFixedSpaceAt: and iots clear that waht it answers is the total amount of free space in a fixed-space segment, which has nothing at all to do with the maximum contiguous free space in the segment.  I'll create an AR.  Thanks!
| >
| > Foer reference here's the erroneous definition, which is correcty only for empty free-space segments:
| >
| > contiguousFixedSpaceAt: index
| > "Answer the number of bytes of contiguous free space in the specified FixedSpace
| > segment."
| > "ObjectMemory current contiguousFixedSpaceAt: 1"
| >
| > ^(self fixedSegmentSizeAt: index)
| > - (self ftEntriesToBytes: (self fixedTableSizeAt: index))
| > - (self fixedDataSizeAt: index)
| >
| >
| Hi Eliot,

| ObjectMemory>>contiguousOldSpaceAt: is implemented similarly.  This I
| guess will be correct if old space is never fragmented, which I guess
| seems kind of plausible, although if you did a non-compacting GC on old
| space then it seems less likely.

The similar implementation is because contiguousOldSpaceAt: was (incorrectly) copy-pasted to implement contiguousFixedSpaceAt:.  The information for each oldSpace segment is the size of the segment, the length of the object table and the size of the objects.  The OT asnd objects start at opposite ends and grow together.  So the hole in the middle is contiguous free space.  There may be free blocks in the object portion of the segment, and these may be larger than the hole, but the hole is still contiguous free space.

A fixed-space segment isn't organized like this at all.  Its a list of blocks, some free, some used.  The used blocks can never be moved (by definition; fixed space segments get compacted on image load).  So there's no hole in the middle to compute.  One would have to search the list to find the largest free block.
---
Eliot Miranda                 ,,,^..^,,,                mailto:[hidden email]
VisualWorks Engineering, Cincom  Smalltalk: scene not herd  Tel +1 408 216 4581
3350 Scott Blvd, Bldg 36 Suite B, Santa Clara, CA 95054 USA Fax +1 408 216 4500