[Spur] endless recursion in forward become

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[Spur] endless recursion in forward become

Bert Freudenberg
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -
Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Eliot Miranda-2
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Bert Freudenberg
 
Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Eliot Miranda-2
 
Hi Bert,


On Jun 22, 2016, at 5:57 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

because that's how Spur works.  Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

Spur chooses to trade memory (creating the copies) over time (searching the entire heap looking fir references).  This is the right trade off until you start becoming objects that are substantial fractions of the entire heap in size.

In your case you could avoid the pain simply by making sure the steam had a wide String as contents in the first place.  I understand that may not be possible.


- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Bert Freudenberg
 
Hi Eliot,

okay, but it still is a bug that I get an endless recursion. AFAICT the bug is in growMemoryByAtLeast: which does not grow the memory as requested.

- Bert -

On Wed, Jun 22, 2016 at 3:44 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Jun 22, 2016, at 5:57 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

because that's how Spur works.  Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

Spur chooses to trade memory (creating the copies) over time (searching the entire heap looking fir references).  This is the right trade off until you start becoming objects that are substantial fractions of the entire heap in size.

In your case you could avoid the pain simply by making sure the steam had a wide String as contents in the first place.  I understand that may not be possible.


- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot



Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

timrowledge
In reply to this post by Eliot Miranda-2


> On 22-06-2016, at 6:44 AM, Eliot Miranda <[hidden email]> wrote:
>
> Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

I could have sworn that you had added the “copy the small one into the big one’s space” optimisation. Maybe it was only added to the list of "things to do".

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
I'm so skeptical that I'm not sure I'm really a skeptic


Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Eliot Miranda-2
In reply to this post by Bert Freudenberg
 
Hi Bert,

On Jun 22, 2016, at 6:54 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

okay, but it still is a bug that I get an endless recursion. AFAICT the bug is in growMemoryByAtLeast: which does not grow the memory as requested.

Do you have a reproducible case for me to look at?  The fix I submitted fixes your original case.  But you could try and fix it yourself.  If the become: is two-way and fails with out-of-memory the failure code should ask to grow memory by the sum of the instance byte sizes.  There us a bytesSzeOfInstance: method (or close) to compute the required space.


- Bert -

On Wed, Jun 22, 2016 at 3:44 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Jun 22, 2016, at 5:57 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

because that's how Spur works.  Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

Spur chooses to trade memory (creating the copies) over time (searching the entire heap looking fir references).  This is the right trade off until you start becoming objects that are substantial fractions of the entire heap in size.

In your case you could avoid the pain simply by making sure the steam had a wide String as contents in the first place.  I understand that may not be possible.


- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot



Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Bert Freudenberg
 
On Wed, Jun 22, 2016 at 10:15 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,
Do you have a reproducible case for me to look at?

Yes, from one email ago:

        (ByteString new: 20000000) become: (WideString new: 20000000)

 The fix I submitted fixes your original case.  But you could try and fix it yourself.  If the become: is two-way and fails with out-of-memory the failure code should ask to grow memory by the sum of the instance byte sizes.

The failure code in elementsExchangeIdentityWith: asks the memory to grow by a fixed amount and retry:

ec == #'insufficient object memory'
ifTrue: [Smalltalk garbageCollect < 1048576
ifTrue: [Smalltalk growMemoryByAtLeast: 1048576].
^ self elementsExchangeIdentityWith: otherArray].

After a few repetitions of this there should be enough room.

The problem is that growMemoryByAtLeast: does not grow the memory, it apparently only assures that the given amount is available. If that is indeed the intention of primitive 180, we should rename it to something like ensureMemoryAtLeastFree:.

 There us a bytesSzeOfInstance: method (or close) to compute the required space.

This seems to work, yes:

ec == #'insufficient object memory' ifTrue:
[Smalltalk ensureMemoryAtLeastFree: ({self. otherArray} detectSum:
[:array | array detectSum:
[:obj | obj class byteSizeOfInstanceOfSize: obj basicSize]]).
^self elementsExchangeIdentityWith: otherArray].

Would renaming growMemoryByAtLeast: (and fixing its comment) make sense?

- Bert -
 
Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Igor Stasenko
In reply to this post by Eliot Miranda-2
 
Hi, Eliot,

why not trimming the object (if its size allows it) to mark rest of now-forwader object to be a freespace immediately? 
Like that you can avoid stressing memory too much by doubling amount of memory needed per each forwaded object involved in become operation.
And that, of course, if direct contents swap is not possible.

On 22 June 2016 at 16:44, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Jun 22, 2016, at 5:57 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

because that's how Spur works.  Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

Spur chooses to trade memory (creating the copies) over time (searching the entire heap looking fir references).  This is the right trade off until you start becoming objects that are substantial fractions of the entire heap in size.

In your case you could avoid the pain simply by making sure the steam had a wide String as contents in the first place.  I understand that may not be possible.


- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot





--
Best regards,
Igor Stasenko.
Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Eliot Miranda-2
In reply to this post by Bert Freudenberg
 
Hi Bert,


On Jun 23, 2016, at 4:11 AM, Bert Freudenberg <[hidden email]> wrote:

On Wed, Jun 22, 2016 at 10:15 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,
Do you have a reproducible case for me to look at?

Yes, from one email ago:

        (ByteString new: 20000000) become: (WideString new: 20000000)

 The fix I submitted fixes your original case.  But you could try and fix it yourself.  If the become: is two-way and fails with out-of-memory the failure code should ask to grow memory by the sum of the instance byte sizes.

The failure code in elementsExchangeIdentityWith: asks the memory to grow by a fixed amount and retry:

ec == #'insufficient object memory'
ifTrue: [Smalltalk garbageCollect < 1048576
ifTrue: [Smalltalk growMemoryByAtLeast: 1048576].
^ self elementsExchangeIdentityWith: otherArray].

After a few repetitions of this there should be enough room.

The problem is that growMemoryByAtLeast: does not grow the memory, it apparently only assures that the given amount is available. If that is indeed the intention of primitive 180, we should rename it to something like ensureMemoryAtLeastFree:.

That's not true.  growMemoryByAtLeast: does indeed grow memory, but does so by adding a segment.  So you don't get contiguous memory of a size that's the sum of extant free space and the amount grown.  You're only guaranteed to get contiguous memory of the size you grow memory by.  


 There us a bytesSzeOfInstance: method (or close) to compute the required space.

This seems to work, yes:

ec == #'insufficient object memory' ifTrue:
[Smalltalk ensureMemoryAtLeastFree: ({self. otherArray} detectSum:
[:array | array detectSum:
[:obj | obj class byteSizeOfInstanceOfSize: obj basicSize]]).
^self elementsExchangeIdentityWith: otherArray].

Would renaming growMemoryByAtLeast: (and fixing its comment) make sense?

I'll fix growMemoryByAtLeast:'s comment.  But commit the above, provided you use growMemoryByAtLeast:.


- Bert -
 
Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Eliot Miranda-2
In reply to this post by Igor Stasenko
 
Hi Igor,

On Jun 23, 2016, at 7:34 AM, Igor Stasenko <[hidden email]> wrote:

Hi, Eliot,

why not trimming the object (if its size allows it) to mark rest of now-forwader object to be a freespace immediately? 
Like that you can avoid stressing memory too much by doubling amount of memory needed per each forwaded object involved in become operation.
And that, of course, if direct contents swap is not possible.

Agreed.  Would you like to have a go and I'll review it?  The problem with truncating short objects is that Spur's memory manager cannot represent a free chunk of only 8 bytes in length.  The minimum sized free chunk (& minimum sized object) is 16 bytes.  And any size higher (24, 32, 40 etc) is ok.  So if you want to shorten a 24 byte object to 16 bytes because it's been turned into a forwarder, you can't because that would leave an 8-byte sliver.  So be sure to check that the object is > 24 bytes long before trying to truncate it.


On 22 June 2016 at 16:44, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Jun 22, 2016, at 5:57 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

because that's how Spur works.  Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

Spur chooses to trade memory (creating the copies) over time (searching the entire heap looking fir references).  This is the right trade off until you start becoming objects that are substantial fractions of the entire heap in size.

In your case you could avoid the pain simply by making sure the steam had a wide String as contents in the first place.  I understand that may not be possible.


- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot





--
Best regards,
Igor Stasenko.
Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Igor Stasenko
 


On 23 June 2016 at 18:00, Eliot Miranda <[hidden email]> wrote:
 
Hi Igor,

On Jun 23, 2016, at 7:34 AM, Igor Stasenko <[hidden email]> wrote:

Hi, Eliot,

why not trimming the object (if its size allows it) to mark rest of now-forwader object to be a freespace immediately? 
Like that you can avoid stressing memory too much by doubling amount of memory needed per each forwaded object involved in become operation.
And that, of course, if direct contents swap is not possible.

Agreed.  Would you like to have a go and I'll review it?  The problem with truncating short objects is that Spur's memory manager cannot represent a free chunk of only 8 bytes in length.  The minimum sized free chunk (& minimum sized object) is 16 bytes.  And any size higher (24, 32, 40 etc) is ok.  So if you want to shorten a 24 byte object to 16 bytes because it's been turned into a forwarder, you can't because that would leave an 8-byte sliver.  So be sure to check that the object is > 24 bytes long before trying to truncate it.
 
That's why i said - if its size allows it.
I could give it try.. Of course i will need some more details, to do it fast, like updating a first-free object pointer, if needed and what limitations there, and things like if it safe to throw this newly-free object space into a soup of free chunks regardless of their location in memory (old heap/eden etc)

I would be glad, if you can give me quick brief of these potential pitfall, since i don't know much about new spur memory model innards.



On 22 June 2016 at 16:44, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Jun 22, 2016, at 5:57 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

because that's how Spur works.  Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

Spur chooses to trade memory (creating the copies) over time (searching the entire heap looking fir references).  This is the right trade off until you start becoming objects that are substantial fractions of the entire heap in size.

In your case you could avoid the pain simply by making sure the steam had a wide String as contents in the first place.  I understand that may not be possible.


- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot





--
Best regards,
Igor Stasenko.




--
Best regards,
Igor Stasenko.
Reply | Threaded
Open this post in threaded view
|

Re: [Spur] endless recursion in forward become

Eliot Miranda-2
 
Hi Igor,

On Thu, Jun 23, 2016 at 8:11 AM, Igor Stasenko <[hidden email]> wrote:
 


On 23 June 2016 at 18:00, Eliot Miranda <[hidden email]> wrote:
 
Hi Igor,

On Jun 23, 2016, at 7:34 AM, Igor Stasenko <[hidden email]> wrote:

Hi, Eliot,

why not trimming the object (if its size allows it) to mark rest of now-forwader object to be a freespace immediately? 
Like that you can avoid stressing memory too much by doubling amount of memory needed per each forwaded object involved in become operation.
And that, of course, if direct contents swap is not possible.

Agreed.  Would you like to have a go and I'll review it?  The problem with truncating short objects is that Spur's memory manager cannot represent a free chunk of only 8 bytes in length.  The minimum sized free chunk (& minimum sized object) is 16 bytes.  And any size higher (24, 32, 40 etc) is ok.  So if you want to shorten a 24 byte object to 16 bytes because it's been turned into a forwarder, you can't because that would leave an 8-byte sliver.  So be sure to check that the object is > 24 bytes long before trying to truncate it.
 
That's why i said - if its size allows it.
I could give it try.. Of course i will need some more details, to do it fast, like updating a first-free object pointer, if needed and what limitations there, and things like if it safe to throw this newly-free object space into a soup of free chunks regardless of their location in memory (old heap/eden etc)

See SpurMemoryManager>>shorten:toIndexableSize: for a utility that is close to what you need.
 

I would be glad, if you can give me quick brief of these potential pitfall, since i don't know much about new spur memory model innards.

I think the sliver problem is the only pitfall.  Basically follow shorten:toIndexableSize: to see what to do with the remnant.  Good luck!

On 22 June 2016 at 16:44, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,


On Jun 22, 2016, at 5:57 AM, Bert Freudenberg <[hidden email]> wrote:

Hi Eliot,

the become-forward works now, indeed.

But why is it different with the swapping become? This still gets into an endless recursion:

(ByteString new: 20000000) become: (WideString new: 20000000)

Why is it creating copies at all?

because that's how Spur works.  Become is lazy; the two objects are turned into forwarders to copies of each other.  If the two objects have the same byte size as heap objects (rounded up to 8 byte units) their contents can be exchanged.  If they are not, two copies must be created and the two originals turned into forwarders to the copies.  Remember the slides from my Cambridge talk.

Spur chooses to trade memory (creating the copies) over time (searching the entire heap looking fir references).  This is the right trade off until you start becoming objects that are substantial fractions of the entire heap in size.

In your case you could avoid the pain simply by making sure the steam had a wide String as contents in the first place.  I understand that may not be possible.


- Bert -

On Mon, Jun 20, 2016 at 10:36 PM, Eliot Miranda <[hidden email]> wrote:
 
Hi Bert,

    it was a regression in argument validation for become introduced in fixing argument validation for the one-way copy hash case.  The code was erroneously checking that it had space to create copies of the input arguments, even though it was a one-way become.  It's a copyHash become and that confused it.  It's now fixed.  I'll generate sources and push and new builds should appear shortly ;-)

On Thu, Jun 16, 2016 at 5:11 AM, Bert Freudenberg <[hidden email]> wrote:
 
I'm reading a 20MB text file. At some point it tries to convert the 20 MB ByteString into an 80 MB WideString using forward become. This fails resulting in an endless loop. The primitive failure code for elementsForwardIdentityTo: (primitive 72) is  #'insufficient object memory' and it tries again after growing:

ec == #'insufficient object memory' ifTrue:
[Smalltalk garbageCollect < 1048576 ifTrue:
[Smalltalk growMemoryByAtLeast: 1048576].
^self elementsForwardIdentityTo: otherArray].

but the growMemoryByAtLeast: does not actually grow the memory:

{Smalltalk garbageCollect.
Smalltalk growMemoryByAtLeast: 1048576.
Smalltalk garbageCollect.}

==>  #(58576608 16777216 58576608)

58 MB are not enough, but it doesn't grow any further.

The question is, why does it try to allocate a new object at all? The WideString exists already. I thought Spur would simply install a forwarder?

To reproduce, execute 

(ByteString new: 20000000) writeStream nextPut: (Character value: 128169)

You probably want to follow that with a user interrupt (Cmd-period).

This happens using the latest Spur VM from github (r201606160944)

- Bert -




--
_,,,^..^,,,_
best, Eliot





--
Best regards,
Igor Stasenko.




--
Best regards,
Igor Stasenko.




--
_,,,^..^,,,_
best, Eliot