Ephemeron bug

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Ephemeron bug

Javier Burroni
Consider the following code:

testNestedEphemeronsWEphemeronKey
   | toBeRescued |
   toBeRescued := EphemeronWRescueMemory
       key: Object new
       value: (EphemeronWRescueMemory key: Object new).
   Smalltalk unusedMemory.
   Ephemeron processRescuedEphemerons.
   Smalltalk unusedMemory.
   self
       assert: toBeRescued hasBeenRescued;
       deny: toBeRescued value key methodDictionaryArray isNil;
       assert: toBeRescued value hasBeenRescued

Where EphemeronWRescueMemory is an Ephemeron's subclass which can
answer if it has been already rescued.
This code will break smalltalk.
The reason for this test to fail, is the existence of an ephemeron (B)
as other ephemeron's value (A).
Following Hayes, whenever ephemeron A rescued, it should be traced as
any other object -loosing the idea of ephemerons, neither in a direct
or indirect way-.
However, the apparent behavior when following object A and finding
ephemerons, is to mark and queue the just found ephemeron -B in this
case-; but never trace the ephemeron -B-. This makes B's reference to
be out of date, and (potentially) point to non valid objects.
In Hayes code:

Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointerQueueingEphemerons.
           ephemeron deref valueField tracePointerQueueingEphemerons].

When the correct code should be:
Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointer.
           ephemeron deref valueField tracePointer].

There are an interesting point regarding Hayes work: the first time a
GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark
all of the objects reachable from this pointer, paying no attention to
ephemerons" p. 183 [1]).
With a small modification, the algorithm may be able to handle object
B as an ephemeron in the first run. To do so, we have to focus on the
fundamental idea of ephemerons: the topological relation between
objects and the ephemeron's key/value. The relation between objects
and the ephemeron itself shall be of no importance.

So, suppose we have this:
VMGarbageCollector>>#rescueEphemerons
   | unknowns rescan rescued |
   rescued := rescan := false.
   unknowns := ephemeronsStack.
   [ephemerons isEmpty] whileFalse: [
       rescan := self followEphemeronsCollectingIn: unknowns.
       rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [
           unknowns do: [:ephemeron |
               self rescueEphemeron: ephemeron.
               rescued := true]].
       unknowns reset].
   rescued ifTrue: [self someEphemeronsRescued]


VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns
   | rescan |
   rescan := false.
   [ephemerons isEmpty] whileFalse: [| ephemeron |
       ephemeron := ephemerons pop.
       ephemeron == nil ifFalse: [
           (self checkReachablePropertyOf: ephemeron)
               ifTrue: [
                   self follow: ephemeron count: ephemeron
_extendedSize startingAt: 1.
                   rescan := true]
               ifFalse: [unknowns add: ephemeron]]].
   ^rescan

VMGarbageCollector>>#rescueEphemeron: ephemeron
   ephemeron _haveNoWeaks
   self follow: ephemeron count: ephemeron _extendedSize startingAt: 1.
   rescuedEphemerons add: ephemeron

This implementation may fail to detect object as 'almost collectable',
whenever a rescued ephemeron refers to an object, when this object is
the key of a non yet seen ephemeron. But the set of correctly rescued
ephemerons is big than the Hayes set (which is a subset)

/jb

[1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism.
Proceedings of the 12th ACM SIGPLAN conference on Object-oriented
programming, systems, languages, and applications.
http://portal.acm.org/citation.cfm?id=263733

--
" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: Ephemeron bug

Leandro Caniglia
Javier,

Have you checked if this issue also happens in other dialects?

/Leandro

On Tue, Apr 26, 2011 at 1:38 PM, Javier Burroni <[hidden email]> wrote:
Consider the following code:

testNestedEphemeronsWEphemeronKey
  | toBeRescued |
  toBeRescued := EphemeronWRescueMemory
      key: Object new
      value: (EphemeronWRescueMemory key: Object new).
  Smalltalk unusedMemory.
  Ephemeron processRescuedEphemerons.
  Smalltalk unusedMemory.
  self
      assert: toBeRescued hasBeenRescued;
      deny: toBeRescued value key methodDictionaryArray isNil;
      assert: toBeRescued value hasBeenRescued

Where EphemeronWRescueMemory is an Ephemeron's subclass which can
answer if it has been already rescued.
This code will break smalltalk.
The reason for this test to fail, is the existence of an ephemeron (B)
as other ephemeron's value (A).
Following Hayes, whenever ephemeron A rescued, it should be traced as
any other object -loosing the idea of ephemerons, neither in a direct
or indirect way-.
However, the apparent behavior when following object A and finding
ephemerons, is to mark and queue the just found ephemeron -B in this
case-; but never trace the ephemeron -B-. This makes B's reference to
be out of date, and (potentially) point to non valid objects.
In Hayes code:

Heap::markPhase3
  EphemeronQueue enumerate:[:ephemeron|
          ephemeron deref signal: almostCollectable.
          ephemeron deref keyField tracePointerQueueingEphemerons.
          ephemeron deref valueField tracePointerQueueingEphemerons].

When the correct code should be:
Heap::markPhase3
  EphemeronQueue enumerate:[:ephemeron|
          ephemeron deref signal: almostCollectable.
          ephemeron deref keyField tracePointer.
          ephemeron deref valueField tracePointer].

There are an interesting point regarding Hayes work: the first time a
GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark
all of the objects reachable from this pointer, paying no attention to
ephemerons" p. 183 [1]).
With a small modification, the algorithm may be able to handle object
B as an ephemeron in the first run. To do so, we have to focus on the
fundamental idea of ephemerons: the topological relation between
objects and the ephemeron's key/value. The relation between objects
and the ephemeron itself shall be of no importance.

So, suppose we have this:
VMGarbageCollector>>#rescueEphemerons
  | unknowns rescan rescued |
  rescued := rescan := false.
  unknowns := ephemeronsStack.
  [ephemerons isEmpty] whileFalse: [
      rescan := self followEphemeronsCollectingIn: unknowns.
      rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [
          unknowns do: [:ephemeron |
              self rescueEphemeron: ephemeron.
              rescued := true]].
      unknowns reset].
  rescued ifTrue: [self someEphemeronsRescued]


VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns
  | rescan |
  rescan := false.
  [ephemerons isEmpty] whileFalse: [| ephemeron |
      ephemeron := ephemerons pop.
      ephemeron == nil ifFalse: [
          (self checkReachablePropertyOf: ephemeron)
              ifTrue: [
                  self follow: ephemeron count: ephemeron
_extendedSize startingAt: 1.
                  rescan := true]
              ifFalse: [unknowns add: ephemeron]]].
  ^rescan

VMGarbageCollector>>#rescueEphemeron: ephemeron
  ephemeron _haveNoWeaks
  self follow: ephemeron count: ephemeron _extendedSize startingAt: 1.
  rescuedEphemerons add: ephemeron

This implementation may fail to detect object as 'almost collectable',
whenever a rescued ephemeron refers to an object, when this object is
the key of a non yet seen ephemeron. But the set of correctly rescued
ephemerons is big than the Hayes set (which is a subset)

/jb

[1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism.
Proceedings of the 12th ACM SIGPLAN conference on Object-oriented
programming, systems, languages, and applications.
http://portal.acm.org/citation.cfm?id=263733

--
" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***
Reply | Threaded
Open this post in threaded view
|

AW: Ephemeron bug

frank.lesser
In reply to this post by Javier Burroni
Hi,

This is not a bug - it is a restriction how Ephemerons should not be used &
are not used.
However there are a few bugs in the current (? I refer to VSE pre 2K)
implementation & plenty of bugs in the VSE framework regarding finalization.

Frank

-----Ursprüngliche Nachricht-----
Von: Using Visual Smalltalk for Windows/Enterprise
[mailto:[hidden email]] Im Auftrag von Javier Burroni
Gesendet: Dienstag, 26. April 2011 18:39
An: [hidden email]
Betreff: Ephemeron bug

Consider the following code:

testNestedEphemeronsWEphemeronKey
   | toBeRescued |
   toBeRescued := EphemeronWRescueMemory
       key: Object new
       value: (EphemeronWRescueMemory key: Object new).
   Smalltalk unusedMemory.
   Ephemeron processRescuedEphemerons.
   Smalltalk unusedMemory.
   self
       assert: toBeRescued hasBeenRescued;
       deny: toBeRescued value key methodDictionaryArray isNil;
       assert: toBeRescued value hasBeenRescued

Where EphemeronWRescueMemory is an Ephemeron's subclass which can
answer if it has been already rescued.
This code will break smalltalk.
The reason for this test to fail, is the existence of an ephemeron (B)
as other ephemeron's value (A).
Following Hayes, whenever ephemeron A rescued, it should be traced as
any other object -loosing the idea of ephemerons, neither in a direct
or indirect way-.
However, the apparent behavior when following object A and finding
ephemerons, is to mark and queue the just found ephemeron -B in this
case-; but never trace the ephemeron -B-. This makes B's reference to
be out of date, and (potentially) point to non valid objects.
In Hayes code:

Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointerQueueingEphemerons.
           ephemeron deref valueField tracePointerQueueingEphemerons].

When the correct code should be:
Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointer.
           ephemeron deref valueField tracePointer].

There are an interesting point regarding Hayes work: the first time a
GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark
all of the objects reachable from this pointer, paying no attention to
ephemerons" p. 183 [1]).
With a small modification, the algorithm may be able to handle object
B as an ephemeron in the first run. To do so, we have to focus on the
fundamental idea of ephemerons: the topological relation between
objects and the ephemeron's key/value. The relation between objects
and the ephemeron itself shall be of no importance.

So, suppose we have this:
VMGarbageCollector>>#rescueEphemerons
   | unknowns rescan rescued |
   rescued := rescan := false.
   unknowns := ephemeronsStack.
   [ephemerons isEmpty] whileFalse: [
       rescan := self followEphemeronsCollectingIn: unknowns.
       rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [
           unknowns do: [:ephemeron |
               self rescueEphemeron: ephemeron.
               rescued := true]].
       unknowns reset].
   rescued ifTrue: [self someEphemeronsRescued]


VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns
   | rescan |
   rescan := false.
   [ephemerons isEmpty] whileFalse: [| ephemeron |
       ephemeron := ephemerons pop.
       ephemeron == nil ifFalse: [
           (self checkReachablePropertyOf: ephemeron)
               ifTrue: [
                   self follow: ephemeron count: ephemeron
_extendedSize startingAt: 1.
                   rescan := true]
               ifFalse: [unknowns add: ephemeron]]].
   ^rescan

VMGarbageCollector>>#rescueEphemeron: ephemeron
   ephemeron _haveNoWeaks
   self follow: ephemeron count: ephemeron _extendedSize startingAt: 1.
   rescuedEphemerons add: ephemeron

This implementation may fail to detect object as 'almost collectable',
whenever a rescued ephemeron refers to an object, when this object is
the key of a non yet seen ephemeron. But the set of correctly rescued
ephemerons is big than the Hayes set (which is a subset)

/jb

[1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism.
Proceedings of the 12th ACM SIGPLAN conference on Object-oriented
programming, systems, languages, and applications.
http://portal.acm.org/citation.cfm?id=263733

--
" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: Ephemeron bug

Javier Burroni
In reply to this post by Leandro Caniglia
Apparently, the issue is not present in VW

On Tue, Apr 26, 2011 at 2:22 PM, Leandro Caniglia <[hidden email]> wrote:

> Javier,
> Have you checked if this issue also happens in other dialects?
> /Leandro
>
> On Tue, Apr 26, 2011 at 1:38 PM, Javier Burroni <[hidden email]>
> wrote:
>>
>> Consider the following code:
>>
>> testNestedEphemeronsWEphemeronKey
>>   | toBeRescued |
>>   toBeRescued := EphemeronWRescueMemory
>>       key: Object new
>>       value: (EphemeronWRescueMemory key: Object new).
>>   Smalltalk unusedMemory.
>>   Ephemeron processRescuedEphemerons.
>>   Smalltalk unusedMemory.
>>   self
>>       assert: toBeRescued hasBeenRescued;
>>       deny: toBeRescued value key methodDictionaryArray isNil;
>>       assert: toBeRescued value hasBeenRescued
>>
>> Where EphemeronWRescueMemory is an Ephemeron's subclass which can
>> answer if it has been already rescued.
>> This code will break smalltalk.
>> The reason for this test to fail, is the existence of an ephemeron (B)
>> as other ephemeron's value (A).
>> Following Hayes, whenever ephemeron A rescued, it should be traced as
>> any other object -loosing the idea of ephemerons, neither in a direct
>> or indirect way-.
>> However, the apparent behavior when following object A and finding
>> ephemerons, is to mark and queue the just found ephemeron -B in this
>> case-; but never trace the ephemeron -B-. This makes B's reference to
>> be out of date, and (potentially) point to non valid objects.
>> In Hayes code:
>>
>> Heap::markPhase3
>>   EphemeronQueue enumerate:[:ephemeron|
>>           ephemeron deref signal: almostCollectable.
>>           ephemeron deref keyField tracePointerQueueingEphemerons.
>>           ephemeron deref valueField tracePointerQueueingEphemerons].
>>
>> When the correct code should be:
>> Heap::markPhase3
>>   EphemeronQueue enumerate:[:ephemeron|
>>           ephemeron deref signal: almostCollectable.
>>           ephemeron deref keyField tracePointer.
>>           ephemeron deref valueField tracePointer].
>>
>> There are an interesting point regarding Hayes work: the first time a
>> GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark
>> all of the objects reachable from this pointer, paying no attention to
>> ephemerons" p. 183 [1]).
>> With a small modification, the algorithm may be able to handle object
>> B as an ephemeron in the first run. To do so, we have to focus on the
>> fundamental idea of ephemerons: the topological relation between
>> objects and the ephemeron's key/value. The relation between objects
>> and the ephemeron itself shall be of no importance.
>>
>> So, suppose we have this:
>> VMGarbageCollector>>#rescueEphemerons
>>   | unknowns rescan rescued |
>>   rescued := rescan := false.
>>   unknowns := ephemeronsStack.
>>   [ephemerons isEmpty] whileFalse: [
>>       rescan := self followEphemeronsCollectingIn: unknowns.
>>       rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [
>>           unknowns do: [:ephemeron |
>>               self rescueEphemeron: ephemeron.
>>               rescued := true]].
>>       unknowns reset].
>>   rescued ifTrue: [self someEphemeronsRescued]
>>
>>
>> VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns
>>   | rescan |
>>   rescan := false.
>>   [ephemerons isEmpty] whileFalse: [| ephemeron |
>>       ephemeron := ephemerons pop.
>>       ephemeron == nil ifFalse: [
>>           (self checkReachablePropertyOf: ephemeron)
>>               ifTrue: [
>>                   self follow: ephemeron count: ephemeron
>> _extendedSize startingAt: 1.
>>                   rescan := true]
>>               ifFalse: [unknowns add: ephemeron]]].
>>   ^rescan
>>
>> VMGarbageCollector>>#rescueEphemeron: ephemeron
>>   ephemeron _haveNoWeaks
>>   self follow: ephemeron count: ephemeron _extendedSize startingAt: 1.
>>   rescuedEphemerons add: ephemeron
>>
>> This implementation may fail to detect object as 'almost collectable',
>> whenever a rescued ephemeron refers to an object, when this object is
>> the key of a non yet seen ephemeron. But the set of correctly rescued
>> ephemerons is big than the Hayes set (which is a subset)
>>
>> /jb
>>
>> [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism.
>> Proceedings of the 12th ACM SIGPLAN conference on Object-oriented
>> programming, systems, languages, and applications.
>> http://portal.acm.org/citation.cfm?id=263733
>>
>> --
>> " To be is to do " ( Socrates )
>> " To be or not to be " ( Shakespeare )
>> " To do is to be " ( Sartre )
>> " Do be do be do " ( Sinatra )
>>
>> ***           this signature added by listserv             ***
>> *** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
>> *** for archive browsing and VSWE-L membership management  ***
>
> *** this signature added by listserv *** *** Visit
> http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing
> and VSWE-L membership management ***



--
" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: Ephemeron bug

Javier Burroni
In reply to this post by frank.lesser
I've used the word 'bug' because the 'member of' relation produce a
crash, which should not be the case. (Besides the should and shouldn´t
in the image space)

On Tue, Apr 26, 2011 at 3:20 PM, Frank Lesser
<[hidden email]> wrote:

> Hi,
>
> This is not a bug - it is a restriction how Ephemerons should not be used &
> are not used.
> However there are a few bugs in the current (? I refer to VSE pre 2K)
> implementation & plenty of bugs in the VSE framework regarding finalization.
>
> Frank
>
> -----Ursprüngliche Nachricht-----
> Von: Using Visual Smalltalk for Windows/Enterprise
> [mailto:[hidden email]] Im Auftrag von Javier Burroni
> Gesendet: Dienstag, 26. April 2011 18:39
> An: [hidden email]
> Betreff: Ephemeron bug
>
> Consider the following code:
>
> testNestedEphemeronsWEphemeronKey
>   | toBeRescued |
>   toBeRescued := EphemeronWRescueMemory
>       key: Object new
>       value: (EphemeronWRescueMemory key: Object new).
>   Smalltalk unusedMemory.
>   Ephemeron processRescuedEphemerons.
>   Smalltalk unusedMemory.
>   self
>       assert: toBeRescued hasBeenRescued;
>       deny: toBeRescued value key methodDictionaryArray isNil;
>       assert: toBeRescued value hasBeenRescued
>
> Where EphemeronWRescueMemory is an Ephemeron's subclass which can
> answer if it has been already rescued.
> This code will break smalltalk.
> The reason for this test to fail, is the existence of an ephemeron (B)
> as other ephemeron's value (A).
> Following Hayes, whenever ephemeron A rescued, it should be traced as
> any other object -loosing the idea of ephemerons, neither in a direct
> or indirect way-.
> However, the apparent behavior when following object A and finding
> ephemerons, is to mark and queue the just found ephemeron -B in this
> case-; but never trace the ephemeron -B-. This makes B's reference to
> be out of date, and (potentially) point to non valid objects.
> In Hayes code:
>
> Heap::markPhase3
>   EphemeronQueue enumerate:[:ephemeron|
>           ephemeron deref signal: almostCollectable.
>           ephemeron deref keyField tracePointerQueueingEphemerons.
>           ephemeron deref valueField tracePointerQueueingEphemerons].
>
> When the correct code should be:
> Heap::markPhase3
>   EphemeronQueue enumerate:[:ephemeron|
>           ephemeron deref signal: almostCollectable.
>           ephemeron deref keyField tracePointer.
>           ephemeron deref valueField tracePointer].
>
> There are an interesting point regarding Hayes work: the first time a
> GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark
> all of the objects reachable from this pointer, paying no attention to
> ephemerons" p. 183 [1]).
> With a small modification, the algorithm may be able to handle object
> B as an ephemeron in the first run. To do so, we have to focus on the
> fundamental idea of ephemerons: the topological relation between
> objects and the ephemeron's key/value. The relation between objects
> and the ephemeron itself shall be of no importance.
>
> So, suppose we have this:
> VMGarbageCollector>>#rescueEphemerons
>   | unknowns rescan rescued |
>   rescued := rescan := false.
>   unknowns := ephemeronsStack.
>   [ephemerons isEmpty] whileFalse: [
>       rescan := self followEphemeronsCollectingIn: unknowns.
>       rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [
>           unknowns do: [:ephemeron |
>               self rescueEphemeron: ephemeron.
>               rescued := true]].
>       unknowns reset].
>   rescued ifTrue: [self someEphemeronsRescued]
>
>
> VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns
>   | rescan |
>   rescan := false.
>   [ephemerons isEmpty] whileFalse: [| ephemeron |
>       ephemeron := ephemerons pop.
>       ephemeron == nil ifFalse: [
>           (self checkReachablePropertyOf: ephemeron)
>               ifTrue: [
>                   self follow: ephemeron count: ephemeron
> _extendedSize startingAt: 1.
>                   rescan := true]
>               ifFalse: [unknowns add: ephemeron]]].
>   ^rescan
>
> VMGarbageCollector>>#rescueEphemeron: ephemeron
>   ephemeron _haveNoWeaks
>   self follow: ephemeron count: ephemeron _extendedSize startingAt: 1.
>   rescuedEphemerons add: ephemeron
>
> This implementation may fail to detect object as 'almost collectable',
> whenever a rescued ephemeron refers to an object, when this object is
> the key of a non yet seen ephemeron. But the set of correctly rescued
> ephemerons is big than the Hayes set (which is a subset)
>
> /jb
>
> [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism.
> Proceedings of the 12th ACM SIGPLAN conference on Object-oriented
> programming, systems, languages, and applications.
> http://portal.acm.org/citation.cfm?id=263733
>
> --
> " To be is to do " ( Socrates )
> " To be or not to be " ( Shakespeare )
> " To do is to be " ( Sartre )
> " Do be do be do " ( Sinatra )
>
> ***           this signature added by listserv             ***
> *** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
> *** for archive browsing and VSWE-L membership management  ***
>
> ***           this signature added by listserv             ***
> *** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
> *** for archive browsing and VSWE-L membership management  ***
>



--
" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: Ephemeron bug

Carl Gundel-3
In reply to this post by Javier Burroni
This is interesting, but how would the bug exhibit itself in an application
day to day?

-Carl

-----Original Message-----
From: Using Visual Smalltalk for Windows/Enterprise
[mailto:[hidden email]] On Behalf Of Javier Burroni
Sent: Tuesday, April 26, 2011 12:39 PM
To: [hidden email]
Subject: Ephemeron bug

Consider the following code:

testNestedEphemeronsWEphemeronKey
   | toBeRescued |
   toBeRescued := EphemeronWRescueMemory
       key: Object new
       value: (EphemeronWRescueMemory key: Object new).
   Smalltalk unusedMemory.
   Ephemeron processRescuedEphemerons.
   Smalltalk unusedMemory.
   self
       assert: toBeRescued hasBeenRescued;
       deny: toBeRescued value key methodDictionaryArray isNil;
       assert: toBeRescued value hasBeenRescued

Where EphemeronWRescueMemory is an Ephemeron's subclass which can
answer if it has been already rescued.
This code will break smalltalk.
The reason for this test to fail, is the existence of an ephemeron (B)
as other ephemeron's value (A).
Following Hayes, whenever ephemeron A rescued, it should be traced as
any other object -loosing the idea of ephemerons, neither in a direct
or indirect way-.
However, the apparent behavior when following object A and finding
ephemerons, is to mark and queue the just found ephemeron -B in this
case-; but never trace the ephemeron -B-. This makes B's reference to
be out of date, and (potentially) point to non valid objects.
In Hayes code:

Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointerQueueingEphemerons.
           ephemeron deref valueField tracePointerQueueingEphemerons].

When the correct code should be:
Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointer.
           ephemeron deref valueField tracePointer].

There are an interesting point regarding Hayes work: the first time a
GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark
all of the objects reachable from this pointer, paying no attention to
ephemerons" p. 183 [1]).
With a small modification, the algorithm may be able to handle object
B as an ephemeron in the first run. To do so, we have to focus on the
fundamental idea of ephemerons: the topological relation between
objects and the ephemeron's key/value. The relation between objects
and the ephemeron itself shall be of no importance.

So, suppose we have this:
VMGarbageCollector>>#rescueEphemerons
   | unknowns rescan rescued |
   rescued := rescan := false.
   unknowns := ephemeronsStack.
   [ephemerons isEmpty] whileFalse: [
       rescan := self followEphemeronsCollectingIn: unknowns.
       rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [
           unknowns do: [:ephemeron |
               self rescueEphemeron: ephemeron.
               rescued := true]].
       unknowns reset].
   rescued ifTrue: [self someEphemeronsRescued]


VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns
   | rescan |
   rescan := false.
   [ephemerons isEmpty] whileFalse: [| ephemeron |
       ephemeron := ephemerons pop.
       ephemeron == nil ifFalse: [
           (self checkReachablePropertyOf: ephemeron)
               ifTrue: [
                   self follow: ephemeron count: ephemeron
_extendedSize startingAt: 1.
                   rescan := true]
               ifFalse: [unknowns add: ephemeron]]].
   ^rescan

VMGarbageCollector>>#rescueEphemeron: ephemeron
   ephemeron _haveNoWeaks
   self follow: ephemeron count: ephemeron _extendedSize startingAt: 1.
   rescuedEphemerons add: ephemeron

This implementation may fail to detect object as 'almost collectable',
whenever a rescued ephemeron refers to an object, when this object is
the key of a non yet seen ephemeron. But the set of correctly rescued
ephemerons is big than the Hayes set (which is a subset)

/jb

[1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism.
Proceedings of the 12th ACM SIGPLAN conference on Object-oriented
programming, systems, languages, and applications.
http://portal.acm.org/citation.cfm?id=263733

--
" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

ExternalHeapHandle freeAddress:size: crash

Carl Gundel-3
In reply to this post by Javier Burroni
Anyone know what to make of this?

KernelDLL(Object)>>vmInterrupt: <#protectionViolationInterrupt>
KernelDLL(DynamicLinkLibrary)>>invalidArgument
KernelDLL>>heapFree: <anExternalHeapHandle> flags: <0> address:
KernelDLL>><aByteArray>
ExternalHeapHandle>>freeAddress: <anExternalHeapAddress> size: <4>
ExternalHeapAddress>>releaseMemory
ExternalHeapAddress(ExternalAddress)>>release
ExternalHeapAddress(ExternalAddress)>>finalize
[] in SystemWeakRegistries>>initialize
BlockClosure>>evaluateWithArguments: <anArray>
WeakRegistry>>rescue: <anEphemeron>
Ephemeron>>rescue
[] in Ephemeron class>>processRescuedEphemerons

Thanks,

-Carl

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Todor Todorov
Heap Corruption!

Use the "Global Flags" app in the Debug Tools for Windows to diagnose this.

-----Original Message-----
From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Carl Gundel
Sent: 28. april 2011 19:09
To: [hidden email]
Subject: ExternalHeapHandle freeAddress:size: crash

Anyone know what to make of this?

KernelDLL(Object)>>vmInterrupt: <#protectionViolationInterrupt> KernelDLL(DynamicLinkLibrary)>>invalidArgument
KernelDLL>>heapFree: <anExternalHeapHandle> flags: <0> address:
KernelDLL>><aByteArray>
ExternalHeapHandle>>freeAddress: <anExternalHeapAddress> size: <4>
ExternalHeapAddress>>releaseMemory
ExternalHeapAddress(ExternalAddress)>>release
ExternalHeapAddress(ExternalAddress)>>finalize
[] in SystemWeakRegistries>>initialize
BlockClosure>>evaluateWithArguments: <anArray>
WeakRegistry>>rescue: <anEphemeron>
Ephemeron>>rescue
[] in Ephemeron class>>processRescuedEphemerons

Thanks,

-Carl

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

frank.lesser
The problem is different. It is Smalltalk which produces the problem - hint
few emails back. It surprises me that it appears here now. It is referred
here internally as the TR problem - the whole base system, GUI & OLE is
affected.

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Gerardo Richarte
On 04/28/2011 05:08 PM, Frank Lesser wrote:
> The problem is different. It is Smalltalk which produces the problem - hint
> few emails back. It surprises me that it appears here now. It is referred
> here internally as the TR problem - the whole base system, GUI & OLE is
> affected.
What do you mean? could you share any extra info so we lean to spot and
solve the problem? thanks!

    gera

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Gerardo Richarte
In reply to this post by Carl Gundel-3
As Todor said, this looks a lot like a Heap Corruption bug (which could
turn into a security bug, but we are not really worried about that).

The situation here is that an ExternalHeapAddress is finalized, and that
should be correct. When it's finalized the memory is returned to the os
(HeapFree()), and here's where we have the crash. There are several
situations that might lead to this type of crashes:

1. The address argument to HeapFree() is incorrect. You should be able
to see that in Smalltalk, place a breakpoint in #freeAddress:size: and
see if the contents of the ByteArray look like a valid heap address (it
could be tricky to break on the offending #freeAddress:size, and tricky
also to know if the address is valid. At any cost, the address inside
the ByteArray must be the result of calling HeapAllocate())

2. The memory in the heap might have been overwritten, and hence the
heap structure corrupted, and hence HeapFree() doesn't know what to do
about it, and panics, and crashes. It's not easy to spot this, but if
you had a native debugger attached to vdevw.exe you could place a
hardware breakpoint on write on the bytes around the offending heap
buffer. This breakpoint should only halt when calling HeapFree() or
HeapAlloc(), otherwise it's a bug.

3. Double free. Calling HeapFree() twice on the same address, even if
the address is correct the first time, could produce a crash. This is
kind of easy to trigger from within Smtalltalk, for example, if you use
#needsFinalization more than once on the same
Object(ExternalHeapAddress). Or if you copy the ExternalHeapAddress and
then do #needsFinalization on the copy, or something like that. This
should be easy to check and stop, if only adding code to
#needsFinalization to check if the object is already in the registry.
(or actually making the registry a Set?)

    oh well... I hope this helps a bit at least, let us know what happened!

    gera

On 04/28/2011 02:09 PM, Carl Gundel wrote:

> Anyone know what to make of this?
>
> KernelDLL(Object)>>vmInterrupt: <#protectionViolationInterrupt>
> KernelDLL(DynamicLinkLibrary)>>invalidArgument
> KernelDLL>>heapFree: <anExternalHeapHandle> flags: <0> address:
> KernelDLL>><aByteArray>
> ExternalHeapHandle>>freeAddress: <anExternalHeapAddress> size: <4>
> ExternalHeapAddress>>releaseMemory
> ExternalHeapAddress(ExternalAddress)>>release
> ExternalHeapAddress(ExternalAddress)>>finalize
> [] in SystemWeakRegistries>>initialize
> BlockClosure>>evaluateWithArguments: <anArray>
> WeakRegistry>>rescue: <anEphemeron>
> Ephemeron>>rescue
> [] in Ephemeron class>>processRescuedEphemerons

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Gerardo Richarte
On 04/28/2011 05:32 PM, Gerardo Richarte wrote:
> 3. Double free. Calling HeapFree() twice on the same address, even if
> the address is correct the first time, could produce a crash. This is
> kind of easy to trigger from within Smtalltalk, for example, if you use
> #needsFinalization more than once on the same
> Object(ExternalHeapAddress).
not correct, sorry... the registry uses am EphemeronDictionary, so it
won't be finalized twice unless you play some tricks (like making a copy
of the ExternalHeapAddress or something similar)

    gera

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

AW: ExternalHeapHandle freeAddress:size: crash

frank.lesser
In reply to this post by Gerardo Richarte
Hi Gerardo,

No I am afraid no I can't - I have given my loved old community the maximum
hint I can do without making trouble to me. I am sure that you will discover
the problem sooner or later yourself. I see that you have gained a lot of
knowledge about Digitalk St - I didn't check other St's (lack of time) but I
am sure the issue is present there also - maybe because there are common
roots.

Frank


 Nachricht-----
Von: Using Visual Smalltalk for Windows/Enterprise
[mailto:[hidden email]] Im Auftrag von Gerardo Richarte
Gesendet: Donnerstag, 28. April 2011 22:32
An: [hidden email]
Betreff: Re: ExternalHeapHandle freeAddress:size: crash

On 04/28/2011 05:08 PM, Frank Lesser wrote:
> The problem is different. It is Smalltalk which produces the problem -
hint
> few emails back. It surprises me that it appears here now. It is referred
> here internally as the TR problem - the whole base system, GUI & OLE is
> affected.
What do you mean? could you share any extra info so we lean to spot and
solve the problem? thanks!

    gera

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: AW: ExternalHeapHandle freeAddress:size: crash

Gerardo Richarte
On 04/28/2011 06:06 PM, Frank Lesser wrote:
> I have given my loved old community the maximum
> hint I can do without making trouble to me. I am sure that you will discover the problem sooner or later yourself.
Loved meaning?

anyway, I missed the hints, I went back over your latest emails and
couldn't find anything else than "bugs related to finalization", which
is obviously what we are dealing with here, given that
#processRescuedEphemerons and #finalize are both in Carl's stack trace.

We may or may not find the bugs you are reffering to it in the future,
truth is that we'll never know, since we don't have much information to
contrast our findings.

Oh well, here we go again, each of us rediscovering the same things. I
understand you are now deep into Haskell, but if you remember what the
bugs are, and wish to help us, we'll be very gratefuller for your help.
In any case, thanks for all you gave the community already.

    gera

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: AW: ExternalHeapHandle freeAddress:size: crash

Todor Todorov
I will try again since the list server rejected the mail first time.

Could be that ppl. are hinting about the nature of finalization and that one cannot be sure about the order ephemerons "go out of scope" ... to use a C/C++ expression or how they are considered obsolete.

I have to check up on ephemerons again, but let's stick to some simple facts.
1. When a graph of objects is no longer reachable, it is considered garbage and can be garbage collected.
2. If an object wants to know about its own death, the GC mechanism sends #finalize just before the memory is released.
3. All objects in this graph are kept alive until all #finalize have completed.
4. There is no ROOT object (or more important object) in that graph!
5. The #finalize methods are called in RANDOM order.
6. Ephemerons is an implementation / a technology to allow weak object references.

See: http://swiki-lifia.info.unlp.edu.ar/ContextAware/uploads/29/Ephemerons%20-%20A%20New%20Finalization%20Mechanism.pdf. It's an article that discusses challenges with weak references (incl. finalization). In VSE it is implemented as a global dictionary if objects that need to be send #finalize. Sending #needsFinalization adds the object to this dictionary and #doesNotNeedFinalization removes it. This dictionary has weak references to the objects, so they will eventually become garbage. This explanation is slightly simplified, but that's the general idea. Otherwise look at SystemWeakRegistries (the root object to all this), Ephemeron, EphemeronDictionary, WeakRegistry and WeakKeyedRegistry.

The .Net documentation clearly says that the order of finalization is random. Same applies to VSE. For example, .Net states that if you have a file and a stream (remember, .Net uses decorator pattern), the file may be finalized and closed before the stream object. The stream will try to write its internal cache to a file that has already been closed. Bad! The .Net documentation clearly states; Leave with it! Make your stream tolerant that the underlying file object may have been closed and disposed by finalization! The data in the buffer is lost, and there is no way to write it to the file!

The .Net documentation states that finalization should not be used to save and close files and similar purposes. It's a last resort to avoid resource leaks. In other words, don't expect finalization to do your job! Use the #release methods; that's what it is for!


So, to go back to the discussion; I am not 100% sure on what Frank is hinting. But I am guessing that he's hinting on the random order of finalization. I am still not sure that's the case. Anyway, a simple solution to avoid that object is freed twice; simply blank the address contents with nil/zero immediately after a release. Windows is clever enough and it knows to ignore NULL pointers and handles. See the OLE framework; it does that.


I personally believe that the problem is simple heap corruption (buffer overwrite) and GC/finalization has little to do with the issue. But anyway, it can be:
1. Heap corruption (buffer overwrite) - use the "Global Flags" app.
2. Double release (as discussed above) - blank the contents of the address after the first release, so second one will fail.
3. Don't create aliases to the same heap address, i.e. don't copy the same heap address to two or more objects that all insist on releasing it.

Just few thoughts.
-- Todor

-----Original Message-----
From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Gerardo Richarte
Sent: 29. april 2011 04:22
To: [hidden email]
Subject: Re: AW: ExternalHeapHandle freeAddress:size: crash

On 04/28/2011 06:06 PM, Frank Lesser wrote:
> I have given my loved old community the maximum hint I can do without
> making trouble to me. I am sure that you will discover the problem sooner or later yourself.
Loved meaning?

anyway, I missed the hints, I went back over your latest emails and couldn't find anything else than "bugs related to finalization", which is obviously what we are dealing with here, given that #processRescuedEphemerons and #finalize are both in Carl's stack trace.

We may or may not find the bugs you are reffering to it in the future, truth is that we'll never know, since we don't have much information to contrast our findings.

Oh well, here we go again, each of us rediscovering the same things. I understand you are now deep into Haskell, but if you remember what the bugs are, and wish to help us, we'll be very gratefuller for your help.
In any case, thanks for all you gave the community already.

    gera

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Carl Gundel-3
Todor,

Thanks for that.  If I decide to go through the trouble of learning how to
use the "Global Flags" app, am I going to be able to do anything about the
problem if it does turn out to be heap corruption?

-Carl

-----Original Message-----

I personally believe that the problem is simple heap corruption (buffer
overwrite) and GC/finalization has little to do with the issue. But anyway,
it can be:
1. Heap corruption (buffer overwrite) - use the "Global Flags" app.

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Todor Todorov
If it is heap corruption, then it is because somebody is corrupting the heap. This happens if the user of the buffer expects it to be bigger than it actually is and overwrites the linked-list information that's at the ends of the buffer. Once the links are overwritten with random data, the heap is corrupt. Windows has no way of managing this heap and it crashes on purpose.

Heap addresses on 32-bit Windows are allocated on 8-byte  boundaries, so the chance of corruption is 1 out of 8. Most probably is copying of strings that do not have NULL terminator. VSE had a bug some time ago.

The Global Flags app controls functions in Windows on what logic to use for heap allocation. It can activate a debug-like version of the heap allocation functions that automatically add extra padding to the buffer with special data in it. The functions later check that this data is not overwritten (as it is not supposed to). There are several options (read the doc, because I can't remember all), but one is to have Windows allocate a virtual page for each buffer (huge waste of memory and performance) and have the CPU monitor read and write operation for overwrites. This way you can crash when the heap is being corrupted and not much later without knowing who did the damage.



-----Original Message-----
From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Carl Gundel
Sent: 29. april 2011 22:23
To: [hidden email]
Subject: Re: ExternalHeapHandle freeAddress:size: crash

Todor,

Thanks for that.  If I decide to go through the trouble of learning how to use the "Global Flags" app, am I going to be able to do anything about the problem if it does turn out to be heap corruption?

-Carl

-----Original Message-----

I personally believe that the problem is simple heap corruption (buffer
overwrite) and GC/finalization has little to do with the issue. But anyway, it can be:
1. Heap corruption (buffer overwrite) - use the "Global Flags" app.

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Andreas Rosenberg
Just a few additional hints.

The key to solve memory corruption problems is to track down the bad guy
that does write to addresses outside of the allocated blocks.

The first important step is to find ways to reproduce the error.

Then you should create log information for:
1.) alloc & free operations (start address, end address, parameters to api calls, caller)
2.) code that is using the memory, should log the memory ranges
    that are being modified
3.) if 2.) is not possible you can try to set breakpoints in a native debugger
    on write access for the addresses that are on the borders of the memory blocks.
   
Perhaps you can switch your implementation to use VirtualAlloc.
Access beyond committed pages will immediately result in access violoations
because no memory is mapped. Working with virtual memory also gives
you the possibility to protect pages and detect illegal accessors on
already committed pages.

Good luck.

Regards

   Andreas

Andreas Rosenberg | eMail: [hidden email]
APIS GmbH         | Phone: +49 9482 9415-0
Im Haslet 42      | Fax: +49 9482 9415-55
93086 Worth/D     | WWW: <http://www.apis.de/>
Germany           | <http://www.fmea.de/>

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

Re: ExternalHeapHandle freeAddress:size: crash

Todor Todorov
I agree with Andreas.

The "Global Flags" app does some of the tracing by identifying the "bad guy" when he strikes, not later.

Switching to virtual alloc is an option. For example, allocate 2 pages, one read-write, the second read-protected and write-protected. Offset the memory address inside the first page so the desired buffer is at the end of the page. This way, if the offender tries to overwrite beyond the buffer, it will GPF when accessing the protected page ... and can be traced and caught. Problem with this approach is 32-byte aligning, but it can be circumvented.

-----Original Message-----
From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Andreas Rosenberg
Sent: 2. maj 2011 10:53
To: [hidden email]
Subject: Re: ExternalHeapHandle freeAddress:size: crash

Just a few additional hints.

The key to solve memory corruption problems is to track down the bad guy that does write to addresses outside of the allocated blocks.

The first important step is to find ways to reproduce the error.

Then you should create log information for:
1.) alloc & free operations (start address, end address, parameters to api calls, caller)
2.) code that is using the memory, should log the memory ranges
    that are being modified
3.) if 2.) is not possible you can try to set breakpoints in a native debugger
    on write access for the addresses that are on the borders of the memory blocks.
   
Perhaps you can switch your implementation to use VirtualAlloc.
Access beyond committed pages will immediately result in access violoations because no memory is mapped. Working with virtual memory also gives you the possibility to protect pages and detect illegal accessors on already committed pages.

Good luck.

Regards

   Andreas

Andreas Rosenberg | eMail: [hidden email]
APIS GmbH         | Phone: +49 9482 9415-0
Im Haslet 42      | Fax: +49 9482 9415-55
93086 Worth/D     | WWW: <http://www.apis.de/>
Germany           | <http://www.fmea.de/>

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
Reply | Threaded
Open this post in threaded view
|

AW: Ephemeron bug

frank.lesser
In reply to this post by Javier Burroni
Javier,
Re-reading your posting I guess you get a crash not because the Ephemerons
are broken in the way you test - maybe because you set the
methodDictionaryArray of the EphemeronWRescueMemory key to nil to detect
whether it is rescued. Setting the mda of any Object to nil should crash the
VSE-VM:

| a | a := Object new. a methodDictionaryArray: nil. a class

Frank

-----Ursprüngliche Nachricht-----
Von: Using Visual Smalltalk for Windows/Enterprise
[mailto:[hidden email]] Im Auftrag von Javier Burroni
Gesendet: Dienstag, 26. April 2011 18:39
An: [hidden email]
Betreff: Ephemeron bug

Consider the following code:

testNestedEphemeronsWEphemeronKey
   | toBeRescued |
   toBeRescued := EphemeronWRescueMemory
       key: Object new
       value: (EphemeronWRescueMemory key: Object new).
   Smalltalk unusedMemory.
   Ephemeron processRescuedEphemerons.
   Smalltalk unusedMemory.
   self
       assert: toBeRescued hasBeenRescued;
       deny: toBeRescued value key methodDictionaryArray isNil;
       assert: toBeRescued value hasBeenRescued

Where EphemeronWRescueMemory is an Ephemeron's subclass which can
answer if it has been already rescued.
This code will break smalltalk.
The reason for this test to fail, is the existence of an ephemeron (B)
as other ephemeron's value (A).
Following Hayes, whenever ephemeron A rescued, it should be traced as
any other object -loosing the idea of ephemerons, neither in a direct
or indirect way-.
However, the apparent behavior when following object A and finding
ephemerons, is to mark and queue the just found ephemeron -B in this
case-; but never trace the ephemeron -B-. This makes B's reference to
be out of date, and (potentially) point to non valid objects.
In Hayes code:

Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointerQueueingEphemerons.
           ephemeron deref valueField tracePointerQueueingEphemerons].

When the correct code should be:
Heap::markPhase3
   EphemeronQueue enumerate:[:ephemeron|
           ephemeron deref signal: almostCollectable.
           ephemeron deref keyField tracePointer.
           ephemeron deref valueField tracePointer].

There are an interesting point regarding Hayes work: the first time a
GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark
all of the objects reachable from this pointer, paying no attention to
ephemerons" p. 183 [1]).
With a small modification, the algorithm may be able to handle object
B as an ephemeron in the first run. To do so, we have to focus on the
fundamental idea of ephemerons: the topological relation between
objects and the ephemeron's key/value. The relation between objects
and the ephemeron itself shall be of no importance.

So, suppose we have this:
VMGarbageCollector>>#rescueEphemerons
   | unknowns rescan rescued |
   rescued := rescan := false.
   unknowns := ephemeronsStack.
   [ephemerons isEmpty] whileFalse: [
       rescan := self followEphemeronsCollectingIn: unknowns.
       rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [
           unknowns do: [:ephemeron |
               self rescueEphemeron: ephemeron.
               rescued := true]].
       unknowns reset].
   rescued ifTrue: [self someEphemeronsRescued]


VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns
   | rescan |
   rescan := false.
   [ephemerons isEmpty] whileFalse: [| ephemeron |
       ephemeron := ephemerons pop.
       ephemeron == nil ifFalse: [
           (self checkReachablePropertyOf: ephemeron)
               ifTrue: [
                   self follow: ephemeron count: ephemeron
_extendedSize startingAt: 1.
                   rescan := true]
               ifFalse: [unknowns add: ephemeron]]].
   ^rescan

VMGarbageCollector>>#rescueEphemeron: ephemeron
   ephemeron _haveNoWeaks
   self follow: ephemeron count: ephemeron _extendedSize startingAt: 1.
   rescuedEphemerons add: ephemeron

This implementation may fail to detect object as 'almost collectable',
whenever a rescued ephemeron refers to an object, when this object is
the key of a non yet seen ephemeron. But the set of correctly rescued
ephemerons is big than the Hayes set (which is a subset)

/jb

[1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism.
Proceedings of the 12th ACM SIGPLAN conference on Object-oriented
programming, systems, languages, and applications.
http://portal.acm.org/citation.cfm?id=263733

--
" To be is to do " ( Socrates )
" To be or not to be " ( Shakespeare )
" To do is to be " ( Sartre )
" Do be do be do " ( Sinatra )

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***

***           this signature added by listserv             ***
*** Visit  http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management  ***
12