Ephemeron bug
Locked
12
12
 
|
Consider the following code:
testNestedEphemeronsWEphemeronKey | toBeRescued | toBeRescued := EphemeronWRescueMemory key: Object new value: (EphemeronWRescueMemory key: Object new). Smalltalk unusedMemory. Ephemeron processRescuedEphemerons. Smalltalk unusedMemory. self assert: toBeRescued hasBeenRescued; deny: toBeRescued value key methodDictionaryArray isNil; assert: toBeRescued value hasBeenRescued Where EphemeronWRescueMemory is an Ephemeron's subclass which can answer if it has been already rescued. This code will break smalltalk. The reason for this test to fail, is the existence of an ephemeron (B) as other ephemeron's value (A). Following Hayes, whenever ephemeron A rescued, it should be traced as any other object -loosing the idea of ephemerons, neither in a direct or indirect way-. However, the apparent behavior when following object A and finding ephemerons, is to mark and queue the just found ephemeron -B in this case-; but never trace the ephemeron -B-. This makes B's reference to be out of date, and (potentially) point to non valid objects. In Hayes code: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointerQueueingEphemerons. ephemeron deref valueField tracePointerQueueingEphemerons]. When the correct code should be: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointer. ephemeron deref valueField tracePointer]. There are an interesting point regarding Hayes work: the first time a GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark all of the objects reachable from this pointer, paying no attention to ephemerons" p. 183 [1]). With a small modification, the algorithm may be able to handle object B as an ephemeron in the first run. To do so, we have to focus on the fundamental idea of ephemerons: the topological relation between objects and the ephemeron's key/value. The relation between objects and the ephemeron itself shall be of no importance. So, suppose we have this: VMGarbageCollector>>#rescueEphemerons | unknowns rescan rescued | rescued := rescan := false. unknowns := ephemeronsStack. [ephemerons isEmpty] whileFalse: [ rescan := self followEphemeronsCollectingIn: unknowns. rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [ unknowns do: [:ephemeron | self rescueEphemeron: ephemeron. rescued := true]]. unknowns reset]. rescued ifTrue: [self someEphemeronsRescued] VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns | rescan | rescan := false. [ephemerons isEmpty] whileFalse: [| ephemeron | ephemeron := ephemerons pop. ephemeron == nil ifFalse: [ (self checkReachablePropertyOf: ephemeron) ifTrue: [ self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescan := true] ifFalse: [unknowns add: ephemeron]]]. ^rescan VMGarbageCollector>>#rescueEphemeron: ephemeron ephemeron _haveNoWeaks self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescuedEphemerons add: ephemeron This implementation may fail to detect object as 'almost collectable', whenever a rescued ephemeron refers to an object, when this object is the key of a non yet seen ephemeron. But the set of correctly rescued ephemerons is big than the Hayes set (which is a subset) /jb [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism. Proceedings of the 12th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications. http://portal.acm.org/citation.cfm?id=263733 -- " To be is to do " ( Socrates ) " To be or not to be " ( Shakespeare ) " To do is to be " ( Sartre ) " Do be do be do " ( Sinatra ) *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
Javier,
Have you checked if this issue also happens in other dialects? /Leandro
*** this signature added by listserv ***
*** Visit http://www.listserv.dfn.de/archives/vswe-l.html ***
*** for archive browsing and VSWE-L membership management ***
On Tue, Apr 26, 2011 at 1:38 PM, Javier Burroni <[hidden email]> wrote: Consider the following code: |
|
|
In reply to this post by Javier Burroni
Hi,
This is not a bug - it is a restriction how Ephemerons should not be used & are not used. However there are a few bugs in the current (? I refer to VSE pre 2K) implementation & plenty of bugs in the VSE framework regarding finalization. Frank -----Ursprüngliche Nachricht----- Von: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] Im Auftrag von Javier Burroni Gesendet: Dienstag, 26. April 2011 18:39 An: [hidden email] Betreff: Ephemeron bug Consider the following code: testNestedEphemeronsWEphemeronKey | toBeRescued | toBeRescued := EphemeronWRescueMemory key: Object new value: (EphemeronWRescueMemory key: Object new). Smalltalk unusedMemory. Ephemeron processRescuedEphemerons. Smalltalk unusedMemory. self assert: toBeRescued hasBeenRescued; deny: toBeRescued value key methodDictionaryArray isNil; assert: toBeRescued value hasBeenRescued Where EphemeronWRescueMemory is an Ephemeron's subclass which can answer if it has been already rescued. This code will break smalltalk. The reason for this test to fail, is the existence of an ephemeron (B) as other ephemeron's value (A). Following Hayes, whenever ephemeron A rescued, it should be traced as any other object -loosing the idea of ephemerons, neither in a direct or indirect way-. However, the apparent behavior when following object A and finding ephemerons, is to mark and queue the just found ephemeron -B in this case-; but never trace the ephemeron -B-. This makes B's reference to be out of date, and (potentially) point to non valid objects. In Hayes code: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointerQueueingEphemerons. ephemeron deref valueField tracePointerQueueingEphemerons]. When the correct code should be: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointer. ephemeron deref valueField tracePointer]. There are an interesting point regarding Hayes work: the first time a GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark all of the objects reachable from this pointer, paying no attention to ephemerons" p. 183 [1]). With a small modification, the algorithm may be able to handle object B as an ephemeron in the first run. To do so, we have to focus on the fundamental idea of ephemerons: the topological relation between objects and the ephemeron's key/value. The relation between objects and the ephemeron itself shall be of no importance. So, suppose we have this: VMGarbageCollector>>#rescueEphemerons | unknowns rescan rescued | rescued := rescan := false. unknowns := ephemeronsStack. [ephemerons isEmpty] whileFalse: [ rescan := self followEphemeronsCollectingIn: unknowns. rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [ unknowns do: [:ephemeron | self rescueEphemeron: ephemeron. rescued := true]]. unknowns reset]. rescued ifTrue: [self someEphemeronsRescued] VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns | rescan | rescan := false. [ephemerons isEmpty] whileFalse: [| ephemeron | ephemeron := ephemerons pop. ephemeron == nil ifFalse: [ (self checkReachablePropertyOf: ephemeron) ifTrue: [ self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescan := true] ifFalse: [unknowns add: ephemeron]]]. ^rescan VMGarbageCollector>>#rescueEphemeron: ephemeron ephemeron _haveNoWeaks self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescuedEphemerons add: ephemeron This implementation may fail to detect object as 'almost collectable', whenever a rescued ephemeron refers to an object, when this object is the key of a non yet seen ephemeron. But the set of correctly rescued ephemerons is big than the Hayes set (which is a subset) /jb [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism. Proceedings of the 12th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications. http://portal.acm.org/citation.cfm?id=263733 -- " To be is to do " ( Socrates ) " To be or not to be " ( Shakespeare ) " To do is to be " ( Sartre ) " Do be do be do " ( Sinatra ) *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
In reply to this post by Leandro Caniglia
Apparently, the issue is not present in VW
On Tue, Apr 26, 2011 at 2:22 PM, Leandro Caniglia <[hidden email]> wrote: > Javier, > Have you checked if this issue also happens in other dialects? > /Leandro > > On Tue, Apr 26, 2011 at 1:38 PM, Javier Burroni <[hidden email]> > wrote: >> >> Consider the following code: >> >> testNestedEphemeronsWEphemeronKey >> | toBeRescued | >> toBeRescued := EphemeronWRescueMemory >> key: Object new >> value: (EphemeronWRescueMemory key: Object new). >> Smalltalk unusedMemory. >> Ephemeron processRescuedEphemerons. >> Smalltalk unusedMemory. >> self >> assert: toBeRescued hasBeenRescued; >> deny: toBeRescued value key methodDictionaryArray isNil; >> assert: toBeRescued value hasBeenRescued >> >> Where EphemeronWRescueMemory is an Ephemeron's subclass which can >> answer if it has been already rescued. >> This code will break smalltalk. >> The reason for this test to fail, is the existence of an ephemeron (B) >> as other ephemeron's value (A). >> Following Hayes, whenever ephemeron A rescued, it should be traced as >> any other object -loosing the idea of ephemerons, neither in a direct >> or indirect way-. >> However, the apparent behavior when following object A and finding >> ephemerons, is to mark and queue the just found ephemeron -B in this >> case-; but never trace the ephemeron -B-. This makes B's reference to >> be out of date, and (potentially) point to non valid objects. >> In Hayes code: >> >> Heap::markPhase3 >> EphemeronQueue enumerate:[:ephemeron| >> ephemeron deref signal: almostCollectable. >> ephemeron deref keyField tracePointerQueueingEphemerons. >> ephemeron deref valueField tracePointerQueueingEphemerons]. >> >> When the correct code should be: >> Heap::markPhase3 >> EphemeronQueue enumerate:[:ephemeron| >> ephemeron deref signal: almostCollectable. >> ephemeron deref keyField tracePointer. >> ephemeron deref valueField tracePointer]. >> >> There are an interesting point regarding Hayes work: the first time a >> GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark >> all of the objects reachable from this pointer, paying no attention to >> ephemerons" p. 183 [1]). >> With a small modification, the algorithm may be able to handle object >> B as an ephemeron in the first run. To do so, we have to focus on the >> fundamental idea of ephemerons: the topological relation between >> objects and the ephemeron's key/value. The relation between objects >> and the ephemeron itself shall be of no importance. >> >> So, suppose we have this: >> VMGarbageCollector>>#rescueEphemerons >> | unknowns rescan rescued | >> rescued := rescan := false. >> unknowns := ephemeronsStack. >> [ephemerons isEmpty] whileFalse: [ >> rescan := self followEphemeronsCollectingIn: unknowns. >> rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [ >> unknowns do: [:ephemeron | >> self rescueEphemeron: ephemeron. >> rescued := true]]. >> unknowns reset]. >> rescued ifTrue: [self someEphemeronsRescued] >> >> >> VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns >> | rescan | >> rescan := false. >> [ephemerons isEmpty] whileFalse: [| ephemeron | >> ephemeron := ephemerons pop. >> ephemeron == nil ifFalse: [ >> (self checkReachablePropertyOf: ephemeron) >> ifTrue: [ >> self follow: ephemeron count: ephemeron >> _extendedSize startingAt: 1. >> rescan := true] >> ifFalse: [unknowns add: ephemeron]]]. >> ^rescan >> >> VMGarbageCollector>>#rescueEphemeron: ephemeron >> ephemeron _haveNoWeaks >> self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. >> rescuedEphemerons add: ephemeron >> >> This implementation may fail to detect object as 'almost collectable', >> whenever a rescued ephemeron refers to an object, when this object is >> the key of a non yet seen ephemeron. But the set of correctly rescued >> ephemerons is big than the Hayes set (which is a subset) >> >> /jb >> >> [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism. >> Proceedings of the 12th ACM SIGPLAN conference on Object-oriented >> programming, systems, languages, and applications. >> http://portal.acm.org/citation.cfm?id=263733 >> >> -- >> " To be is to do " ( Socrates ) >> " To be or not to be " ( Shakespeare ) >> " To do is to be " ( Sartre ) >> " Do be do be do " ( Sinatra ) >> >> *** this signature added by listserv *** >> *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** >> *** for archive browsing and VSWE-L membership management *** > > *** this signature added by listserv *** *** Visit > http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing > and VSWE-L membership management *** -- " To be is to do " ( Socrates ) " To be or not to be " ( Shakespeare ) " To do is to be " ( Sartre ) " Do be do be do " ( Sinatra ) *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
In reply to this post by frank.lesser
I've used the word 'bug' because the 'member of' relation produce a
crash, which should not be the case. (Besides the should and shouldn´t in the image space) On Tue, Apr 26, 2011 at 3:20 PM, Frank Lesser <[hidden email]> wrote: > Hi, > > This is not a bug - it is a restriction how Ephemerons should not be used & > are not used. > However there are a few bugs in the current (? I refer to VSE pre 2K) > implementation & plenty of bugs in the VSE framework regarding finalization. > > Frank > > -----Ursprüngliche Nachricht----- > Von: Using Visual Smalltalk for Windows/Enterprise > [mailto:[hidden email]] Im Auftrag von Javier Burroni > Gesendet: Dienstag, 26. April 2011 18:39 > An: [hidden email] > Betreff: Ephemeron bug > > Consider the following code: > > testNestedEphemeronsWEphemeronKey > | toBeRescued | > toBeRescued := EphemeronWRescueMemory > key: Object new > value: (EphemeronWRescueMemory key: Object new). > Smalltalk unusedMemory. > Ephemeron processRescuedEphemerons. > Smalltalk unusedMemory. > self > assert: toBeRescued hasBeenRescued; > deny: toBeRescued value key methodDictionaryArray isNil; > assert: toBeRescued value hasBeenRescued > > Where EphemeronWRescueMemory is an Ephemeron's subclass which can > answer if it has been already rescued. > This code will break smalltalk. > The reason for this test to fail, is the existence of an ephemeron (B) > as other ephemeron's value (A). > Following Hayes, whenever ephemeron A rescued, it should be traced as > any other object -loosing the idea of ephemerons, neither in a direct > or indirect way-. > However, the apparent behavior when following object A and finding > ephemerons, is to mark and queue the just found ephemeron -B in this > case-; but never trace the ephemeron -B-. This makes B's reference to > be out of date, and (potentially) point to non valid objects. > In Hayes code: > > Heap::markPhase3 > EphemeronQueue enumerate:[:ephemeron| > ephemeron deref signal: almostCollectable. > ephemeron deref keyField tracePointerQueueingEphemerons. > ephemeron deref valueField tracePointerQueueingEphemerons]. > > When the correct code should be: > Heap::markPhase3 > EphemeronQueue enumerate:[:ephemeron| > ephemeron deref signal: almostCollectable. > ephemeron deref keyField tracePointer. > ephemeron deref valueField tracePointer]. > > There are an interesting point regarding Hayes work: the first time a > GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark > all of the objects reachable from this pointer, paying no attention to > ephemerons" p. 183 [1]). > With a small modification, the algorithm may be able to handle object > B as an ephemeron in the first run. To do so, we have to focus on the > fundamental idea of ephemerons: the topological relation between > objects and the ephemeron's key/value. The relation between objects > and the ephemeron itself shall be of no importance. > > So, suppose we have this: > VMGarbageCollector>>#rescueEphemerons > | unknowns rescan rescued | > rescued := rescan := false. > unknowns := ephemeronsStack. > [ephemerons isEmpty] whileFalse: [ > rescan := self followEphemeronsCollectingIn: unknowns. > rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [ > unknowns do: [:ephemeron | > self rescueEphemeron: ephemeron. > rescued := true]]. > unknowns reset]. > rescued ifTrue: [self someEphemeronsRescued] > > > VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns > | rescan | > rescan := false. > [ephemerons isEmpty] whileFalse: [| ephemeron | > ephemeron := ephemerons pop. > ephemeron == nil ifFalse: [ > (self checkReachablePropertyOf: ephemeron) > ifTrue: [ > self follow: ephemeron count: ephemeron > _extendedSize startingAt: 1. > rescan := true] > ifFalse: [unknowns add: ephemeron]]]. > ^rescan > > VMGarbageCollector>>#rescueEphemeron: ephemeron > ephemeron _haveNoWeaks > self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. > rescuedEphemerons add: ephemeron > > This implementation may fail to detect object as 'almost collectable', > whenever a rescued ephemeron refers to an object, when this object is > the key of a non yet seen ephemeron. But the set of correctly rescued > ephemerons is big than the Hayes set (which is a subset) > > /jb > > [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism. > Proceedings of the 12th ACM SIGPLAN conference on Object-oriented > programming, systems, languages, and applications. > http://portal.acm.org/citation.cfm?id=263733 > > -- > " To be is to do " ( Socrates ) > " To be or not to be " ( Shakespeare ) > " To do is to be " ( Sartre ) > " Do be do be do " ( Sinatra ) > > *** this signature added by listserv *** > *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** > *** for archive browsing and VSWE-L membership management *** > > *** this signature added by listserv *** > *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** > *** for archive browsing and VSWE-L membership management *** > -- " To be is to do " ( Socrates ) " To be or not to be " ( Shakespeare ) " To do is to be " ( Sartre ) " Do be do be do " ( Sinatra ) *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
In reply to this post by Javier Burroni
This is interesting, but how would the bug exhibit itself in an application
day to day? -Carl -----Original Message----- From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Javier Burroni Sent: Tuesday, April 26, 2011 12:39 PM To: [hidden email] Subject: Ephemeron bug Consider the following code: testNestedEphemeronsWEphemeronKey | toBeRescued | toBeRescued := EphemeronWRescueMemory key: Object new value: (EphemeronWRescueMemory key: Object new). Smalltalk unusedMemory. Ephemeron processRescuedEphemerons. Smalltalk unusedMemory. self assert: toBeRescued hasBeenRescued; deny: toBeRescued value key methodDictionaryArray isNil; assert: toBeRescued value hasBeenRescued Where EphemeronWRescueMemory is an Ephemeron's subclass which can answer if it has been already rescued. This code will break smalltalk. The reason for this test to fail, is the existence of an ephemeron (B) as other ephemeron's value (A). Following Hayes, whenever ephemeron A rescued, it should be traced as any other object -loosing the idea of ephemerons, neither in a direct or indirect way-. However, the apparent behavior when following object A and finding ephemerons, is to mark and queue the just found ephemeron -B in this case-; but never trace the ephemeron -B-. This makes B's reference to be out of date, and (potentially) point to non valid objects. In Hayes code: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointerQueueingEphemerons. ephemeron deref valueField tracePointerQueueingEphemerons]. When the correct code should be: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointer. ephemeron deref valueField tracePointer]. There are an interesting point regarding Hayes work: the first time a GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark all of the objects reachable from this pointer, paying no attention to ephemerons" p. 183 [1]). With a small modification, the algorithm may be able to handle object B as an ephemeron in the first run. To do so, we have to focus on the fundamental idea of ephemerons: the topological relation between objects and the ephemeron's key/value. The relation between objects and the ephemeron itself shall be of no importance. So, suppose we have this: VMGarbageCollector>>#rescueEphemerons | unknowns rescan rescued | rescued := rescan := false. unknowns := ephemeronsStack. [ephemerons isEmpty] whileFalse: [ rescan := self followEphemeronsCollectingIn: unknowns. rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [ unknowns do: [:ephemeron | self rescueEphemeron: ephemeron. rescued := true]]. unknowns reset]. rescued ifTrue: [self someEphemeronsRescued] VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns | rescan | rescan := false. [ephemerons isEmpty] whileFalse: [| ephemeron | ephemeron := ephemerons pop. ephemeron == nil ifFalse: [ (self checkReachablePropertyOf: ephemeron) ifTrue: [ self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescan := true] ifFalse: [unknowns add: ephemeron]]]. ^rescan VMGarbageCollector>>#rescueEphemeron: ephemeron ephemeron _haveNoWeaks self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescuedEphemerons add: ephemeron This implementation may fail to detect object as 'almost collectable', whenever a rescued ephemeron refers to an object, when this object is the key of a non yet seen ephemeron. But the set of correctly rescued ephemerons is big than the Hayes set (which is a subset) /jb [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism. Proceedings of the 12th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications. http://portal.acm.org/citation.cfm?id=263733 -- " To be is to do " ( Socrates ) " To be or not to be " ( Shakespeare ) " To do is to be " ( Sartre ) " Do be do be do " ( Sinatra ) *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
ExternalHeapHandle freeAddress:size: crash
|
|
In reply to this post by Javier Burroni
Anyone know what to make of this?
KernelDLL(Object)>>vmInterrupt: <#protectionViolationInterrupt> KernelDLL(DynamicLinkLibrary)>>invalidArgument KernelDLL>>heapFree: <anExternalHeapHandle> flags: <0> address: KernelDLL>><aByteArray> ExternalHeapHandle>>freeAddress: <anExternalHeapAddress> size: <4> ExternalHeapAddress>>releaseMemory ExternalHeapAddress(ExternalAddress)>>release ExternalHeapAddress(ExternalAddress)>>finalize [] in SystemWeakRegistries>>initialize BlockClosure>>evaluateWithArguments: <anArray> WeakRegistry>>rescue: <anEphemeron> Ephemeron>>rescue [] in Ephemeron class>>processRescuedEphemerons Thanks, -Carl *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
Heap Corruption!
Use the "Global Flags" app in the Debug Tools for Windows to diagnose this. -----Original Message----- From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Carl Gundel Sent: 28. april 2011 19:09 To: [hidden email] Subject: ExternalHeapHandle freeAddress:size: crash Anyone know what to make of this? KernelDLL(Object)>>vmInterrupt: <#protectionViolationInterrupt> KernelDLL(DynamicLinkLibrary)>>invalidArgument KernelDLL>>heapFree: <anExternalHeapHandle> flags: <0> address: KernelDLL>><aByteArray> ExternalHeapHandle>>freeAddress: <anExternalHeapAddress> size: <4> ExternalHeapAddress>>releaseMemory ExternalHeapAddress(ExternalAddress)>>release ExternalHeapAddress(ExternalAddress)>>finalize [] in SystemWeakRegistries>>initialize BlockClosure>>evaluateWithArguments: <anArray> WeakRegistry>>rescue: <anEphemeron> Ephemeron>>rescue [] in Ephemeron class>>processRescuedEphemerons Thanks, -Carl *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
The problem is different. It is Smalltalk which produces the problem - hint
few emails back. It surprises me that it appears here now. It is referred here internally as the TR problem - the whole base system, GUI & OLE is affected. *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
On 04/28/2011 05:08 PM, Frank Lesser wrote:
> The problem is different. It is Smalltalk which produces the problem - hint > few emails back. It surprises me that it appears here now. It is referred > here internally as the TR problem - the whole base system, GUI & OLE is > affected. What do you mean? could you share any extra info so we lean to spot and solve the problem? thanks! gera *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
In reply to this post by Carl Gundel-3
As Todor said, this looks a lot like a Heap Corruption bug (which could
turn into a security bug, but we are not really worried about that). The situation here is that an ExternalHeapAddress is finalized, and that should be correct. When it's finalized the memory is returned to the os (HeapFree()), and here's where we have the crash. There are several situations that might lead to this type of crashes: 1. The address argument to HeapFree() is incorrect. You should be able to see that in Smalltalk, place a breakpoint in #freeAddress:size: and see if the contents of the ByteArray look like a valid heap address (it could be tricky to break on the offending #freeAddress:size, and tricky also to know if the address is valid. At any cost, the address inside the ByteArray must be the result of calling HeapAllocate()) 2. The memory in the heap might have been overwritten, and hence the heap structure corrupted, and hence HeapFree() doesn't know what to do about it, and panics, and crashes. It's not easy to spot this, but if you had a native debugger attached to vdevw.exe you could place a hardware breakpoint on write on the bytes around the offending heap buffer. This breakpoint should only halt when calling HeapFree() or HeapAlloc(), otherwise it's a bug. 3. Double free. Calling HeapFree() twice on the same address, even if the address is correct the first time, could produce a crash. This is kind of easy to trigger from within Smtalltalk, for example, if you use #needsFinalization more than once on the same Object(ExternalHeapAddress). Or if you copy the ExternalHeapAddress and then do #needsFinalization on the copy, or something like that. This should be easy to check and stop, if only adding code to #needsFinalization to check if the object is already in the registry. (or actually making the registry a Set?) oh well... I hope this helps a bit at least, let us know what happened! gera On 04/28/2011 02:09 PM, Carl Gundel wrote: > Anyone know what to make of this? > > KernelDLL(Object)>>vmInterrupt: <#protectionViolationInterrupt> > KernelDLL(DynamicLinkLibrary)>>invalidArgument > KernelDLL>>heapFree: <anExternalHeapHandle> flags: <0> address: > KernelDLL>><aByteArray> > ExternalHeapHandle>>freeAddress: <anExternalHeapAddress> size: <4> > ExternalHeapAddress>>releaseMemory > ExternalHeapAddress(ExternalAddress)>>release > ExternalHeapAddress(ExternalAddress)>>finalize > [] in SystemWeakRegistries>>initialize > BlockClosure>>evaluateWithArguments: <anArray> > WeakRegistry>>rescue: <anEphemeron> > Ephemeron>>rescue > [] in Ephemeron class>>processRescuedEphemerons *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
On 04/28/2011 05:32 PM, Gerardo Richarte wrote:
> 3. Double free. Calling HeapFree() twice on the same address, even if > the address is correct the first time, could produce a crash. This is > kind of easy to trigger from within Smtalltalk, for example, if you use > #needsFinalization more than once on the same > Object(ExternalHeapAddress). not correct, sorry... the registry uses am EphemeronDictionary, so it won't be finalized twice unless you play some tricks (like making a copy of the ExternalHeapAddress or something similar) gera *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
AW: ExternalHeapHandle freeAddress:size: crash
|
|
In reply to this post by Gerardo Richarte
Hi Gerardo,
No I am afraid no I can't - I have given my loved old community the maximum hint I can do without making trouble to me. I am sure that you will discover the problem sooner or later yourself. I see that you have gained a lot of knowledge about Digitalk St - I didn't check other St's (lack of time) but I am sure the issue is present there also - maybe because there are common roots. Frank Nachricht----- Von: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] Im Auftrag von Gerardo Richarte Gesendet: Donnerstag, 28. April 2011 22:32 An: [hidden email] Betreff: Re: ExternalHeapHandle freeAddress:size: crash On 04/28/2011 05:08 PM, Frank Lesser wrote: > The problem is different. It is Smalltalk which produces the problem - hint > few emails back. It surprises me that it appears here now. It is referred > here internally as the TR problem - the whole base system, GUI & OLE is > affected. What do you mean? could you share any extra info so we lean to spot and solve the problem? thanks! gera *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: AW: ExternalHeapHandle freeAddress:size: crash
|
|
On 04/28/2011 06:06 PM, Frank Lesser wrote:
> I have given my loved old community the maximum > hint I can do without making trouble to me. I am sure that you will discover the problem sooner or later yourself. Loved meaning? anyway, I missed the hints, I went back over your latest emails and couldn't find anything else than "bugs related to finalization", which is obviously what we are dealing with here, given that #processRescuedEphemerons and #finalize are both in Carl's stack trace. We may or may not find the bugs you are reffering to it in the future, truth is that we'll never know, since we don't have much information to contrast our findings. Oh well, here we go again, each of us rediscovering the same things. I understand you are now deep into Haskell, but if you remember what the bugs are, and wish to help us, we'll be very gratefuller for your help. In any case, thanks for all you gave the community already. gera *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: AW: ExternalHeapHandle freeAddress:size: crash
|
|
I will try again since the list server rejected the mail first time.
Could be that ppl. are hinting about the nature of finalization and that one cannot be sure about the order ephemerons "go out of scope" ... to use a C/C++ expression or how they are considered obsolete. I have to check up on ephemerons again, but let's stick to some simple facts. 1. When a graph of objects is no longer reachable, it is considered garbage and can be garbage collected. 2. If an object wants to know about its own death, the GC mechanism sends #finalize just before the memory is released. 3. All objects in this graph are kept alive until all #finalize have completed. 4. There is no ROOT object (or more important object) in that graph! 5. The #finalize methods are called in RANDOM order. 6. Ephemerons is an implementation / a technology to allow weak object references. See: http://swiki-lifia.info.unlp.edu.ar/ContextAware/uploads/29/Ephemerons%20-%20A%20New%20Finalization%20Mechanism.pdf. It's an article that discusses challenges with weak references (incl. finalization). In VSE it is implemented as a global dictionary if objects that need to be send #finalize. Sending #needsFinalization adds the object to this dictionary and #doesNotNeedFinalization removes it. This dictionary has weak references to the objects, so they will eventually become garbage. This explanation is slightly simplified, but that's the general idea. Otherwise look at SystemWeakRegistries (the root object to all this), Ephemeron, EphemeronDictionary, WeakRegistry and WeakKeyedRegistry. The .Net documentation clearly says that the order of finalization is random. Same applies to VSE. For example, .Net states that if you have a file and a stream (remember, .Net uses decorator pattern), the file may be finalized and closed before the stream object. The stream will try to write its internal cache to a file that has already been closed. Bad! The .Net documentation clearly states; Leave with it! Make your stream tolerant that the underlying file object may have been closed and disposed by finalization! The data in the buffer is lost, and there is no way to write it to the file! The .Net documentation states that finalization should not be used to save and close files and similar purposes. It's a last resort to avoid resource leaks. In other words, don't expect finalization to do your job! Use the #release methods; that's what it is for! So, to go back to the discussion; I am not 100% sure on what Frank is hinting. But I am guessing that he's hinting on the random order of finalization. I am still not sure that's the case. Anyway, a simple solution to avoid that object is freed twice; simply blank the address contents with nil/zero immediately after a release. Windows is clever enough and it knows to ignore NULL pointers and handles. See the OLE framework; it does that. I personally believe that the problem is simple heap corruption (buffer overwrite) and GC/finalization has little to do with the issue. But anyway, it can be: 1. Heap corruption (buffer overwrite) - use the "Global Flags" app. 2. Double release (as discussed above) - blank the contents of the address after the first release, so second one will fail. 3. Don't create aliases to the same heap address, i.e. don't copy the same heap address to two or more objects that all insist on releasing it. Just few thoughts. -- Todor -----Original Message----- From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Gerardo Richarte Sent: 29. april 2011 04:22 To: [hidden email] Subject: Re: AW: ExternalHeapHandle freeAddress:size: crash On 04/28/2011 06:06 PM, Frank Lesser wrote: > I have given my loved old community the maximum hint I can do without > making trouble to me. I am sure that you will discover the problem sooner or later yourself. Loved meaning? anyway, I missed the hints, I went back over your latest emails and couldn't find anything else than "bugs related to finalization", which is obviously what we are dealing with here, given that #processRescuedEphemerons and #finalize are both in Carl's stack trace. We may or may not find the bugs you are reffering to it in the future, truth is that we'll never know, since we don't have much information to contrast our findings. Oh well, here we go again, each of us rediscovering the same things. I understand you are now deep into Haskell, but if you remember what the bugs are, and wish to help us, we'll be very gratefuller for your help. In any case, thanks for all you gave the community already. gera *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
Todor,
Thanks for that. If I decide to go through the trouble of learning how to use the "Global Flags" app, am I going to be able to do anything about the problem if it does turn out to be heap corruption? -Carl -----Original Message----- I personally believe that the problem is simple heap corruption (buffer overwrite) and GC/finalization has little to do with the issue. But anyway, it can be: 1. Heap corruption (buffer overwrite) - use the "Global Flags" app. *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
If it is heap corruption, then it is because somebody is corrupting the heap. This happens if the user of the buffer expects it to be bigger than it actually is and overwrites the linked-list information that's at the ends of the buffer. Once the links are overwritten with random data, the heap is corrupt. Windows has no way of managing this heap and it crashes on purpose.
Heap addresses on 32-bit Windows are allocated on 8-byte boundaries, so the chance of corruption is 1 out of 8. Most probably is copying of strings that do not have NULL terminator. VSE had a bug some time ago. The Global Flags app controls functions in Windows on what logic to use for heap allocation. It can activate a debug-like version of the heap allocation functions that automatically add extra padding to the buffer with special data in it. The functions later check that this data is not overwritten (as it is not supposed to). There are several options (read the doc, because I can't remember all), but one is to have Windows allocate a virtual page for each buffer (huge waste of memory and performance) and have the CPU monitor read and write operation for overwrites. This way you can crash when the heap is being corrupted and not much later without knowing who did the damage. -----Original Message----- From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Carl Gundel Sent: 29. april 2011 22:23 To: [hidden email] Subject: Re: ExternalHeapHandle freeAddress:size: crash Todor, Thanks for that. If I decide to go through the trouble of learning how to use the "Global Flags" app, am I going to be able to do anything about the problem if it does turn out to be heap corruption? -Carl -----Original Message----- I personally believe that the problem is simple heap corruption (buffer overwrite) and GC/finalization has little to do with the issue. But anyway, it can be: 1. Heap corruption (buffer overwrite) - use the "Global Flags" app. *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
Just a few additional hints.
The key to solve memory corruption problems is to track down the bad guy that does write to addresses outside of the allocated blocks. The first important step is to find ways to reproduce the error. Then you should create log information for: 1.) alloc & free operations (start address, end address, parameters to api calls, caller) 2.) code that is using the memory, should log the memory ranges that are being modified 3.) if 2.) is not possible you can try to set breakpoints in a native debugger on write access for the addresses that are on the borders of the memory blocks. Perhaps you can switch your implementation to use VirtualAlloc. Access beyond committed pages will immediately result in access violoations because no memory is mapped. Working with virtual memory also gives you the possibility to protect pages and detect illegal accessors on already committed pages. Good luck. Regards Andreas Andreas Rosenberg | eMail: [hidden email] APIS GmbH | Phone: +49 9482 9415-0 Im Haslet 42 | Fax: +49 9482 9415-55 93086 Worth/D | WWW: <http://www.apis.de/> Germany | <http://www.fmea.de/> *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
Re: ExternalHeapHandle freeAddress:size: crash
|
|
I agree with Andreas.
The "Global Flags" app does some of the tracing by identifying the "bad guy" when he strikes, not later. Switching to virtual alloc is an option. For example, allocate 2 pages, one read-write, the second read-protected and write-protected. Offset the memory address inside the first page so the desired buffer is at the end of the page. This way, if the offender tries to overwrite beyond the buffer, it will GPF when accessing the protected page ... and can be traced and caught. Problem with this approach is 32-byte aligning, but it can be circumvented. -----Original Message----- From: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] On Behalf Of Andreas Rosenberg Sent: 2. maj 2011 10:53 To: [hidden email] Subject: Re: ExternalHeapHandle freeAddress:size: crash Just a few additional hints. The key to solve memory corruption problems is to track down the bad guy that does write to addresses outside of the allocated blocks. The first important step is to find ways to reproduce the error. Then you should create log information for: 1.) alloc & free operations (start address, end address, parameters to api calls, caller) 2.) code that is using the memory, should log the memory ranges that are being modified 3.) if 2.) is not possible you can try to set breakpoints in a native debugger on write access for the addresses that are on the borders of the memory blocks. Perhaps you can switch your implementation to use VirtualAlloc. Access beyond committed pages will immediately result in access violoations because no memory is mapped. Working with virtual memory also gives you the possibility to protect pages and detect illegal accessors on already committed pages. Good luck. Regards Andreas Andreas Rosenberg | eMail: [hidden email] APIS GmbH | Phone: +49 9482 9415-0 Im Haslet 42 | Fax: +49 9482 9415-55 93086 Worth/D | WWW: <http://www.apis.de/> Germany | <http://www.fmea.de/> *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
|
|
In reply to this post by Javier Burroni
Javier,
Re-reading your posting I guess you get a crash not because the Ephemerons are broken in the way you test - maybe because you set the methodDictionaryArray of the EphemeronWRescueMemory key to nil to detect whether it is rescued. Setting the mda of any Object to nil should crash the VSE-VM: | a | a := Object new. a methodDictionaryArray: nil. a class Frank -----Ursprüngliche Nachricht----- Von: Using Visual Smalltalk for Windows/Enterprise [mailto:[hidden email]] Im Auftrag von Javier Burroni Gesendet: Dienstag, 26. April 2011 18:39 An: [hidden email] Betreff: Ephemeron bug Consider the following code: testNestedEphemeronsWEphemeronKey | toBeRescued | toBeRescued := EphemeronWRescueMemory key: Object new value: (EphemeronWRescueMemory key: Object new). Smalltalk unusedMemory. Ephemeron processRescuedEphemerons. Smalltalk unusedMemory. self assert: toBeRescued hasBeenRescued; deny: toBeRescued value key methodDictionaryArray isNil; assert: toBeRescued value hasBeenRescued Where EphemeronWRescueMemory is an Ephemeron's subclass which can answer if it has been already rescued. This code will break smalltalk. The reason for this test to fail, is the existence of an ephemeron (B) as other ephemeron's value (A). Following Hayes, whenever ephemeron A rescued, it should be traced as any other object -loosing the idea of ephemerons, neither in a direct or indirect way-. However, the apparent behavior when following object A and finding ephemerons, is to mark and queue the just found ephemeron -B in this case-; but never trace the ephemeron -B-. This makes B's reference to be out of date, and (potentially) point to non valid objects. In Hayes code: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointerQueueingEphemerons. ephemeron deref valueField tracePointerQueueingEphemerons]. When the correct code should be: Heap::markPhase3 EphemeronQueue enumerate:[:ephemeron| ephemeron deref signal: almostCollectable. ephemeron deref keyField tracePointer. ephemeron deref valueField tracePointer]. There are an interesting point regarding Hayes work: the first time a GC is run, the ephemeron B will be handle as a non-ephemeron ("Mark all of the objects reachable from this pointer, paying no attention to ephemerons" p. 183 [1]). With a small modification, the algorithm may be able to handle object B as an ephemeron in the first run. To do so, we have to focus on the fundamental idea of ephemerons: the topological relation between objects and the ephemeron's key/value. The relation between objects and the ephemeron itself shall be of no importance. So, suppose we have this: VMGarbageCollector>>#rescueEphemerons | unknowns rescan rescued | rescued := rescan := false. unknowns := ephemeronsStack. [ephemerons isEmpty] whileFalse: [ rescan := self followEphemeronsCollectingIn: unknowns. rescan ifTrue: [ephemerons addAll: unknowns] ifFalse: [ unknowns do: [:ephemeron | self rescueEphemeron: ephemeron. rescued := true]]. unknowns reset]. rescued ifTrue: [self someEphemeronsRescued] VMGarbageCollector>>#followEphemeronsCollectingIn: unknowns | rescan | rescan := false. [ephemerons isEmpty] whileFalse: [| ephemeron | ephemeron := ephemerons pop. ephemeron == nil ifFalse: [ (self checkReachablePropertyOf: ephemeron) ifTrue: [ self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescan := true] ifFalse: [unknowns add: ephemeron]]]. ^rescan VMGarbageCollector>>#rescueEphemeron: ephemeron ephemeron _haveNoWeaks self follow: ephemeron count: ephemeron _extendedSize startingAt: 1. rescuedEphemerons add: ephemeron This implementation may fail to detect object as 'almost collectable', whenever a rescued ephemeron refers to an object, when this object is the key of a non yet seen ephemeron. But the set of correctly rescued ephemerons is big than the Hayes set (which is a subset) /jb [1]: Hayes, B. 1997. Ephemerons: a new finalization mechanism. Proceedings of the 12th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications. http://portal.acm.org/citation.cfm?id=263733 -- " To be is to do " ( Socrates ) " To be or not to be " ( Shakespeare ) " To do is to be " ( Sartre ) " Do be do be do " ( Sinatra ) *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** *** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management *** |
«
Return to Visual Smalltalk Enterprise
|
1 view|%1 views
| Free forum by Nabble | Edit this page |