Something I don't see in ImageSegment
Locked
Something I don't see in ImageSegment
 
|
Hi. ImageSegment uses GC to compute the out pointers. Basically, to a given array of roots (a graph) it does:
1) It marks the rootArray and all root objects by turning on the Garbage Collector bit in the object header of those particular objects. 2) Then does a mark pass over all objects. This will stop at our marked roots leaving our segment unmarked in their shadow. 3) Finally, it unmarks the rootArray and all root objects. 4) Only external objects are now marked. The unmarked objects in the graph of objects of the arrays are those who will be put in the WordArray (the segment) and those who are marked to the outPointers. However, I don't see where step 2) is done in Interpreter >> primitiveStoreImageSegment In a piece of such method it does: "Then do a mark pass over all objects. This will stop at our marked roots, thus leaving our segment unmarked in their shadow." savedYoungStart := youngStart. youngStart := self startOfMemory. "process all of memory" "clear the recycled context lists" freeContexts := NilContext. freeLargeContexts := NilContext. self markAndTraceInterpreterOops. "and special objects array" youngStart := savedYoungStart. "Finally unmark the rootArray and all root objects." self longAt: arrayOfRoots put: ((self longAt: arrayOfRoots) bitAnd: AllButMarkBit). ......continue....... But here I see it is only marking Interpreter oops, not all objects. And I don't see where all objects are marked even in the code that follows this little piece. You can see the comment: "Then do a mark pass over all objects" and even without the comment it makes sense to mark all objects in order to detect outPointers. Does anyone has a hint? Thanks Mariano _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: Something I don't see in ImageSegment
|
|
Hi Mariano, Hi All,
response below. But does anyone have any tests for the image segment primitives? I'm refactoring them and would like some tests.
2010/6/3 Mariano Martinez Peck <[hidden email]> Hi. ImageSegment uses GC to compute the out pointers. Basically, to a given array of roots (a graph) it does: markInterpreterOops marks all objects accessible from the interpreter oops, including the specialObjectsArray, recursively marking any unmarked objects and their referents. i.e. it marks all objects reachable from the roots of the system. This would mark all objects in the system were it not for the fact that the primitive marks the rootArray and so markInterpreterOops does not marl objects /only/ reachable from the rootArray. So once the rootArray objects have been unmarked, only those objects that should go in the image segment are unmarked.
So the primitive works by - marking the objects in rootArray, preventing marking of objects reachable from them in the next step - recursively marking all objects reachable from the roots (except those already marked)
- unmarking root objects, leaving the transitive closure of objects accessible from the roots and no where else unmarked - writing the unmarked objects to an image segment HTH Eliot
_______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: Something I don't see in ImageSegment
|
|
On 3 juin 2010, at 18:55, Eliot Miranda wrote: > Hi Mariano, Hi All, > > response below. But does anyone have any tests for the image segment primitives? I'm refactoring them and would like some tests. > I don't have tests, but it's cool that you take care of this. The code I saw was really ugly with huge methods. Noury > 2010/6/3 Mariano Martinez Peck <[hidden email]> > Hi. ImageSegment uses GC to compute the out pointers. Basically, to a given array of roots (a graph) it does: > > 1) It marks the rootArray and all root objects by turning on the Garbage Collector bit in the object header of those particular objects. > 2) Then does a mark pass over all objects. This will stop at our marked roots leaving our segment unmarked in their shadow. > 3) Finally, it unmarks the rootArray and all root objects. > 4) Only external objects are now marked. > > The unmarked objects in the graph of objects of the arrays are those who will be put in the WordArray (the segment) and those who are marked to the outPointers. > > However, I don't see where step 2) is done in Interpreter >> primitiveStoreImageSegment > > In a piece of such method it does: > > > "Then do a mark pass over all objects. This will stop at our marked roots, > thus leaving our segment unmarked in their shadow." > savedYoungStart := youngStart. > youngStart := self startOfMemory. "process all of memory" > "clear the recycled context lists" > freeContexts := NilContext. > freeLargeContexts := NilContext. > self markAndTraceInterpreterOops. "and special objects array" > youngStart := savedYoungStart. > > "Finally unmark the rootArray and all root objects." > self longAt: arrayOfRoots put: ((self longAt: arrayOfRoots) bitAnd: AllButMarkBit). > ......continue....... > > But here I see it is only marking Interpreter oops, not all objects. And I don't see where all objects are marked even in the code that follows this little piece. > > markInterpreterOops marks all objects accessible from the interpreter oops, including the specialObjectsArray, recursively marking any unmarked objects and their referents. i.e. it marks all objects reachable from the roots of the system. This would mark all objects in the system were it not for the fact that the primitive marks the rootArray and so markInterpreterOops does not marl objects /only/ reachable from the rootArray. So once the rootArray objects have been unmarked, only those objects that should go in the image segment are unmarked. > > So the primitive works by > - marking the objects in rootArray, preventing marking of objects reachable from them in the next step > - recursively marking all objects reachable from the roots (except those already marked) > - unmarking root objects, leaving the transitive closure of objects accessible from the roots and no where else unmarked > - writing the unmarked objects to an image segment > > > HTH > Eliot > > > You can see the comment: "Then do a mark pass over all objects" and even without the comment it makes sense to mark all objects in order to detect outPointers. > > Does anyone has a hint? > > Thanks > > Mariano > > _______________________________________________ > Pharo-project mailing list > [hidden email] > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project > > _______________________________________________ > Pharo-project mailing list > [hidden email] > http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project _______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: Something I don't see in ImageSegment
|
|
In reply to this post by Eliot Miranda-2
2010/6/3 Eliot Miranda <[hidden email]> Hi Mariano, Hi All, Hi Eliot. First, congratulations!! I though several times to refactor the primitives what I was not brave enough. I was not confidence of myself hahaha Originally, ImageSegment didn't have a single test. Only some class side examples that in Pharo at least were not even working. Adrian Lienhard wrote the first 2 or 3 tests for them. Then I needed to understand ImageSegment for my PhD, and the best way to understand them were writing unit tests ;) So, now if you take a Pharo 1.1 image you will see there are 26 tests. The classes are ImageSegmentTest, ImageSegmentTestExport and ImageSegmentTestSwap. The tests are quite simple and cover not too much. They can be improved and there are more to write. They may be even not completly correct as I wrote them to understand ImageSegment, not because I already understood them ;) In addition, there is one failing that I yet don't understand why: #testOutPointers3 I asked in mailing list but not really an answer in my opinion: http://forum.world.st/Something-I-don-t-understand-with-ImageSegments-tp1296982p1296982.html If you know why it is failing, tell me :) continue answering above...
Ok...but where is the roots of the system? it is the instVar rootTable of ObjectMemory? so all the root table is also marked? At the beginning I though it was this that you are saying. But then I check in the senders of markAndTraceInterpreterOops and you can see in ObjectMemory >> markPhase that it does: "trace the interpreter's objects, including the active stack and special objects array" self markAndTraceInterpreterOops. statSpecialMarkCount := statMarkCount. "trace the roots" 1 to: rootTableCount do: [:i | oop := rootTable at: i. self markAndTrace: oop]. 1 to: extraRootCount do:[:i| oop := (extraRoots at: i) at: 0. (self isIntegerObject: oop) ifFalse:[self markAndTrace: oop]]. So...it is manually iterating the table. Why? shouldn't be done as you told me? In summary, my question is: is these code doing this (iterating the table) even if it is not necessary or ImageSegment should also do it?
yes, this is clear :) Thanks for the help. Mariano
_______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: Something I don't see in ImageSegment
|
|
2010/6/4 Mariano Martinez Peck <[hidden email]>
I wrote a test yesterday anyway. Perhaps we can combine at some stage. I can certainly use the tests you've written though. Thanks!
The roots of the system are
- the specialObjectsArray - the current context - the extraRoots table (addGCRoot/removeGCRoot's underlying table) - suspended callbacks (processes using the FFI's callback facility)
While the specialObjectsArray contains references to known classes (which indirectly refer to Smalltalk and the entire class hierarchy) and to the process scheduler (Smalltalk associationAt: #Processor) it also contains the external semaphore table, the Character table (byte character instances) and the specialSelectors table (for special selector sends).
The rootTable (better known as a remembered set) is part of the garbage collector and contains any and all old objects that may reference a young object. These objects are roots for an incremental GC, and they get voided at the start of a full GC and after an incrementalGC. See clearRootsTable.
It's an essential part of the generational GC implementation. It is nothing to do with tracing for image segments.
cheers Eliot
_______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: Something I don't see in ImageSegment
|
|
In reply to this post by Mariano Martinez Peck
2010/6/4 Mariano Martinez Peck <[hidden email]>
Its to do with compact classes. 'mariano' class indexIfCompact 11 Object new indexIfCompact 0
The store segment primitive does not consider class references from objects with compact class as external pointers. Arguably this is a bug because it means an image segment is only safely loadable into an image with the same compact classes, but that's a restriction we're willing to live with.
HTH Eliot
_______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: Something I don't see in ImageSegment
|
|
In reply to this post by Eliot Miranda-2
2010/6/4 Eliot Miranda <[hidden email]>
Great. If you want feel free to attach your test and I commit it. Or you create an issue and you do it by yourself.
OK, I didn't know all this :)
Ok...so, my problem was I though that rootTable were indeed the roots of the system, not a particular GC table. Then, I confused because as in GC it needed to iterate it, I though I had to do that also in ImageSegment.
_______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
Re: Something I don't see in ImageSegment
|
|
In reply to this post by Eliot Miranda-2
2010/6/4 Eliot Miranda <[hidden email]>
Thanks Eliot for the hints. But still I don't understand. I didn't understand your comment: "The store segment primitive does not consider class references from objects with compact class as external pointers" In addition, look this example: testOutPointersWithSpecialSharedObjects | anObject segment root a| anObject := TestResult basicNew. anObject timeStamp: Date dateAndTimeNow asString. root := Array with: anObject. segment := ImageSegment new copyFromRoots: root sizeHint: 5000 areUnique: false. self assert: segment outPointers size = 1. self assert: (segment outPointers includes: nil) Of course, this is failing, and the outPointers is: {TestResult. nil}. I don't understand why the CLASS TestResult is there. So..for every simple object I store, it will also store its class? I really really read the primitive to store and try to understand the very low levels. But I cannot see where (image side or primitives) the class can be added to outPointer, nor why. I think it shouldn't be included. Because it is NOT referenced from inside my subgraph. Maybe there is something I am not seeing. I am getting nervous hahahah Now, another question. Date dateAndTimeNow asString answer a ByteString something like this: 'an Array(3 August 2010 12:40:17 pm)' It is ok that this variable is NOT in the outPointers, because it is ONLY accesible from the subgraph. But now, if I change to this: anObject timeStamp: 'an Array(3 August 2010 12:40:17 pm)'. Then outPointers size is 3 and it is: {TestResult. 'an Array(3 August 2010 12:40:17 pm)'. nil} grrrrrrr is that when building string in this way it is referenced from some other place? Thanks!! Mariano
_______________________________________________ Pharo-project mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project |
«
Return to Pharo Smalltalk Developers
|
1 view|%1 views
| Free forum by Nabble | Edit this page |