A new version of System was added to project The Inbox:
http://source.squeak.org/inbox/System-cwp.662.mcz ==================== Summary ==================== Name: System-cwp.662 Author: cwp Time: 12 January 2014, 10:23:54.166 am UUID: 38c55d98-af64-4c5e-abbf-df86ad07fc7b Ancestors: System-cwp.661 Remove the marker object from allObjectsDo:, and return to iterating until 0 is encountered. Also, fetch the next object before evaluating the block, to allow the block to use #become: safely. =============== Diff against System-cwp.661 =============== Item was changed: ----- Method: SystemNavigation>>allObjectsDo: (in category 'query') ----- allObjectsDo: aBlock "Evaluate the argument, aBlock, for each object in the system + excluding SmallIntegers. New objects created by aBlock will + be included in the enumeration, and it is up to the caller to avoid + creating new objects faster than they are enumerated. It's + important that the next object is fetched before the block is + evaluated, because the block may use #become: to change the + identity of object." + + | object nextObject | - excluding SmallIntegers. With closures, this needs to use an end - marker (lastObject) since activation of the block will create new - contexts and cause an infinite loop." - | object lastObject | object := self someObject. + [nextObject := object nextObject. + aBlock value: object. + object := nextObject. + 0 == nextObject] + whileFalse. + ! - lastObject := Object new. - [lastObject == object or: [0 == object]] - whileFalse: [aBlock value: object. - object := object nextObject]! |
That diff is a bit hard to read, so here's the implementation:
allObjectsDo: aBlock "Evaluate the argument, aBlock, for each object in the system excluding SmallIntegers. New objects created by aBlock will
be included in the enumeration, and it is up to the caller to avoid creating new objects faster than they are enumerated. It's
important that the next object is fetched before the block is evaluated, because the block may use #become: to change the
identity of object." | object nextObject |
object := self someObject. [nextObject := object nextObject.
aBlock value: object. object := nextObject. 0 == nextObject]
whileFalse. This version has the following properties: - objects created inside the block will be enumerated - callers may use become inside the block - callers must avoid infinite loop I think these are the most desirable semantics for #allObjectsDo:. However, the risk with this implementation is that it will execute an infinite loop on VMs that create a new context object for each block activation. Cog and the StackVM don't, but John's Mac VM 5.7.4.1 does, according to a quick test I just did. I don't know what the behaviour of older Window and Linux VMs are.
So the question is, what VMs are we planning to support for in 4.5, and do they recycle activation contexts? Colin |
On Sun, Jan 12, 2014 at 10:50:03AM -0500, Colin Putney wrote:
> That diff is a bit hard to read, so here's the implementation: > > > allObjectsDo: aBlock > "Evaluate the argument, aBlock, for each object in the system > excluding SmallIntegers. New objects created by aBlock will > be included in the enumeration, and it is up to the caller to avoid > creating new objects faster than they are enumerated. It's > important that the next object is fetched before the block is > evaluated, because the block may use #become: to change the > identity of object." > > | object nextObject | > object := self someObject. > [nextObject := object nextObject. > aBlock value: object. > object := nextObject. > 0 == nextObject] > whileFalse. > > > This version has the following properties: > > - objects created inside the block will be enumerated > - callers may use become inside the block > - callers must avoid infinite loop > > I think these are the most desirable semantics for #allObjectsDo:. However, > the risk with this implementation is that it will execute an infinite loop > on VMs that create a new context object for each block activation. Cog and > the StackVM don't, but John's Mac VM 5.7.4.1 does, according to a quick > test I just did. I don't know what the behaviour of older Window and Linux > VMs are. > > So the question is, what VMs are we planning to support for in 4.5, and do > they recycle activation contexts? > Certainly the up to date interpreter VM and Cog/StackInterpreter will be used, but I think that there are some folks who still need to use John's Mac VM, so it would be good if we could find a solution that works there also. But I also like your implementation here, it's simple and easy to understand. Dave |
2014/1/12 David T. Lewis <[hidden email]>
I wonder in which case we'd want to do that? I mean that it's difficult to fully manage (absence of) object creation. Especially if we insert lazy behaviors which rely on some sort of proxy (like in Xtreams). allObjectsThatExistAtThisPointInTimeDo: is a reasonable behavior IMO.
|
On 12.01.2014, at 18:40, Nicolas Cellier <[hidden email]> wrote:
Me too. We want to enumerate all objects *created until* sending #allObjectsDo. .
|
On Sun, Jan 12, 2014 at 12:44 PM, Bert Freudenberg <[hidden email]> wrote:
We do? Why? It seems to me that it's better to know, when the loop is finished, that we've seen all objects in the image. If we're looking for certain kinds of objects, (e.g., obsolete behaviours) we know that we've found them all.
Sure, the implementation I posted has tradeoffs. There's the risk of creating an infinite loop, either because of block contexts being created on older VMs, or because the caller creates new objects faster than we can enumerate them. If you're arguing that enumerating only existing objects is an acceptable way to mitigate that risk, ok, that's a valuable viewpoint.
But all other things being equal, I don't see why we'd prefer to enumerate only existing objects. Colin |
On 12.01.2014, at 19:40, Colin Putney <[hidden email]> wrote: > > > > On Sun, Jan 12, 2014 at 12:44 PM, Bert Freudenberg <[hidden email]> wrote: > > Me too. We want to enumerate all objects *created until* sending #allObjectsDo. > > We do? Why? It seems to me that it's better to know, when the loop is finished, that we've seen all objects in the image. If we're looking for certain kinds of objects, (e.g., obsolete behaviours) we know that we've found them all. > > Sure, the implementation I posted has tradeoffs. There's the risk of creating an infinite loop, either because of block contexts being created on older VMs, or because the caller creates new objects faster than we can enumerate them. If you're arguing that enumerating only existing objects is an acceptable way to mitigate that risk, ok, that's a valuable viewpoint. > > But all other things being equal, I don't see why we'd prefer to enumerate only existing objects. You would need to use #allObjectsDo:. but when we enumerate all object, the new ones too, we end up having proxies for proxies for proxies ad infinitum (endless non-loop…) and we can’t know it… signature.asc (1K) Download Attachment |
In reply to this post by David T. Lewis
On Sun, Jan 12, 2014 at 11:07:19AM -0500, David T. Lewis wrote:
> On Sun, Jan 12, 2014 at 10:50:03AM -0500, Colin Putney wrote: > > That diff is a bit hard to read, so here's the implementation: > > > > > > allObjectsDo: aBlock > > "Evaluate the argument, aBlock, for each object in the system > > excluding SmallIntegers. New objects created by aBlock will > > be included in the enumeration, and it is up to the caller to avoid > > creating new objects faster than they are enumerated. It's > > important that the next object is fetched before the block is > > evaluated, because the block may use #become: to change the > > identity of object." > > > > | object nextObject | > > object := self someObject. > > [nextObject := object nextObject. > > aBlock value: object. > > object := nextObject. > > 0 == nextObject] > > whileFalse. > > > > > > This version has the following properties: > > > > - objects created inside the block will be enumerated > > - callers may use become inside the block > > - callers must avoid infinite loop > > > > I think these are the most desirable semantics for #allObjectsDo:. However, > > the risk with this implementation is that it will execute an infinite loop > > on VMs that create a new context object for each block activation. Cog and > > the StackVM don't, but John's Mac VM 5.7.4.1 does, according to a quick > > test I just did. I don't know what the behaviour of older Window and Linux > > VMs are. > > > > So the question is, what VMs are we planning to support for in 4.5, and do > > they recycle activation contexts? > > > > Certainly the up to date interpreter VM and Cog/StackInterpreter will be > used, but I think that there are some folks who still need to use John's > Mac VM, so it would be good if we could find a solution that works there > also. > > But I also like your implementation here, it's simple and easy to understand. Oops, I should have checked before replying. This implementation does not seem to work on any standard interpreter VM, so definitely not a good thing. Dave |
In reply to this post by Tobias Pape
On Sun, Jan 12, 2014 at 1:45 PM, Tobias Pape <[hidden email]> wrote: Say you want to wrap every object in the system in a proxy. Yeah, that's an example of creating objects faster than we can enumerate them. (Or, exactly as fast as we can enumerate them, I suppose). It wouldn't have worked pre-closures, either.
The consensus so far is that we like the safety of a sentinel object, as it lets us do whatever we want within the loop, and potentially not seeing all the objects in the image is an acceptable trade-off for that safety. Fair enough.
Colin |
In reply to this post by Colin Putney-3
For me, "allObjectsDo:" is shorthand for "allObjects do:". - Bert - smime.p7s (8K) Download Attachment |
In reply to this post by Colin Putney-3
On Sun, Jan 12, 2014 at 02:02:08PM -0500, Colin Putney wrote:
> On Sun, Jan 12, 2014 at 1:45 PM, Tobias Pape <[hidden email]> wrote: > > > > Say you want to wrap every object in the system in a proxy. > > You would need to use #allObjectsDo:. but when we enumerate all > > object, the new ones too, we end up having proxies for proxies for > > proxies ad infinitum (endless non-loop?) and we can?t know it? > > > > Yeah, that's an example of creating objects faster than we can enumerate > them. (Or, exactly as fast as we can enumerate them, I suppose). It > wouldn't have worked pre-closures, either. > > The consensus so far is that we like the safety of a sentinel object, as it > lets us do whatever we want within the loop, and potentially not seeing all > the objects in the image is an acceptable trade-off for that safety. Fair > enough. > (Changing subject line because original discussion spans commit notices) The most recent inbox entry System-cwp.663 seems to work fine on an interpreter VM. For my understanding, can you clarify, with respect to the original concern that involves MC proxies - does the problem occur with the interpreter VM as well as Cog, or was it specific to one kind of VM? It is worth noting that allObjectsDo: relies on assumptions about how the objects memory works internally. It requires that #someObject will always answer the object at the lowest address in the object memory, and also that a newly allocated object will always be placed at a higher address location than all other objects. Either of these assumptions is likely to be a problem as new and better object memories and garbage collectors are implemented. Dave |
On 12.01.2014, at 20:42, David T. Lewis <[hidden email]> wrote: > On Sun, Jan 12, 2014 at 02:02:08PM -0500, Colin Putney wrote: >> On Sun, Jan 12, 2014 at 1:45 PM, Tobias Pape <[hidden email]> wrote: >> >> >>> Say you want to wrap every object in the system in a proxy. >>> You would need to use #allObjectsDo:. but when we enumerate all >>> object, the new ones too, we end up having proxies for proxies for >>> proxies ad infinitum (endless non-loop?) and we can?t know it? >>> >> >> Yeah, that's an example of creating objects faster than we can enumerate >> them. (Or, exactly as fast as we can enumerate them, I suppose). It >> wouldn't have worked pre-closures, either. >> >> The consensus so far is that we like the safety of a sentinel object, as it >> lets us do whatever we want within the loop, and potentially not seeing all >> the objects in the image is an acceptable trade-off for that safety. Fair >> enough. >> > > (Changing subject line because original discussion spans commit notices) > > The most recent inbox entry System-cwp.663 seems to work fine on an > interpreter VM. > > For my understanding, can you clarify, with respect to the original concern > that involves MC proxies - does the problem occur with the interpreter VM > as well as Cog, or was it specific to one kind of VM? > > It is worth noting that allObjectsDo: relies on assumptions about how > the objects memory works internally. It requires that #someObject will > always answer the object at the lowest address in the object memory, and > also that a newly allocated object will always be placed at a higher > address location than all other objects. Either of these assumptions is > likely to be a problem as new and better object memories and garbage > collectors are implemented. > > Dave Right (as Eliot's vm-dev post shows). So IMHO the only sensible semantics of allObjectsDo: is as in "allObjects do:" - which might be implemented as a primitive in some VMs soonish. It *should not* enumerate objects created after calling the method. - Bert - |
On 12 January 2014 21:53, Bert Freudenberg <[hidden email]> wrote:
+1 - Bert - -- Best regards, Igor Stasenko. |
In reply to this post by Bert Freudenberg
On Sun, Jan 12, 2014 at 09:53:27PM +0100, Bert Freudenberg wrote:
> > On 12.01.2014, at 20:42, David T. Lewis <[hidden email]> wrote: > > > It is worth noting that allObjectsDo: relies on assumptions about how > > the objects memory works internally. It requires that #someObject will > > always answer the object at the lowest address in the object memory, and > > also that a newly allocated object will always be placed at a higher > > address location than all other objects. Either of these assumptions is > > likely to be a problem as new and better object memories and garbage > > collectors are implemented. > > > > Dave > > Right (as Eliot's vm-dev post shows). > > So IMHO the only sensible semantics of allObjectsDo: is as in "allObjects do:" - > which might be implemented as a primitive in some VMs soonish. It *should not* > enumerate objects created after calling the method. > On Sun, Jan 12, 2014 at 12:01:00PM -0800, Eliot Miranda wrote: > > The bug is in implementing allObjects in terms of someObject and nextObject > in the first place. It's cheap and cheerful but horribly error-prone and > restrictive. It's cheap because the collection of objects doesn't have to > be created, and on the original 16-bit Smalltalk machines that was really > important. It's horribly restrictive because it assumes much about the > implementation. > > Before closures a sentinel wasn't even needed because enumerating the block > didn't create a new object (the block context was reused). So the code had > to be rewritten just to support closures. > > Spur has a generation scavenger operating in a distinct new space and that > doesn't jive well with a consistent ordering at all. So far the system is > limping along by tenuring all objects on someObject and someInstance (so > that newSpace is either empty, or doesn't contain any instances of a > specific class) and having nextObject enumerate only objects in oldSpace. > > But I think now we can afford a primitive that answers all the objects > (remember that average object size means that such a collection will be ~ > 10% of the heap, average object size in Squeak V3 is about 10.6 words). At > least that's what Spur will do, along with an allInstancesOf: primitive. > And then the become example won't cause any problems at all. Far more > reliable. I suppose there are circumstances when enumerating without a > container is the only feasible approach, but VisualWorks has got along with > only an allObjects primitive for a long time now. I suspect we can too. > it should be ok there also. If no objections or better suggestions I will commit it to VMMaker tomorrow. InterpreterPrimitives>>primitiveAllObjects "Answer an array of all objects that exist when the primitive is called, excluding those that may be garbage collected as a side effect of allocating the result array. Multiple references to nil in the last slots of the array are an indication that garbage collection occured, such that some of the unreferenced objects that existed at the time of calling the primitive no longer exist. Sender is responsible for handling multiple references to nil in the result array." Dave InterpreterPrimitives-primitiveAllObjects-dtl.1.cs (2K) Download Attachment |
On 13.01.2014, at 02:13, David T. Lewis <[hidden email]> wrote:
> On Sun, Jan 12, 2014 at 12:01:00PM -0800, Eliot Miranda wrote: >> >> But I think now we can afford a primitive that answers all the objects >> (remember that average object size means that such a collection will be ~ >> 10% of the heap, average object size in Squeak V3 is about 10.6 words). At >> least that's what Spur will do, along with an allInstancesOf: primitive. >> And then the become example won't cause any problems at all. Far more >> reliable. I suppose there are circumstances when enumerating without a >> container is the only feasible approach, but VisualWorks has got along with >> only an allObjects primitive for a long time now. I suspect we can too. >> > > Implementation attached. Works on interpreter VM, not yet tested on Cog but > it should be ok there also. If no objections or better suggestions I will > commit it to VMMaker tomorrow. > > InterpreterPrimitives>>primitiveAllObjects > "Answer an array of all objects that exist when the primitive is called, excluding those > that may be garbage collected as a side effect of allocating the result array. Multiple > references to nil in the last slots of the array are an indication that garbage collection > occured, such that some of the unreferenced objects that existed at the time of calling > the primitive no longer exist. Sender is responsible for handling multiple references to > nil in the result array." > > Dave > > <InterpreterPrimitives-primitiveAllObjects-dtl.1.cs> Nice, except that I'd fill the remaining slots with 0 instead of nil. Even better: allocate the array as object count + 1 and *always* put a 0 last. That way the image code cannot ever "forget" to check for 0. In the image I think the code should still prefer the someObject/nextObject interface, because allocating the large array is unnecessary in almost all cases. We could say a VM is allowed to fail the someObject/nextObject prims if it supports primitiveAllObjects, which otherwise is optional. - Bert - |
On Mon, Jan 13, 2014 at 12:28:33PM +0100, Bert Freudenberg wrote:
> > On 13.01.2014, at 02:13, David T. Lewis <[hidden email]> wrote: > > > On Sun, Jan 12, 2014 at 12:01:00PM -0800, Eliot Miranda wrote: > >> > >> But I think now we can afford a primitive that answers all the objects > >> (remember that average object size means that such a collection will be ~ > >> 10% of the heap, average object size in Squeak V3 is about 10.6 words). At > >> least that's what Spur will do, along with an allInstancesOf: primitive. > >> And then the become example won't cause any problems at all. Far more > >> reliable. I suppose there are circumstances when enumerating without a > >> container is the only feasible approach, but VisualWorks has got along with > >> only an allObjects primitive for a long time now. I suspect we can too. > >> > > > > Implementation attached. Works on interpreter VM, not yet tested on Cog but > > it should be ok there also. If no objections or better suggestions I will > > commit it to VMMaker tomorrow. > > > > InterpreterPrimitives>>primitiveAllObjects > > "Answer an array of all objects that exist when the primitive is called, excluding those > > that may be garbage collected as a side effect of allocating the result array. Multiple > > references to nil in the last slots of the array are an indication that garbage collection > > occured, such that some of the unreferenced objects that existed at the time of calling > > the primitive no longer exist. Sender is responsible for handling multiple references to > > nil in the result array." > > > > Dave > > > > <InterpreterPrimitives-primitiveAllObjects-dtl.1.cs> > > Nice, except that I'd fill the remaining slots with 0 instead of nil. Even better: allocate the array as object count + 1 and *always* put a 0 last. That way the image code cannot ever "forget" to check for 0. > Someone would surely accuse me of being an unreformed C programmer ;-) Another approach might be to provide an explicit object count in addition to the array of objects, so the primitive would answer an array with object count and with the all objects array. Or we might force a GC in the primitive in order to prevent the condition from ever happening, at the cost of an unnecessary GC. > In the image I think the code should still prefer the someObject/nextObject interface, because allocating the large array is unnecessary in almost all cases. We could say a VM is allowed to fail the someObject/nextObject prims if it supports primitiveAllObjects, which otherwise is optional. > > - Bert - |
On 13.01.2014, at 13:16, David T. Lewis <[hidden email]> wrote: > > On Mon, Jan 13, 2014 at 12:28:33PM +0100, Bert Freudenberg wrote: >> >> On 13.01.2014, at 02:13, David T. Lewis <[hidden email]> wrote: >> >>> On Sun, Jan 12, 2014 at 12:01:00PM -0800, Eliot Miranda wrote: >>>> >>>> But I think now we can afford a primitive that answers all the objects >>>> (remember that average object size means that such a collection will be ~ >>>> 10% of the heap, average object size in Squeak V3 is about 10.6 words). At >>>> least that's what Spur will do, along with an allInstancesOf: primitive. >>>> And then the become example won't cause any problems at all. Far more >>>> reliable. I suppose there are circumstances when enumerating without a >>>> container is the only feasible approach, but VisualWorks has got along with >>>> only an allObjects primitive for a long time now. I suspect we can too. >>>> >>> >>> Implementation attached. Works on interpreter VM, not yet tested on Cog but >>> it should be ok there also. If no objections or better suggestions I will >>> commit it to VMMaker tomorrow. >>> >>> InterpreterPrimitives>>primitiveAllObjects >>> "Answer an array of all objects that exist when the primitive is called, excluding those >>> that may be garbage collected as a side effect of allocating the result array. Multiple >>> references to nil in the last slots of the array are an indication that garbage collection >>> occured, such that some of the unreferenced objects that existed at the time of calling >>> the primitive no longer exist. Sender is responsible for handling multiple references to >>> nil in the result array." >>> >>> Dave >>> >>> <InterpreterPrimitives-primitiveAllObjects-dtl.1.cs> >> >> Nice, except that I'd fill the remaining slots with 0 instead of nil. Even better: allocate the array as object count + 1 and *always* put a 0 last. That way the image code cannot ever "forget" to check for 0. >> > > Someone would surely accuse me of being an unreformed C programmer ;-) Nah, I don't think so. It just mimics the old behavior of answering 0 after the last object. Simplifies the image code (which would otherwise have to keep track whether it enumerated nil yet or not). > Another approach might be to provide an explicit object count in addition to the > array of objects, so the primitive would answer an array with object count and > with the all objects array. That's even uglier ;^) > Or we might force a GC in the primitive in order to prevent the condition from > ever happening, at the cost of an unnecessary GC. Forcing a GC would remove the neat feature of being able to access - Bert - >> In the image I think the code should still prefer the someObject/nextObject interface, because allocating the large array is unnecessary in almost all cases. We could say a VM is allowed to fail the someObject/nextObject prims if it supports primitiveAllObjects, which otherwise is optional. >> >> - Bert - |
In reply to this post by David T. Lewis
Hi David, On Sun, Jan 12, 2014 at 5:13 PM, David T. Lewis <[hidden email]> wrote:
Instead of filling the unused slots with nil or 0, I think you should shorten the object so that it contains each object only once, and contains only the objects. Cog contains some code for shortening. See [New]ObjectMemory>>shorten:toIndexableSize:.
HTH -- best, Eliot
|
2014/1/13 Eliot Miranda <[hidden email]>
Oh, I missed this one. I can't check now, could it cleanly address my hugly trick of modifying the header like explained at http://smallissimo.blogspot.fr/2013/04/still-hacking-largeintegerplugins-in.html ? |
Hi Nicolas,
On Mon, Jan 13, 2014 at 10:20 AM, Nicolas Cellier <[hidden email]> wrote:
It would on SqueakV3. On Spur it could be more difficult as there's no support for one-double-word free objects, so one can't free a singe 64-bit pair (objects are rounded up to 64-bits in length). But perhaps you should inflict this on me to force my hand ;-). The problem is that a single 64-bit word is big enough for an object header but not for an object header plus an indirection in the first slot, which takes 129 bits, when rounded up to a 64-bit boundary. So such fragments are only reclaimable if an object next to them is freed. They can't be compacted because there's no forwarding pointer. So I expect in Spur I'd just have to disable the code and always create the copy. However, what *can* be shortened, always, is the last object in eden. So if the object is newly created it can easily be shortened; all that needs ot happen is the allocation pointer, freeStart, is cut-back.
HTH, Eliot
|
Free forum by Nabble | Edit this page |