Prune stack serialization

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Prune stack serialization

tinchodias
Hi folks

In Fuel, we serialize a block closure with its state, including its outerContext. This enables to serialize a sorted collection with its sortBlock:

| bytes result |
bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a :b | a > b ]).
result := FLMaterializer materializeFromByteArray: bytes. 
result 
addAll: #(1 2 3);
yourself.
---> a SortedCollection(3 2 1)

Here the problem: the byte array is huge! (800kb) because we are serializing unneeded context for the sort block. 

We wonder how to prune it and save time and space.

Thanks in advance
Martín
Reply | Threaded
Open this post in threaded view
|

Re: Prune stack serialization

Nicolas Cellier
2011/12/2 Martin Dias <[hidden email]>:

> Hi folks
>
> In Fuel, we serialize a block closure with its state, including its
> outerContext. This enables to serialize a sorted collection with its
> sortBlock:
>
> | bytes result |
> bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a
> :b | a > b ]).
> result := FLMaterializer materializeFromByteArray: bytes.
> result
> addAll: #(1 2 3);
> yourself.
> ---> a SortedCollection(3 2 1)
>
> Here the problem: the byte array is huge! (800kb) because we are serializing
> unneeded context for the sort block.
>
> We wonder how to prune it and save time and space.
>
> Thanks in advance
> Martín

In the case of such clean block, there is no need of outer context
during block execution.
However I don't know if implementation makes it possible to ignore the
context...
That's more a question directed to Eliot ;)

Nicolas

Reply | Threaded
Open this post in threaded view
|

Re: Prune stack serialization

Eliot Miranda-2


On Fri, Dec 2, 2011 at 10:20 AM, Nicolas Cellier <[hidden email]> wrote:
2011/12/2 Martin Dias <[hidden email]>:
> Hi folks
>
> In Fuel, we serialize a block closure with its state, including its
> outerContext. This enables to serialize a sorted collection with its
> sortBlock:
>
> | bytes result |
> bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a
> :b | a > b ]).
> result := FLMaterializer materializeFromByteArray: bytes.
> result
> addAll: #(1 2 3);
> yourself.
> ---> a SortedCollection(3 2 1)
>
> Here the problem: the byte array is huge! (800kb) because we are serializing
> unneeded context for the sort block.
>
> We wonder how to prune it and save time and space.
>
> Thanks in advance
> Martín

In the case of such clean block, there is no need of outer context
during block execution.
However I don't know if implementation makes it possible to ignore the
context...
That's more a question directed to Eliot ;)

Arguably there is a bug in my closure implementation, which is that both the receiver and the method are fetched from the outerContext.  That's not a bug which can be fixed without a new VM/image combination and may be something I'll look at long-term, but is something we have to live with at the moment.  This means that you *do* have to serialize the outerContext.  But the outerContext is used only for the receiver and method.  So you don't need to full serialize the outerContext.  In particular you don't need to serialize any of the outerContext's stack contents or its sender.  This needs special handling, I guess in BlockClosure, to substitute a suitably reduced outerContext, but it shouldn't be hard to do.  e.g.

BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver outerContext receiver
method: outerContext method
args: #()


BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver self receiver
method: self method
args: #()



Nicolas




--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck
Thanks both. I am right to assume that if the block refers to temp vars, parameters, or whatever in another scope, then such solution won't work. I mean, if I have this example for example:

| bytes result blah |
blah := 42.
bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a :b | (a + blah) > b ]).

Then the 'blah' is in a different context. So the mentioned solution works for "clean" closures, which are "self contained". In the other cases (such as this example), we should serialize the whole stack. Is this correct?

Thanks!

On Fri, Dec 2, 2011 at 7:41 PM, Eliot Miranda <[hidden email]> wrote:


On Fri, Dec 2, 2011 at 10:20 AM, Nicolas Cellier <[hidden email]> wrote:
2011/12/2 Martin Dias <[hidden email]>:
> Hi folks
>
> In Fuel, we serialize a block closure with its state, including its
> outerContext. This enables to serialize a sorted collection with its
> sortBlock:
>
> | bytes result |
> bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a
> :b | a > b ]).
> result := FLMaterializer materializeFromByteArray: bytes.
> result
> addAll: #(1 2 3);
> yourself.
> ---> a SortedCollection(3 2 1)
>
> Here the problem: the byte array is huge! (800kb) because we are serializing
> unneeded context for the sort block.
>
> We wonder how to prune it and save time and space.
>
> Thanks in advance
> Martín

In the case of such clean block, there is no need of outer context
during block execution.
However I don't know if implementation makes it possible to ignore the
context...
That's more a question directed to Eliot ;)

Arguably there is a bug in my closure implementation, which is that both the receiver and the method are fetched from the outerContext.  That's not a bug which can be fixed without a new VM/image combination and may be something I'll look at long-term, but is something we have to live with at the moment.  This means that you *do* have to serialize the outerContext.  But the outerContext is used only for the receiver and method.  So you don't need to full serialize the outerContext.  In particular you don't need to serialize any of the outerContext's stack contents or its sender.  This needs special handling, I guess in BlockClosure, to substitute a suitably reduced outerContext, but it shouldn't be hard to do.  e.g.

BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver outerContext receiver
method: outerContext method
args: #()


BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver self receiver
method: self method
args: #()



Nicolas




--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Eliot Miranda-2


On Fri, Dec 2, 2011 at 10:55 AM, Mariano Martinez Peck <[hidden email]> wrote:
Thanks both. I am right to assume that if the block refers to temp vars, parameters, or whatever in another scope, then such solution won't work. I mean, if I have this example for example:

| bytes result blah |
blah := 42.
bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a :b | (a + blah) > b ]).

Then the 'blah' is in a different context. So the mentioned solution works for "clean" closures, which are "self contained". In the other cases (such as this example), we should serialize the whole stack. Is this correct?

No.  The closure implementation arranges that any and all temporary variables accessed by the closure are directly accessible from the closure without accessing the outer contexts.  You were there for my presentation Mariano :)  And you understood the issue then right ? :)  Remember the closure compiler *copies* any temporary variables that are not changed after being closed over into a closure's copiedValues, and *moves* any temporary variables that could be changed after being closed over into indirection vectors, and these indirection vectors are copied into the copied values of the closure.  But, as I just said, the receiver and method are not copied (to save time at closure creation) since these are in the outerContext already.  These are fetched from the outerContext when the closure is activated.

That's the trade-off, slower closure creation and larger closures and duplication if a closure refers to its method and receiver directly, faster closure creation, smaller closures, less duplication and very slightly slower activation if closures refer to their receiver and method indirectly.  The additional trade-off is this issue; the indirect approach entails more problems for serialization and for implementing compile-time clean closures (since the compiler needs to create an outerContext for the method and nil for the non-existent receiver - clean closures can't refer to self).

Is this clear or as clear as mud?
 

Thanks!


On Fri, Dec 2, 2011 at 7:41 PM, Eliot Miranda <[hidden email]> wrote:


On Fri, Dec 2, 2011 at 10:20 AM, Nicolas Cellier <[hidden email]> wrote:
2011/12/2 Martin Dias <[hidden email]>:
> Hi folks
>
> In Fuel, we serialize a block closure with its state, including its
> outerContext. This enables to serialize a sorted collection with its
> sortBlock:
>
> | bytes result |
> bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a
> :b | a > b ]).
> result := FLMaterializer materializeFromByteArray: bytes.
> result
> addAll: #(1 2 3);
> yourself.
> ---> a SortedCollection(3 2 1)
>
> Here the problem: the byte array is huge! (800kb) because we are serializing
> unneeded context for the sort block.
>
> We wonder how to prune it and save time and space.
>
> Thanks in advance
> Martín

In the case of such clean block, there is no need of outer context
during block execution.
However I don't know if implementation makes it possible to ignore the
context...
That's more a question directed to Eliot ;)

Arguably there is a bug in my closure implementation, which is that both the receiver and the method are fetched from the outerContext.  That's not a bug which can be fixed without a new VM/image combination and may be something I'll look at long-term, but is something we have to live with at the moment.  This means that you *do* have to serialize the outerContext.  But the outerContext is used only for the receiver and method.  So you don't need to full serialize the outerContext.  In particular you don't need to serialize any of the outerContext's stack contents or its sender.  This needs special handling, I guess in BlockClosure, to substitute a suitably reduced outerContext, but it shouldn't be hard to do.  e.g.

BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver outerContext receiver
method: outerContext method
args: #()


BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver self receiver
method: self method
args: #()



Nicolas




--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com







--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck


On Fri, Dec 2, 2011 at 8:04 PM, Eliot Miranda <[hidden email]> wrote:


On Fri, Dec 2, 2011 at 10:55 AM, Mariano Martinez Peck <[hidden email]> wrote:
Thanks both. I am right to assume that if the block refers to temp vars, parameters, or whatever in another scope, then such solution won't work. I mean, if I have this example for example:

| bytes result blah |
blah := 42.
bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a :b | (a + blah) > b ]).

Then the 'blah' is in a different context. So the mentioned solution works for "clean" closures, which are "self contained". In the other cases (such as this example), we should serialize the whole stack. Is this correct?

No.  The closure implementation arranges that any and all temporary variables accessed by the closure are directly accessible from the closure without accessing the outer contexts.  

Ahhhhhhh :)
 
You were there for my presentation Mariano :)  And you understood the issue then right ? :)  

Well, yes, but I usually understand in the second or third time ;)
 
Remember the closure compiler *copies* any temporary variables that are not changed after being closed over into a closure's copiedValues, and *moves* any temporary variables that could be changed after being closed over into indirection vectors, and these indirection vectors are copied into the copied values of the closure.  

Excellent. So everything is inside the closure. When you say "copiedValues" I guess you use the variable part of BlockClosure, is that right?
 
But, as I just said, the receiver and method are not copied (to save time at closure creation) since these are in the outerContext already.  These are fetched from the outerContext when the closure is activated.

Got it.
 

That's the trade-off, slower closure creation and larger closures and duplication if a closure refers to its method and receiver directly, faster closure creation, smaller closures, less duplication and very slightly slower activation if closures refer to their receiver and method indirectly.  The additional trade-off is this issue; the indirect approach entails more problems for serialization and for implementing compile-time clean closures (since the compiler needs to create an outerContext for the method and nil for the non-existent receiver - clean closures can't refer to self).

Is this clear or as clear as mud?

Much clear now. One last question. So far, then, I didn't find/think a scenario where serializing the whole stack (from a closure)  make sense. So, if I just want to serialize a closure and be able to materialize it, then it is *never* necessary the whole stack, in all scenarios?  or I am missing a case where the whole stack is needed?   I am talking only about when serializing closures, because if I directly serialize a MethodContext, it may happen that I want the whole stack (like the example I showed in ESUG where I serialized the whole Debugger and then I continue debugging in another image).

Thanks!
 
 

Thanks!


On Fri, Dec 2, 2011 at 7:41 PM, Eliot Miranda <[hidden email]> wrote:


On Fri, Dec 2, 2011 at 10:20 AM, Nicolas Cellier <[hidden email]> wrote:
2011/12/2 Martin Dias <[hidden email]>:
> Hi folks
>
> In Fuel, we serialize a block closure with its state, including its
> outerContext. This enables to serialize a sorted collection with its
> sortBlock:
>
> | bytes result |
> bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a
> :b | a > b ]).
> result := FLMaterializer materializeFromByteArray: bytes.
> result
> addAll: #(1 2 3);
> yourself.
> ---> a SortedCollection(3 2 1)
>
> Here the problem: the byte array is huge! (800kb) because we are serializing
> unneeded context for the sort block.
>
> We wonder how to prune it and save time and space.
>
> Thanks in advance
> Martín

In the case of such clean block, there is no need of outer context
during block execution.
However I don't know if implementation makes it possible to ignore the
context...
That's more a question directed to Eliot ;)

Arguably there is a bug in my closure implementation, which is that both the receiver and the method are fetched from the outerContext.  That's not a bug which can be fixed without a new VM/image combination and may be something I'll look at long-term, but is something we have to live with at the moment.  This means that you *do* have to serialize the outerContext.  But the outerContext is used only for the receiver and method.  So you don't need to full serialize the outerContext.  In particular you don't need to serialize any of the outerContext's stack contents or its sender.  This needs special handling, I guess in BlockClosure, to substitute a suitably reduced outerContext, but it shouldn't be hard to do.  e.g.

BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver outerContext receiver
method: outerContext method
args: #()


BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver self receiver
method: self method
args: #()



Nicolas




--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com







--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck


Is this clear or as clear as mud?

Much clear now. One last question. So far, then, I didn't find/think a scenario where serializing the whole stack (from a closure)  make sense. So, if I just want to serialize a closure and be able to materialize it, then it is *never* necessary the whole stack, in all scenarios?  or I am missing a case where the whole stack is needed?   I am talking only about when serializing closures, because if I directly serialize a MethodContext, it may happen that I want the whole stack (like the example I showed in ESUG where I serialized the whole Debugger and then I continue debugging in another image).


For example, would this work?

| bytes result blah |
blah := 42.
bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a :b | (a + blah) > b. ^ blah ]).

has the return something to do with this?

Thanks


Thanks!
 
 

Thanks!


On Fri, Dec 2, 2011 at 7:41 PM, Eliot Miranda <[hidden email]> wrote:


On Fri, Dec 2, 2011 at 10:20 AM, Nicolas Cellier <[hidden email]> wrote:
2011/12/2 Martin Dias <[hidden email]>:
> Hi folks
>
> In Fuel, we serialize a block closure with its state, including its
> outerContext. This enables to serialize a sorted collection with its
> sortBlock:
>
> | bytes result |
> bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a
> :b | a > b ]).
> result := FLMaterializer materializeFromByteArray: bytes.
> result
> addAll: #(1 2 3);
> yourself.
> ---> a SortedCollection(3 2 1)
>
> Here the problem: the byte array is huge! (800kb) because we are serializing
> unneeded context for the sort block.
>
> We wonder how to prune it and save time and space.
>
> Thanks in advance
> Martín

In the case of such clean block, there is no need of outer context
during block execution.
However I don't know if implementation makes it possible to ignore the
context...
That's more a question directed to Eliot ;)

Arguably there is a bug in my closure implementation, which is that both the receiver and the method are fetched from the outerContext.  That's not a bug which can be fixed without a new VM/image combination and may be something I'll look at long-term, but is something we have to live with at the moment.  This means that you *do* have to serialize the outerContext.  But the outerContext is used only for the receiver and method.  So you don't need to full serialize the outerContext.  In particular you don't need to serialize any of the outerContext's stack contents or its sender.  This needs special handling, I guess in BlockClosure, to substitute a suitably reduced outerContext, but it shouldn't be hard to do.  e.g.

BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver outerContext receiver
method: outerContext method
args: #()


BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver self receiver
method: self method
args: #()



Nicolas




--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com







--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck


On Fri, Dec 2, 2011 at 8:18 PM, Mariano Martinez Peck <[hidden email]> wrote:


Is this clear or as clear as mud?

Much clear now. One last question. So far, then, I didn't find/think a scenario where serializing the whole stack (from a closure)  make sense. So, if I just want to serialize a closure and be able to materialize it, then it is *never* necessary the whole stack, in all scenarios?  or I am missing a case where the whole stack is needed?   I am talking only about when serializing closures, because if I directly serialize a MethodContext, it may happen that I want the whole stack (like the example I showed in ESUG where I serialized the whole Debugger and then I continue debugging in another image).


For example, would this work?


| bytes result blah |
blah := 42.
bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a :b | (a + blah) > b. ^ blah ]).


of course it doesn't make sense to do a return there. I was just an example, imagine another place where there is a blockclosure I want to serialize and there is a return inside.


 
has the return something to do with this?

Thanks


Thanks!
 
 

Thanks!


On Fri, Dec 2, 2011 at 7:41 PM, Eliot Miranda <[hidden email]> wrote:


On Fri, Dec 2, 2011 at 10:20 AM, Nicolas Cellier <[hidden email]> wrote:
2011/12/2 Martin Dias <[hidden email]>:
> Hi folks
>
> In Fuel, we serialize a block closure with its state, including its
> outerContext. This enables to serialize a sorted collection with its
> sortBlock:
>
> | bytes result |
> bytes := FLSerializer serializeToByteArray: (SortedCollection sortBlock: [:a
> :b | a > b ]).
> result := FLMaterializer materializeFromByteArray: bytes.
> result
> addAll: #(1 2 3);
> yourself.
> ---> a SortedCollection(3 2 1)
>
> Here the problem: the byte array is huge! (800kb) because we are serializing
> unneeded context for the sort block.
>
> We wonder how to prune it and save time and space.
>
> Thanks in advance
> Martín

In the case of such clean block, there is no need of outer context
during block execution.
However I don't know if implementation makes it possible to ignore the
context...
That's more a question directed to Eliot ;)

Arguably there is a bug in my closure implementation, which is that both the receiver and the method are fetched from the outerContext.  That's not a bug which can be fixed without a new VM/image combination and may be something I'll look at long-term, but is something we have to live with at the moment.  This means that you *do* have to serialize the outerContext.  But the outerContext is used only for the receiver and method.  So you don't need to full serialize the outerContext.  In particular you don't need to serialize any of the outerContext's stack contents or its sender.  This needs special handling, I guess in BlockClosure, to substitute a suitably reduced outerContext, but it shouldn't be hard to do.  e.g.

BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver outerContext receiver
method: outerContext method
args: #()


BlockClosure methods for: '*Fuel-serialization'
outerContextForSerialization
^MethodContext
sender: nil
receiver self receiver
method: self method
args: #()



Nicolas




--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com







--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck
In reply to this post by Eliot Miranda-2


On Fri, Dec 2, 2011 at 8:30 PM, Juan Vuletich <[hidden email]> wrote:
Eliot Miranda wrote:


On Fri, Dec 2, 2011 at 10:55 AM, Mariano Martinez Peck <[hidden email] <mailto:[hidden email]>> wrote:

   Thanks both. I am right to assume that if the block refers to temp
   vars, parameters, or whatever in another scope, then such solution
   won't work. I mean, if I have this example for example:

   | bytes result blah |
   blah := 42.
   bytes := FLSerializer serializeToByteArray: (SortedCollection
   sortBlock: [:a :b | (a + blah) > b ]).

   Then the 'blah' is in a different context. So the mentioned
   solution works for "clean" closures, which are "self contained".
   In the other cases (such as this example), we should serialize the
   whole stack. Is this correct?


No.  The closure implementation arranges that any and all temporary variables accessed by the closure are directly accessible from the closure without accessing the outer contexts.
 ...

WRT clean closures, check what I did in Cuis to serialize SortedCollections. See implementors and senders of #isClean.


Nice. Thanks Juan. I was checking your code, and that's exactly why I asked Eliot. In your method you say:

isClean
    "A clean closure is one that doesn't really need the home context because:
        - It doesn't send messages to self or super
        - It doesn't access any instance variable
        - It doesn't access any outer temp
        - It doesn't do ^ return"
.....

So... my question is, WHAT do I need to serialize if I want to be able to serialize also "non clean". I mean, for each item on that list, what do I need apart from the closure instance and the receiver and method from the outerContext ?  the whole stack of contexts ?


Thanks a lot in advance!



--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck
The following method looks complicated for me. Do you know if it is still valid in latest Cog ?
Thanks :)


eliotsClosureMeasurementsOn: m over: aFiveArgBlock
    "
    See senders.
    Or try something like:
        Smalltalk
            eliotsClosureMeasurementsOn: FileList >> #defaultContents
            over: [ :closuresCount :hasCopiedValuesForClosure :hasIndirectTemps :anyClosureHasCopied :anyClosureDoesUAR :anyClosureUsesSelf |
                (Array with: closuresCount with: hasCopiedValuesForClosure with: hasIndirectTemps with: anyClosureHasCopied with: anyClosureDoesUAR with: anyClosureUsesSelf)]

    From http://www.mirandabanda.org/cogblog/2008/11/14/mechanised-modifications-and-miscellaneous-measurements/
    by Eliot Miranda
    "
    | s nextScanStart thisClosureHasCopied closuresCount hasIndirectTemps blkPc blkSz anyClosureHasCopied anyClosureDoesUAR anyClosureUsesSelf analyzedClosures |
    closuresCount := 0.
    hasIndirectTemps := false.
    anyClosureHasCopied :=  anyClosureDoesUAR := anyClosureUsesSelf := false.
    s := InstructionStream on: m.
    s scanFor: [ :b |
        b = 16r8F "16r8F = 143 closure creation" ifTrue: [
            closuresCount := closuresCount + 1].
        (b = 16r8A "16r8A = 138indirect temp vector creation" and: [ s followingByte <= 127]) ifTrue: [
                hasIndirectTemps := true].
        false].
    nextScanStart := m initialPC.
    analyzedClosures := 0.
    [ analyzedClosures < closuresCount ] whileTrue: [
        s pc: nextScanStart; scanFor: [ :b | b = 16r8F ].    "16r8F = 143 Search for first closure"
        analyzedClosures := analyzedClosures + 1.
        thisClosureHasCopied := s followingByte >= 16r10.
        anyClosureHasCopied := anyClosureHasCopied | thisClosureHasCopied.
        blkSz := s interpretNextInstructionFor: BlockStartLocator new.        "Findout size of first closure"
        blkPc := s pc.
        s scanFor: [ :b |
            s pc >= (blkPc + blkSz)
                ifTrue: [
                    nextScanStart := s pc.
                    true]
                ifFalse: [
                    b = 16r8F ifTrue: [           
                        thisClosureHasCopied := s followingByte >= 16r10.
                        anyClosureHasCopied := anyClosureHasCopied | thisClosureHasCopied.
                        analyzedClosures := analyzedClosures + 1 ].
                    anyClosureDoesUAR := anyClosureDoesUAR or: [s willReturn and: [s willBlockReturn not]].
                    anyClosureUsesSelf := anyClosureUsesSelf or: [b = 16r70 "pushSelf"
                                        or: [b < 16r10 "pushInstVar"
                                        or: [(b = 16r80 and: [s followingByte <= 16r3F]) "pushInstVar"
                                        or: [(b between: 16r60 and: 16r60 + 7) "storePopInstVar"
                                        or: [(b = 16r82 and: [s followingByte <= 63]) "storePopInstVar"
                                        or: [(b = 16r81 and: [s followingByte <= 63]) "storeInstVar"
                                        or: [b = 16r84 and: [s followingByte = 160]]]]]]]].
                    false]]].
    ^aFiveArgBlock valueWithArguments: (Array
            with: closuresCount
            with: hasIndirectTemps
            with: anyClosureHasCopied
            with: anyClosureDoesUAR
            with: anyClosureUsesSelf)

On Fri, Dec 2, 2011 at 9:40 PM, Mariano Martinez Peck <[hidden email]> wrote:


On Fri, Dec 2, 2011 at 8:30 PM, Juan Vuletich <[hidden email]> wrote:
Eliot Miranda wrote:


On Fri, Dec 2, 2011 at 10:55 AM, Mariano Martinez Peck <[hidden email] <mailto:[hidden email]>> wrote:

   Thanks both. I am right to assume that if the block refers to temp
   vars, parameters, or whatever in another scope, then such solution
   won't work. I mean, if I have this example for example:

   | bytes result blah |
   blah := 42.
   bytes := FLSerializer serializeToByteArray: (SortedCollection
   sortBlock: [:a :b | (a + blah) > b ]).

   Then the 'blah' is in a different context. So the mentioned
   solution works for "clean" closures, which are "self contained".
   In the other cases (such as this example), we should serialize the
   whole stack. Is this correct?


No.  The closure implementation arranges that any and all temporary variables accessed by the closure are directly accessible from the closure without accessing the outer contexts.
 ...

WRT clean closures, check what I did in Cuis to serialize SortedCollections. See implementors and senders of #isClean.


Nice. Thanks Juan. I was checking your code, and that's exactly why I asked Eliot. In your method you say:

isClean
    "A clean closure is one that doesn't really need the home context because:
        - It doesn't send messages to self or super
        - It doesn't access any instance variable
        - It doesn't access any outer temp
        - It doesn't do ^ return"
.....

So... my question is, WHAT do I need to serialize if I want to be able to serialize also "non clean". I mean, for each item on that list, what do I need apart from the closure instance and the receiver and method from the outerContext ?  the whole stack of contexts ?


Thanks a lot in advance!



--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck
In reply to this post by Mariano Martinez Peck

   WRT clean closures, check what I did in Cuis to serialize
   SortedCollections. See implementors and senders of #isClean.


Nice. Thanks Juan. I was checking your code, and that's exactly why I asked Eliot. In your method you say:

isClean
   "A clean closure is one that doesn't really need the home context because:
       - It doesn't send messages to self or super
       - It doesn't access any instance variable
       - It doesn't access any outer temp
       - It doesn't do ^ return"
.....

So... my question is, WHAT do I need to serialize if I want to be able to serialize also "non clean". I mean, for each item on that list, what do I need apart from the closure instance and the receiver and method from the outerContext ?  the whole stack of contexts ?


I think it is needed to know why you need to serialize a non-clean closure. If it is just to be able to evaluate it,

well, some people may just need that, but there are cases where I need it completely. I mean, I need to serialize the whole possible graph of objects. This is an example for what I am doing in my PhD. We are also experiment with remote messages and things like that and sometimes we pass a whole stack. In that case I thin it also make sense. But again, these are particular cases. I agree that most of the times the closure will be "clean"
 
maybe you don't need the stack at all. My trick could be augmented by serializing any referenced objects, but not the stack.


Yes, maybe I was not clear. When I refer to "stack" I refer all instances of ContextPart (well, subclasses) and all the objects reachable from there.

 
Cheers,
Juan Vuletich



Thanks a lot in advance!



--
Mariano
http://marianopeck.wordpress.com





--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck
In reply to this post by Mariano Martinez Peck
Thank you very much Eliot and Juan for this nice discussion. I have just commit this new functionality and it seems to work fine. I ended up using Eliot's #abstractBytecodes since the one from Juan didn't work out of the box in Pharo.


On Fri, Dec 2, 2011 at 10:38 PM, Eliot Miranda <[hidden email]> wrote:
Hi Juan,

On Fri, Dec 2, 2011 at 11:59 AM, Juan Vuletich <[hidden email]> wrote:
Mariano Martinez Peck wrote:


On Fri, Dec 2, 2011 at 9:46 PM, Eliot Miranda <[hidden email] <mailto:[hidden email]>> wrote:



   On Fri, Dec 2, 2011 at 12:40 PM, Mariano Martinez Peck
   <[hidden email] <mailto:[hidden email]>> wrote:



       On Fri, Dec 2, 2011 at 8:30 PM, Juan Vuletich
       <[hidden email] <mailto:[hidden email]>> wrote:

           Eliot Miranda wrote:



               On Fri, Dec 2, 2011 at 10:55 AM, Mariano Martinez Peck
               <[hidden email] <mailto:[hidden email]>
               <mailto:[hidden email]

               <mailto:[hidden email]>>> wrote:

                  Thanks both. I am right to assume that if the block
               refers to temp
                  vars, parameters, or whatever in another scope,
               then such solution
                  won't work. I mean, if I have this example for example:

                  | bytes result blah |
                  blah := 42.
                  bytes := FLSerializer serializeToByteArray:
               (SortedCollection
                  sortBlock: [:a :b | (a + blah) > b ]).

                  Then the 'blah' is in a different context. So the
               mentioned
                  solution works for "clean" closures, which are
               "self contained".
                  In the other cases (such as this example), we
               should serialize the
                  whole stack. Is this correct?


               No.  The closure implementation arranges that any and
               all temporary variables accessed by the closure are
               directly accessible from the closure without accessing
               the outer contexts.
                ...


           WRT clean closures, check what I did in Cuis to serialize
           SortedCollections. See implementors and senders of #isClean.


       Nice. Thanks Juan. I was checking your code, and that's
       exactly why I asked Eliot. In your method you say:

       isClean
           "A clean closure is one that doesn't really need the home
       context because:
               - It doesn't send messages to self or super
               - It doesn't access any instance variable
               - It doesn't access any outer temp
               - It doesn't do ^ return"
       .....

       So... my question is, WHAT do I need to serialize if I want to
       be able to serialize also "non clean".


   IMO, dy default, everything reachable from the closure, including
   the outerContext chain and their senders.  But that's only a
   default.  What makes sense in any given context is up to the
   application, e.g. see Seaside.  One size won't fit all.  You
   should handle the common cases (e.g. clean blocks and everything
   else) and leave it up to the user to figure out what they need to
   do in complex situations, making sure you provide them with the
   hooks they'll need, such as the ability to substitute objects when
   serializing etc, and perhaps some useful examples.


+1   Then we will do the following: we will check whether the closure is clean. If it is, then we serialize just the closure plus its outerContext but only with receiver and method. If it is not clean, then we serialize the whole stack. That's all by default. If then someone needs something different, then to use the hooks :)
So the final question is whether what Juan is using in Cuis is up to date. If it is, then I have all I need :)
Thank you guys.

I can say that it does indeed work. But I'd be really grateful if Eliot took a good look at that code!

There is a much better way, using InstructionStream>>interpreetNextInstructionFor: which abstracts away from the bytecode encoding and allows one to simply use messages.  e.g. look at 

BlockClosure methods for private
abstractBytecodes
"Answer an OrderedCollection of the abstract bytecodes sent in the receiver."
| scanner blockCreationMessage end abstractBytecodes |
scanner := InstructionStream new.
"find the creation message to find the extent of the block."
scanner  method: self method pc: self method initialPC.
[scanner pc < startpc] whileTrue:
[[scanner interpretNextInstructionFor: nil]
on: MessageNotUnderstood
do: [:ex| blockCreationMessage := ex message]].
end := blockCreationMessage arguments last + startpc - 1.
"now collect all the bytecode messages in the block."
abstractBytecodes := OrderedCollection new.
scanner method: self method pc: startpc.
[scanner pc < end] whileTrue:
[[scanner interpretNextInstructionFor: nil]
on: MessageNotUnderstood
do: [:ex| abstractBytecodes add: ex message selector]].
^abstractBytecodes

and (SortedCollection sortBlock: [:a :b| a compare: b caseSensitive: false]) sortBlock abstractBytecodes evaluates to

an OrderedCollection(#pushTemporaryVariable: #pushTemporaryVariable: #pushConstant: #send:super:numArgs:)

So a check for clean blocks would check for a block's abstractBytecodes not including some particular set of banned messages (which I *think* is pushReceiver pushReceiverVariable: popIntoReceiverVariable: storeIntoReceiverVariable: methodReturnConstant: methodReturnReceiver methodReturnTop).

e.g.

isClean
      ^(self abstractBytecodes intersection: #(pushReceiver pushReceiverVariable: popIntoReceiverVariable: storeIntoReceiverVariable: methodReturnConstant: methodReturnReceiver methodReturnTop)) isEmpty


Cheers,
Juan Vuletich


 

       I mean, for each item on that list, what do I need apart from
       the closure instance and the receiver and method from the
       outerContext ?  the whole stack of contexts ?


       Thanks a lot in advance!



       --         Mariano
       http://marianopeck.wordpress.com







   --     best,
   Eliot







--
Mariano
http://marianopeck.wordpress.com

------------------------------------------------------------------------


 
------------------------------------------------------------------------

No virus found in this message.
Checked by AVG - www.avg.com <http://www.avg.com>
Version: 10.0.1411 / Virus Database: 2102/4052 - Release Date: 12/02/11






--
best,
Eliot







--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: [squeak-dev] Re: Prune stack serialization

Mariano Martinez Peck
In reply to this post by Mariano Martinez Peck
Hi guys. When we implemented this possibility of pruning the closures when they were clean, one of our crazy tests started to fail. It is not that the test is very important, but I am interested in trying to understand why it fails. The test (simplied) is:

    | aSortedCollection materialized |
    instanceVariableForTesting := false.
   
    aSortedCollection := SortedCollection sortBlock: [:a :b |
        instanceVariableForTesting
            ifTrue: [ a <= b ]
            ifFalse: [ a >= b ] ].

    materialized := self resultOfSerializeAndMaterialize: aSortedCollection.

    materialized addAll: #(2 3 1).
    aSortedCollection addAll: #(2 3 1).


Well, the thing is that the materialized SortedCollection, it is sortBLock, the closure is quite crazy and the 'instanceVariableForTesting' doesn't have the boolean but instead a #Processor    WTF??   So then the #addAll:  fails with a MustBeBoolean. 

Do you think there is something I should check for this closure?  I think it may have to be related with the instanceVariable...

Just for the record, what we are doing is something like:

serializeReferencesOf: aBlockClosure with: aSerialization   

    1 to: aBlockClosure basicSize do: [ :index |
        aSerialization encodeReferenceTo: (aBlockClosure basicAt: index) ].
   
    aBlockClosure isClean
        ifTrue: [
            aSerialization stream nextPut: 1.
            aSerialization encodeReferenceTo:  aBlockClosure method.
            aSerialization encodeReferenceTo:  aBlockClosure receiver.
            ]
        ifFalse: [
            aSerialization stream nextPut: 0.
            aSerialization encodeReferenceTo:  aBlockClosure outerContext.
        ].
   
    aSerialization encodeReferenceTo:  aBlockClosure startpc.
    aSerialization encodeReferenceTo:  aBlockClosure numArgs.
   


-----

materializeReferencesOf: aBlockClosure with: aMaterialization

    | methodOrContext method context |   
    1 to: aBlockClosure basicSize do: [ :index |
        aBlockClosure basicAt: index put: (aMaterialization nextEncodedReference).
    ].

    (aMaterialization stream next = 1)
        ifTrue: [ " it was a clean BlockClosure, otherwise it should have been 0"
            method := aMaterialization nextEncodedReference.
            aBlockClosure
                    outerContext:     (MethodContext
                                        sender: nil
                                        receiver: aMaterialization nextEncodedReference
                                        method: method
                                        arguments: #() )
                    startpc: aMaterialization nextEncodedReference
                    numArgs: aMaterialization nextEncodedReference.           
            ]
        ifFalse: [   
                context := aMaterialization nextEncodedReference.
                aBlockClosure
                    outerContext: context
                    startpc: aMaterialization nextEncodedReference
                    numArgs: aMaterialization nextEncodedReference].
   

-----


Thanks in advance!

 


On Sun, Dec 4, 2011 at 8:33 PM, Mariano Martinez Peck <[hidden email]> wrote:
Thanks Juan. With the latest version of Eliot, the following are failing:


  self deny: [ | outerBlockTemp | [ outerBlockTemp printString ] isClean ] value.
  self deny: [ | outerBlockTemp | [ outerBlockTemp :=7 ] isClean ] value.
  self deny: [ tempVar + 1 ] isClean.
  self deny: [ tempVar := 1 ] isClean.
  self deny: [ ClassVar + 1 ] isClean.
  self deny: [ ClassVar := 1 ] isClean.

Thanks


On Sat, Dec 3, 2011 at 3:01 AM, Juan Vuletich <[hidden email]> wrote:
Eliot Miranda wrote:



On Fri, Dec 2, 20Tha11 at 4:01 PM, Juan Vuletich <[hidden email] <mailto:[hidden email]>> wrote:

   Eliot Miranda wrote:

       Hi Juan,

       ...



          I can say that it does indeed work. But I'd be really
       grateful if
          Eliot took a good look at that code!


       There is a much better way, using
       InstructionStream>>interpreetNextInstructionFor: which
       abstracts away from the bytecode encoding and allows one to
       simply use messages.  e.g. look at

       and (SortedCollection sortBlock: [:a :b| a compare: b
       caseSensitive: false]) sortBlock abstractBytecodes evaluates to

       an OrderedCollection(#pushTemporaryVariable:
       #pushTemporaryVariable: #pushConstant: #send:super:numArgs:)

       So a check for clean blocks would check for a block's
       abstractBytecodes not including some particular set of banned
       messages (which I *think* is pushReceiver
       pushReceiverVariable: popIntoReceiverVariable:
       storeIntoReceiverVariable: methodReturnConstant:
       methodReturnReceiver methodReturnTop).

       e.g.

       isClean
            ^(self abstractBytecodes intersection: #(pushReceiver
       pushReceiverVariable: popIntoReceiverVariable:
       storeIntoReceiverVariable: methodReturnConstant:
       methodReturnReceiver methodReturnTop)) isEmpty


          Cheers,
          Juan Vuletich




       --         best,
       Eliot


   Thanks Eliot. However, it looks to me that this needs more tweaking.

<blush>Damn right :)  I've done zero testing.  It was just an example</blush>
 
   For instance, your method answers true for
     [ xx size ] isClean
   and
     [ ^ nil ] isClean
   while mine answers false for both. Maybe we could serialize non
   local returns using the Compiler (although we won't be able to
   return anywhere), but external variable acess is a problem.

   Will keep playing with this for a while...


Can you (can I??) write some tests?  What's xx in the above?  Ah! [^nil] isn't clean in my code since it should be pc <= end, not pc < end!!  Try the attached...

This is a start:

testIsClean
  "
  ClosureTests new testIsClean
  "
  | tempVar |
  tempVar := 1.
  self assert: [ 3 + 4 ] isClean.
  self assert: [ :a | a * 2 ] isClean.
  self assert: [ Smalltalk size ] isClean.
  self assert: [ :blockArg | blockArg printString ] isClean.
  self assert: [ | blockTemp | blockTemp printString ] isClean.
  self assert: [ | blockTemp | blockTemp :=7 ] isClean.
  self deny: [ | outerBlockTemp | [ outerBlockTemp printString ] isClean ] value.
  self deny: [ | outerBlockTemp | [ outerBlockTemp :=7 ] isClean ] value.
  self deny: [ tempVar + 1 ] isClean.
  self deny: [ tempVar := 1 ] isClean.
  self deny: [ ivar + 1 ] isClean.
  self deny: [ ivar := 1 ] isClean.
  self deny: [ ^ true ] isClean.
  self deny: [ self printString ] isClean.
  self deny: [ ^ self ] isClean.
  self deny: [ ClassVar + 1 ] isClean.
  self deny: [ ClassVar := 1 ] isClean.

This test passes with my implementation.


 

   Cheers,
   Juan Vuletich




--
best,
Eliot


Cheers,
Juan Vuletich




--
Mariano
http://marianopeck.wordpress.com




--
Mariano
http://marianopeck.wordpress.com