binary serialization

> Robert,
>
> As far as I can remember, the problem of substitutions at
> materialziation time was because... as you may have read, Fuel first
> serializes "objects" and then the references. At materialization, it
> first creates the objects and then, on a second step, sets the
> references betwen objects. So the problem was WHERE to place the hook,
> at which point in time. If we do it just after objects were created,
> then the substitution code would NOT have access to any of the instVars
> of that object. If we do it AFTER objects creation and after objects
> references, then there is no easy way to "substitute" which doesn't
> involve #become: (because the graph is already constructed) And you know
> that #become: is still terrible slow because it scans full memory. That
> will change in Spur.

> As for Marea and Ghost,
>
> Ghost proxies paper: https://hal.inria.fr/hal-01081236/document
> Current Ghost repo: http://smalltalkhub.com/#!/~CAR/Ghost
>
> Marea paper: http://www.jot.fm/issues/issue_2013_01/article2.pdf
> Current repo: http://ss3.gemstone.com/ss/Marea.html
>
> And finally, my PhD thesis:
> https://tel.archives-ouvertes.fr/tel-00764991/document

> In Marea I needed custom clusters for my proxies because the serializer
> itself sends messages to the objects being serialized. My proxies would
> bring back graphs from a secondary memory. So if I was serializing a
> graph that had proxies already, I didn't want that. So I hooked my
> custom cluster for proxies that send only a few special messages to the
> proxy that these understand and answer rather than intercept those
> messages.
>
> As for how to extend Fuel for this, I recommend to check the code of
> Marea. See categories  'Marea-Serializers'  and 'Marea-Proxies'.
> I have a Marea one click here:
> https://www.dropbox.com/sh/xp8jzyypmz0898j/AACRdHno6V7UfhaJ1ofTPPXva?dl=0
>
> Cheers,

>
>
> On Wed, Oct 14, 2015 at 11:40 AM, Robert Withers
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>     Good morning, Max. Thank you for the example. I got a little
>     confused, between migrations and substitutions. My issue with no-arg
>     blocks, I believe, is the inability to access my scope object to
>     maintain the object tables.
>
>     I'm attempting to write my own Materializer, Decoder and
>     Materialization. At the moment, I'm just going to walk the graph,
>     testing and do #becomes:. See how well that works when I get a test.
>
>     It's really good to know about that other list.
>
>     thanks so much ^^
>     Robert
>
>     On 10/14/2015 04:15 AM, Max Leske wrote:
>
>         BTW, there is a dedicated Fuel mailing list:
>         [hidden email]
>         <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>
>
>         Max
>
>
>             On 14 Oct 2015, at 09:45, Max Leske <[hidden email]
>             <mailto:[hidden email]>
>             <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>
>
>                 On 14 Oct 2015, at 04:39, Robert Withers
>                 <[hidden email]
>                 <mailto:[hidden email]>
>                 <mailto:[hidden email]
>                 <mailto:[hidden email]>>> wrote:
>
>
>                 On 10/13/2015 09:43 PM, Mariano Martinez Peck wrote:
>
>
>
>                     On Tue, Oct 13, 2015 at 10:33 PM, Robert Withers
>                     <[hidden email]
>                     <mailto:[hidden email]>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>> wrote:
>
>                        Hi Mariano,
>
>                        This presents me with a big challenge, then. I
>                     read the docs and
>                        explored the code and the only other aspect not
>                     mentioned, beyond
>                        instance creation (#fuelNew #fuelNew:) and
>                     postMaterialization
>                        (#fuelAfterMaterialization), is migrations.
>                     However, migration only
>                        allows for instanceVar mappings, no code blocks.
>
>
>                     What do you mean that migrations only allows instVar
>                     mappings , and no
>                     code blocks? I mean, what do you mean by code blocks?
>
>
>             Sounds to me like this (see FuelOutStackDebuAction):
>
>             serializeTestFailureContext: aContext toFileNamed: aFilename
>             | serializer |
>
>             serializer := FLSerializer newDefault.
>             self encodeDebugInformationOn: serializer.
>             serializer addPostMaterializationAction: [ :materialization |
>             Smalltalk tools debugger
>             openOn: Processor activeProcess
>             context: materialization root
>             label: 'External stack'
>             contents: nil
>             fullView: false ].
>
>             serializer
>             " use the sender context, generally the current context is not
>             interesting"
>             serialize: aContext
>             toFileNamed: aFilename
>
>             This stores a block in the serialization which is evaluated
>             after
>             materialization. The only requirement is that it’s a clean
>             block (no
>             closure!).
>
>                     We also support class renames. This is here:
>                     http://rmod.inria.fr/web/software/Fuel/Version1.9/Documentation/Migration?_s=Pvs4DYqPRjfEy8sg&_k=X6ltJu7FDSxumm4X&_n&22
>
>                     Which kind of migration example you have in mind
>                     that would not be
>                     supported? An example would help.
>
>
>                 Well, my pics will demonstrate. I am interested in doing
>                 more than
>                 mappping ivars or a class rename. I want to do a total
>                 substitution,
>                 then a further substitution on the receiving, import side:
>
>                 Vat1: anObject (Class A) ---> On wire: desc (Descriptor)
>                 ---> Vat2:
>                 aProxy (Class FarERef)
>
>                 A desc is substituted for a PassByProxy object, then a
>                 FarERef is
>                 substituted for the desc.
>
>
>                        #fuelAccept: is a serialization side method.
>
>                        If Fuel supports substitution on serialization, I
>                     don't understand
>                        why no substitution support on materialization.
>
>
>                     There was a reason, which I cannot remember
>                     completely. Maybe Martin or
>                     Max can remember.
>
>
>                 It seems your focus was pickling to disk then back. My
>                 focus is
>                 distributed proxy graphs, which has different use cases.
>
>
>
>                        I am definitely going to use the world-class Fuel
>                     binary
>                        serialization system. However, I find myself
>                     needing to extend Fuel
>                        to support substitution on materialization.
>                     Perhaps the solution is
>                        a custom decoder.
>
>
>                     I have made custom clusters for example for my Ghost
>                     proxies of Marea
>                     system. It was a perfect example of how I could
>                     extent Fuel besides the
>                     common hooks. Fuel provides many places for
>                     extending , like clusters,
>                     analyzer,  etc
>
>
>                 Right on, exactly! Could you tell me more about your
>                 Ghost proxies
>                 and Marea, please? As well, could you mention how you
>                 select a custom
>                 cluster on the serialization side?
>
>
>                 thanks so much ^^
>                 Robert
>
>
>
>                        No, a bit more. It looks like I need a new
>                        FLSubstitutePointerObjectCluster, write them on
>                     serialization with
>                        the substitute, then do unsubstitution on
>                     materialization, since the
>                        cluster controls materialization and not the decoder.
>
>                        Does this approach seem sound to you, from a you
>                     know architecture
>                        and design approach?
>
>
>                     There was an issue. Hope other can remember. If not,
>                     I will try to
>                     explan what I remember tomorrow.
>
>
>                        thanks so much ^^
>                        Robert
>
>                        On 10/13/2015 04:49 PM, Mariano Martinez Peck wrote:
>
>                            No, unfortunately, as far as I can remember,
>                     we do not have
>                            that. There
>                            are another hooks you may use but only in
>                     certain scenarios
>                            (#fuelNew,
>                            #fuelAfterMaterialization, global sends,
>                     etc). But everything is
>                            listed
>                            in
>                     http://rmod.inria.fr/web/software/Fuel/Version1.9/Documentation/CustomizingGraph
>                               so if you didn't find anything of help in
>                     there there are
>                     chances
>                            there isn't anything.
>
>                            Cheers,
>
>                            On Tue, Oct 13, 2015 at 5:30 PM, Robert Withers
>                            <[hidden email]
>                     <mailto:[hidden email]>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>> wrote:
>
>                                 Yes, I meant dynamic substitution on
>                     materialization, to
>                            use the
>                                 correct terminology.
>
>                                 thanks,
>                                 Robert
>
>
>                                 On 10/13/2015 11:40 AM, Max Leske wrote:
>
>
>                                         On 13 Oct 2015, at 17:16, Robert
>                     Withers
>                                         <[hidden email]
>                     <mailto:[hidden email]>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>> wrote:
>
>                                         Every extra source helps, thank
>                     you. I see how to do
>                                         non-stream substitutions on
>                     materializations, but the
>                                         documentation did not indicate a
>                     way to do non-stream
>                                         substitutions on serialization.
>                     Is it possible?
>
>
>                                     I don’t understand what you mean by
>                     “non-stream”. Could
>                            you give
>                                     an example?
>
>
>                                         thanks,
>                                         Robert
>
>                                         On 10/13/2015 09:00 AM, Mariano
>                     Martinez Peck wrote:
>
>                                             Hi Robert,
>
>                                             As for the documentation,
>                     you have LOTS of
>                            tests, you
>                                             have the chapter
>                                             Torsten pasted, you have
>                     this documentation:
>                     http://rmod.inria.fr/web/software/Fuel
>
>                                             But also, as for internals,
>                     there is a journal
>                            paper we
>                                             wrote:
>                     http://rmod.lille.inria.fr/archives/papers/Dias12a-SPE-Fuel.pdf
>
>                                             Let us know how it goes,
>
>
>                                             On Tue, Oct 13, 2015 at 6:00
>                     AM, Torsten Bergmann
>                                             <[hidden email]
>                     <mailto:[hidden email]> <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                     <mailto:[hidden email] <mailto:[hidden email]>>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]> <mailto:[hidden email]
>                     <mailto:[hidden email]>>>
>                                             <mailto:[hidden email]
>                     <mailto:[hidden email]> <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]> <mailto:[hidden email]
>                     <mailto:[hidden email]>>>>> wrote:
>
>                                                  Hi Robert,
>
>                                                  Also checkout the
>                     chapter on Fuel in Pharo
>                                             Enterprise book:
>
>                     https://ci.inria.fr/pharo-contribution/view/Books/job/EnterprisePharoBook/lastSuccessfulBuild/artifact/book-result/EnterprisePharo-A4.pdf
>
>                                                  Bye
>                                                  Torsten
>
>                         Gesendet: Dienstag, 13. Oktober 2015 um
>
>                            09:44 Uhr
>
>                         Von: "Robert Withers"
>
>                                             <[hidden email]
>                     <mailto:[hidden email]>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>
>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>>>
>
>                         An: [hidden email]
>                         <mailto:[hidden email]>
>                         <mailto:[hidden email]
>                         <mailto:[hidden email]>>
>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>>
>
>                         Betreff: Re: [Pharo-dev] binary
>
>                            serialization
>
>
>                         Yes, I have to do object substitutions.
>
>                            Thanks
>                                             for the link!
>
>
>                         thanks,
>                         Robert
>
>                         On 10/13/2015 03:43 AM, Max Leske wrote:
>
>
>                                 On 13 Oct 2015, at 09:40, Robert Withers
>
>
>                     <[hidden email]
>                     <mailto:[hidden email]>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>>> wrote:
>
>
>                                 Sven and Torsten, that's a binary
>
>                                             serialization library! It
>                                                  will take time to learn
>                     it and how to use
>                            mappers.
>
>
>                                 What is the format; is it language
>
>                            neutral?
>
>
>                             For quick serialization you don’t
>
>                            need to do
>                                             anything. It works
>                                                  for (almost) all
>                     objects. Only if you
>                     want to
>                                             exclude things or
>                                                  treat some objects in a
>                     special way, you
>                            will need
>                                             to do some stuff.
>
>
>                             Documentation:
>
>                     http://rmod.inria.fr/web/software/Fuel.
>
>
>
>
>                                 thanks,
>                                 Robert
>
>                                 On 10/13/2015 01:21 AM, Sven Van
>
>                            Caekenberghe
>                                             wrote:
>
>                                     Yes, it is called FUEL and it is a
>
>                            standard
>                                             part of the
>                                                  image. See FLSerializer
>                     and FLMaterializer.
>
>
>                                         On 13 Oct 2015, at 06:59, Robert
>
>                            Withers
>
>                     <[hidden email]
>                     <mailto:[hidden email]>
>                     <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>
>
>                       <mailto:[hidden email]
>                     <mailto:[hidden email]>
>                            <mailto:[hidden email]
>                     <mailto:[hidden email]>>>>> wrote:
>
>
>                                         Does Pharo have stream classes to
>
>                            binary
>                                             de/serialize an
>                                                  object, such that the
>                     protocol accepts an
>                            object as
>                                             an argument and
>                                                  converts it to a byteArray?
>
>
>                                         --
>                                         thanks,
>                                         Robert
>
>
>
>
>
>
>
>
>
>
>
>
>                                             --
>                                             Mariano
>                     http://marianopeck.wordpress.com
>
>
>
>
>
>
>
>
>                            --
>                            Mariano
>                     http://marianopeck.wordpress.com
>
>
>
>
>
>                     --
>                     Mariano
>                     http://marianopeck.wordpress.com
>
>                 <Exporting Vat.jpg><Importing Vat.jpg>
>
>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com

>
>
> On Wed, Oct 14, 2015 at 12:09 PM, Robert Withers
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>
>     On 10/14/2015 11:01 AM, Mariano Martinez Peck wrote:
>
>         Robert,
>
>         As far as I can remember, the problem of substitutions at
>         materialziation time was because... as you may have read, Fuel first
>         serializes "objects" and then the references. At materialization, it
>         first creates the objects and then, on a second step, sets the
>         references betwen objects. So the problem was WHERE to place the
>         hook,
>         at which point in time. If we do it just after objects were created,
>         then the substitution code would NOT have access to any of the
>         instVars
>         of that object. If we do it AFTER objects creation and after objects
>         references, then there is no easy way to "substitute" which doesn't
>         involve #become: (because the graph is already constructed) And
>         you know
>         that #become: is still terrible slow because it scans full
>         memory. That
>         will change in Spur.
>
>
>     The trick I learned from Gemstone is to use forwarding proxies,
>
>
> Well, Spur will/does something similar called lazy become. Basically it
> lets a forwarding pointer object and then takes advantage of next GC
> pass or whatever in order to resolve such proxy in  lazy manner.
>
>     looking up into the FLobjectId dictionary (decoder>>objects?) when
>     stitching the references. When you copy the proxies on substitution,
>     it stitches normally at reference time.
>
>
> Yes, that could work. The problem is if you need instVars of the object
> you want to substitute. Imagine you have a class called Client and
> instVar 'age'. And you want to substitute Client with instances of
> OldClient if 'age' is > 10.  At that point in time, the reference to the
> instVar 10 has not yet been filled. Yet, you need to replace the object
> at graph construction time.
>

>
>
>         As for Marea and Ghost,
>
>         Ghost proxies paper: https://hal.inria.fr/hal-01081236/document
>         Current Ghost repo: http://smalltalkhub.com/#!/~CAR/Ghost
>
>         Marea paper: http://www.jot.fm/issues/issue_2013_01/article2.pdf
>         Current repo: http://ss3.gemstone.com/ss/Marea.html
>
>         And finally, my PhD thesis:
>         https://tel.archives-ouvertes.fr/tel-00764991/document
>
>
>     Nicely done, sir. I'll check them out, thankyou.
>
>
> Thanks, feel free to ask questions.
>
>         In Marea I needed custom clusters for my proxies because the
>         serializer
>         itself sends messages to the objects being serialized. My
>         proxies would
>         bring back graphs from a secondary memory. So if I was serializing a
>         graph that had proxies already, I didn't want that. So I hooked my
>         custom cluster for proxies that send only a few special messages
>         to the
>         proxy that these understand and answer rather than intercept those
>         messages.
>
>         As for how to extend Fuel for this, I recommend to check the code of
>         Marea. See categories  'Marea-Serializers'  and 'Marea-Proxies'.
>         I have a Marea one click here:
>         https://www.dropbox.com/sh/xp8jzyypmz0898j/AACRdHno6V7UfhaJ1ofTPPXva?dl=0
>
>         Cheers,
>
>
>     thanks so much ^^
>     Robert
>
>
>
>         On Wed, Oct 14, 2015 at 11:40 AM, Robert Withers
>         <[hidden email] <mailto:[hidden email]>
>         <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>
>              Good morning, Max. Thank you for the example. I got a little
>              confused, between migrations and substitutions. My issue
>         with no-arg
>              blocks, I believe, is the inability to access my scope
>         object to
>              maintain the object tables.
>
>              I'm attempting to write my own Materializer, Decoder and
>              Materialization. At the moment, I'm just going to walk the
>         graph,
>              testing and do #becomes:. See how well that works when I
>         get a test.
>
>              It's really good to know about that other list.
>
>              thanks so much ^^
>              Robert
>
>              On 10/14/2015 04:15 AM, Max Leske wrote:
>
>                  BTW, there is a dedicated Fuel mailing list:
>         [hidden email]
>         <mailto:[hidden email]>
>                  <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                  <mailto:[hidden email]
>         <mailto:[hidden email]>
>                  <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                  Max
>
>
>                      On 14 Oct 2015, at 09:45, Max Leske
>         <[hidden email] <mailto:[hidden email]>
>                      <mailto:[hidden email] <mailto:[hidden email]>>
>                      <mailto:[hidden email]
>         <mailto:[hidden email]> <mailto:[hidden email]
>         <mailto:[hidden email]>>>> wrote:
>
>
>                          On 14 Oct 2015, at 04:39, Robert Withers
>                          <[hidden email]
>         <mailto:[hidden email]>
>                          <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                          <mailto:[hidden email]
>         <mailto:[hidden email]>
>                          <mailto:[hidden email]
>         <mailto:[hidden email]>>>> wrote:
>
>
>                          On 10/13/2015 09:43 PM, Mariano Martinez Peck
>         wrote:
>
>
>
>                              On Tue, Oct 13, 2015 at 10:33 PM, Robert
>         Withers
>                              <[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>> wrote:
>
>                                 Hi Mariano,
>
>                                 This presents me with a big challenge,
>         then. I
>                              read the docs and
>                                 explored the code and the only other
>         aspect not
>                              mentioned, beyond
>                                 instance creation (#fuelNew #fuelNew:) and
>                              postMaterialization
>                                 (#fuelAfterMaterialization), is migrations.
>                              However, migration only
>                                 allows for instanceVar mappings, no code
>         blocks.
>
>
>                              What do you mean that migrations only
>         allows instVar
>                              mappings , and no
>                              code blocks? I mean, what do you mean by
>         code blocks?
>
>
>                      Sounds to me like this (see FuelOutStackDebuAction):
>
>                      serializeTestFailureContext: aContext toFileNamed:
>         aFilename
>                      | serializer |
>
>                      serializer := FLSerializer newDefault.
>                      self encodeDebugInformationOn: serializer.
>                      serializer addPostMaterializationAction: [
>         :materialization |
>                      Smalltalk tools debugger
>                      openOn: Processor activeProcess
>                      context: materialization root
>                      label: 'External stack'
>                      contents: nil
>                      fullView: false ].
>
>                      serializer
>                      " use the sender context, generally the current
>         context is not
>                      interesting"
>                      serialize: aContext
>                      toFileNamed: aFilename
>
>                      This stores a block in the serialization which is
>         evaluated
>                      after
>                      materialization. The only requirement is that it’s
>         a clean
>                      block (no
>                      closure!).
>
>                              We also support class renames. This is here:
>         http://rmod.inria.fr/web/software/Fuel/Version1.9/Documentation/Migration?_s=Pvs4DYqPRjfEy8sg&_k=X6ltJu7FDSxumm4X&_n&22
>
>                              Which kind of migration example you have in
>         mind
>                              that would not be
>                              supported? An example would help.
>
>
>                          Well, my pics will demonstrate. I am interested
>         in doing
>                          more than
>                          mappping ivars or a class rename. I want to do
>         a total
>                          substitution,
>                          then a further substitution on the receiving,
>         import side:
>
>                          Vat1: anObject (Class A) ---> On wire: desc
>         (Descriptor)
>                          ---> Vat2:
>                          aProxy (Class FarERef)
>
>                          A desc is substituted for a PassByProxy object,
>         then a
>                          FarERef is
>                          substituted for the desc.
>
>
>                                 #fuelAccept: is a serialization side method.
>
>                                 If Fuel supports substitution on
>         serialization, I
>                              don't understand
>                                 why no substitution support on
>         materialization.
>
>
>                              There was a reason, which I cannot remember
>                              completely. Maybe Martin or
>                              Max can remember.
>
>
>                          It seems your focus was pickling to disk then
>         back. My
>                          focus is
>                          distributed proxy graphs, which has different
>         use cases.
>
>
>
>                                 I am definitely going to use the
>         world-class Fuel
>                              binary
>                                 serialization system. However, I find myself
>                              needing to extend Fuel
>                                 to support substitution on materialization.
>                              Perhaps the solution is
>                                 a custom decoder.
>
>
>                              I have made custom clusters for example for
>         my Ghost
>                              proxies of Marea
>                              system. It was a perfect example of how I could
>                              extent Fuel besides the
>                              common hooks. Fuel provides many places for
>                              extending , like clusters,
>                              analyzer,  etc
>
>
>                          Right on, exactly! Could you tell me more about
>         your
>                          Ghost proxies
>                          and Marea, please? As well, could you mention
>         how you
>                          select a custom
>                          cluster on the serialization side?
>
>
>                          thanks so much ^^
>                          Robert
>
>
>
>                                 No, a bit more. It looks like I need a new
>                                 FLSubstitutePointerObjectCluster, write
>         them on
>                              serialization with
>                                 the substitute, then do unsubstitution on
>                              materialization, since the
>                                 cluster controls materialization and not
>         the decoder.
>
>                                 Does this approach seem sound to you,
>         from a you
>                              know architecture
>                                 and design approach?
>
>
>                              There was an issue. Hope other can
>         remember. If not,
>                              I will try to
>                              explan what I remember tomorrow.
>
>
>                                 thanks so much ^^
>                                 Robert
>
>                                 On 10/13/2015 04:49 PM, Mariano Martinez
>         Peck wrote:
>
>                                     No, unfortunately, as far as I can
>         remember,
>                              we do not have
>                                     that. There
>                                     are another hooks you may use but
>         only in
>                              certain scenarios
>                                     (#fuelNew,
>                                     #fuelAfterMaterialization, global sends,
>                              etc). But everything is
>                                     listed
>                                     in
>         http://rmod.inria.fr/web/software/Fuel/Version1.9/Documentation/CustomizingGraph
>                                        so if you didn't find anything of
>         help in
>                              there there are
>                              chances
>                                     there isn't anything.
>
>                                     Cheers,
>
>                                     On Tue, Oct 13, 2015 at 5:30 PM,
>         Robert Withers
>                                     <[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>> wrote:
>
>                                          Yes, I meant dynamic
>         substitution on
>                              materialization, to
>                                     use the
>                                          correct terminology.
>
>                                          thanks,
>                                          Robert
>
>
>                                          On 10/13/2015 11:40 AM, Max
>         Leske wrote:
>
>
>                                                  On 13 Oct 2015, at
>         17:16, Robert
>                              Withers
>
>         <[hidden email] <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>> wrote:
>
>                                                  Every extra source
>         helps, thank
>                              you. I see how to do
>                                                  non-stream substitutions on
>                              materializations, but the
>                                                  documentation did not
>         indicate a
>                              way to do non-stream
>                                                  substitutions on
>         serialization.
>                              Is it possible?
>
>
>                                              I don’t understand what you
>         mean by
>                              “non-stream”. Could
>                                     you give
>                                              an example?
>
>
>                                                  thanks,
>                                                  Robert
>
>                                                  On 10/13/2015 09:00 AM,
>         Mariano
>                              Martinez Peck wrote:
>
>                                                      Hi Robert,
>
>                                                      As for the
>         documentation,
>                              you have LOTS of
>                                     tests, you
>                                                      have the chapter
>                                                      Torsten pasted, you
>         have
>                              this documentation:
>         http://rmod.inria.fr/web/software/Fuel
>
>                                                      But also, as for
>         internals,
>                              there is a journal
>                                     paper we
>                                                      wrote:
>         http://rmod.lille.inria.fr/archives/papers/Dias12a-SPE-Fuel.pdf
>
>                                                      Let us know how it
>         goes,
>
>
>                                                      On Tue, Oct 13,
>         2015 at 6:00
>                              AM, Torsten Bergmann
>                                                      <[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>> <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]> <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>> <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>
>
>         <mailto:[hidden email] <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>> <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>> <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>>> wrote:
>
>                                                           Hi Robert,
>
>                                                           Also checkout the
>                              chapter on Fuel in Pharo
>                                                      Enterprise book:
>
>         https://ci.inria.fr/pharo-contribution/view/Books/job/EnterprisePharoBook/lastSuccessfulBuild/artifact/book-result/EnterprisePharo-A4.pdf
>
>                                                           Bye
>                                                           Torsten
>
>                                  Gesendet: Dienstag, 13. Oktober 2015 um
>
>                                     09:44 Uhr
>
>                                  Von: "Robert Withers"
>
>
>         <[hidden email] <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>
>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>>>
>
>                                  An: [hidden email]
>         <mailto:[hidden email]>
>                                  <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                  <mailto:[hidden email]
>         <mailto:[hidden email]>
>                                  <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>>
>
>                                  Betreff: Re: [Pharo-dev] binary
>
>                                     serialization
>
>
>                                  Yes, I have to do object substitutions.
>
>                                     Thanks
>                                                      for the link!
>
>
>                                  thanks,
>                                  Robert
>
>                                  On 10/13/2015 03:43 AM, Max Leske wrote:
>
>
>                                          On 13 Oct 2015, at 09:40,
>         Robert Withers
>
>
>                              <[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>>> wrote:
>
>
>                                          Sven and Torsten, that's a binary
>
>                                                      serialization
>         library! It
>                                                           will take time
>         to learn
>                              it and how to use
>                                     mappers.
>
>
>                                          What is the format; is it language
>
>                                     neutral?
>
>
>                                      For quick serialization you don’t
>
>                                     need to do
>                                                      anything. It works
>                                                           for (almost) all
>                              objects. Only if you
>                              want to
>                                                      exclude things or
>                                                           treat some
>         objects in a
>                              special way, you
>                                     will need
>                                                      to do some stuff.
>
>
>                                      Documentation:
>
>         http://rmod.inria.fr/web/software/Fuel.
>
>
>
>
>                                          thanks,
>                                          Robert
>
>                                          On 10/13/2015 01:21 AM, Sven Van
>
>                                     Caekenberghe
>                                                      wrote:
>
>                                              Yes, it is called FUEL and
>         it is a
>
>                                     standard
>                                                      part of the
>                                                           image. See
>         FLSerializer
>                              and FLMaterializer.
>
>
>                                                  On 13 Oct 2015, at
>         06:59, Robert
>
>                                     Withers
>
>                              <[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>
>
>                                <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>
>                                     <mailto:[hidden email]
>         <mailto:[hidden email]>
>                              <mailto:[hidden email]
>         <mailto:[hidden email]>>>>>> wrote:
>
>
>                                                  Does Pharo have stream
>         classes to
>
>                                     binary
>                                                      de/serialize an
>                                                           object, such
>         that the
>                              protocol accepts an
>                                     object as
>                                                      an argument and
>                                                           converts it to
>         a byteArray?
>
>
>                                                  --
>                                                  thanks,
>                                                  Robert
>
>
>
>
>
>
>
>
>
>
>
>
>                                                      --
>                                                      Mariano
>         http://marianopeck.wordpress.com
>
>
>
>
>
>
>
>
>                                     --
>                                     Mariano
>         http://marianopeck.wordpress.com
>
>
>
>
>
>                              --
>                              Mariano
>         http://marianopeck.wordpress.com
>
>                          <Exporting Vat.jpg><Importing Vat.jpg>
>
>
>
>
>
>
>         --
>         Mariano
>         http://marianopeck.wordpress.com
>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com

> > Which kind of migration example you have in mind that would not be
> > supported? An example would help.
>
> Well, my pics will demonstrate. I am interested in doing more than
> mappping ivars or a class rename. I want to do a total substitution,
> then a further substitution on the receiving, import side:
>
> Vat1: anObject (Class A) ---> On wire: desc (Descriptor) ---> Vat2:
> aProxy (Class FarERef)
>
> A desc is substituted for a PassByProxy object, then a FarERef is
> substituted for the desc.
>
>>
>>     #fuelAccept: is a serialization side method.
>>
>>     If Fuel supports substitution on serialization, I don't understand
>>     why no substitution support on materialization.
>>
>>
>> There was a reason, which I cannot remember completely. Maybe Martin or
>> Max can remember.
>
> It seems your focus was pickling to disk then back. My focus is
> distributed proxy graphs, which has different use cases.

> >
>> > Which kind of migration example you have in mind that would not be
>> > supported? An example would help.
>>
>> Well, my pics will demonstrate. I am interested in doing more than
>> mappping ivars or a class rename. I want to do a total substitution,
>> then a further substitution on the receiving, import side:
>>
>> Vat1: anObject (Class A) ---> On wire: desc (Descriptor) ---> Vat2:
>> aProxy (Class FarERef)
>>
>> A desc is substituted for a PassByProxy object, then a FarERef is
>> substituted for the desc.
>>
>>>
>>>     #fuelAccept: is a serialization side method.
>>>
>>>     If Fuel supports substitution on serialization, I don't understand
>>>     why no substitution support on materialization.
>>>
>>>
>>> There was a reason, which I cannot remember completely. Maybe Martin or
>>> Max can remember.
>>
>> It seems your focus was pickling to disk then back. My focus is
>> distributed proxy graphs, which has different use cases.
> I'm interesting in knowing more :)
> Especially Vat and others.

>
>
> On 10/22/2015 06:19 AM, stepharo wrote:
>> >
>>> > Which kind of migration example you have in mind that would not be
>>> > supported? An example would help.
>>>
>>> Well, my pics will demonstrate. I am interested in doing more than
>>> mappping ivars or a class rename. I want to do a total substitution,
>>> then a further substitution on the receiving, import side:
>>>
>>> Vat1: anObject (Class A) ---> On wire: desc (Descriptor) ---> Vat2:
>>> aProxy (Class FarERef)
>>>
>>> A desc is substituted for a PassByProxy object, then a FarERef is
>>> substituted for the desc.
>>>
>>>>
>>>>     #fuelAccept: is a serialization side method.
>>>>
>>>>     If Fuel supports substitution on serialization, I don't understand
>>>>     why no substitution support on materialization.
>>>>
>>>>
>>>> There was a reason, which I cannot remember completely. Maybe
>>>> Martin or
>>>> Max can remember.
>>>
>>> It seems your focus was pickling to disk then back. My focus is
>>> distributed proxy graphs, which has different use cases.
>> I'm interesting in knowing more :)
>> Especially Vat and others.
>
> Firstly, I should credit these ideas: I leaned of Elib 9 years ago,
> from http://erights.org.
>
> A vat is an event-loop. This ensures single-threading with only one
> thread executing a stack as it schedules to activity. A vat is
> conceptually a stack and a queue. This is an executing stack of
> immediate calls and a queue of eventual sends that will be scheduled
> into the stack as evaluates. When you have a farReference to an object
> in a different vat, then you need to serialize the graph of arguments
> sent to the remote object with an eventual send: to queue the send in
> the remote vat, pending evaluation as a stack.
>
> When a graph is serialized, any PassByProxy objects sent as an
> argument to a remote method evaluation must be substituted on
> serialization with a descriptor describing the export of that far
> reference. This descriptor substitution is not a Fuel migration
> between versions of one object class nor is it a Fuel substitution,
> which replaces a subset of ivars, it is a true substitution, a
> descriptor of class FarDescriptor replaced for an arbitrary
> PasssByProxy object...let's say Customer <name, age, lastFourSSN>. So
> aCustomer <name: "Robert", age: 45, lastFourSSN: 9876> will get
> substituted with aFarReferenceDesc <wireId: 1, wireCount: 1>. On the
> receiving side, who sees this FerReference, if gets substituted again
> for a FarRef.
>
> aCustomer ---> farDesc ---> farRef
>
> Does this help to answer your questions? Fire away with more questions
> and I will try to answer. Reading about Elib on the erights site is
> another great source of information, sicne these guys came up with it.
>
> My thoughts in the VM are to add a pool to the stack and the queue of
> an event-loop.  For processes/continuations what are waiting for a
> signal, they are not scheduled into the queue. I believe that this
> model is a good one for a single-image-threaded execution machinery: a
> scheduling event-loop. This is pure intuition.
>
> Best,
> Robert
>
>> Stef
>>
>>
>>
>
>
>