[ANN] Ephemeron Support is Ready

Hi Eliot,

Pharo does AFAIK the same with the source files when you're navigating source code.

Now, I remember that while adapting the finalization scheme that you sent me to Pharo (because there are indeed subtle differences), I noticed that there was a missing loop.

[looking for the code... found!]

finalizationProcess
    "The finalization process arranges to send mourn to each element of the VM's finalization queue,
     which is accessed via primitiveFetchMourner.  The VM signals FinalizationSemaphore whenever
     the queue is non-empty.  This process loops, waiting on the semaphore, fetches the first element
     of the queue and then spawns a process at a higher priority to acually send the mourn messages.
     If an error occurs in the higher priority mourn loop process then this process will simply spawn
     another process, hence ensuring that errors in finalization methods don't break finalization.
 
     In addition this process also runs the old finalization scheme, supporting clients of the older,
     WeakRegistry based scheme.  Hopefully this will go away when all cleints have moved over."
    | throttle firstMourner |
    throttle := Semaphore new.
    [FinalizationSemaphore wait; initSignals.
     "Support the old scheme until things have changed over..."
     self doOldFinalization.
     [firstMourner := self primitiveFetchMourner.
      firstMourner notNil] whileTrue:
        [[throttle signal.
          self mournLoopWith: firstMourner] forkAt: Processor activePriority + 1.
         throttle wait]]

At first I was using that code that you sent me and I noticed that the finalization process in there is a loop that is never evaluated! So I updated it to the following using a [true] whileTrue:


finalizationProcess
    "The finalization process arranges to send mourn to each element of the VM's finalization queue,
     which is accessed via primitiveFetchMourner.  The VM signals FinalizationSemaphore whenever
     the queue is non-empty.  This process loops, waiting on the semaphore, fetches the first element
     of the queue and then spawns a process at a higher priority to acually send the mourn messages.
     If an error occurs in the higher priority mourn loop process then this process will simply spawn
     another process, hence ensuring that errors in finalization methods don't break finalization.

     In addition this process also runs the old finalization scheme, supporting clients of the older,
     WeakRegistry based scheme.  Hopefully this will go away when all cleints have moved over."
    | throttle firstMourner |
    throttle := Semaphore new.
    [true] whileTrue: [FinalizationSemaphore wait; initSignals.
     "Support the old scheme until things have changed over..."
     self doOldFinalization.
     [firstMourner := self primitiveFetchMourner.
      firstMourner notNil] whileTrue:
        [[throttle signal.
          self mournLoopWith: firstMourner] forkAt: Processor activePriority + 1.
         throttle wait]]

Maybe that's the reason of weak arrays not being finalized in squeak?

Guille

-------- Original Message --------

Hi Guille,

   good news!  But I'm seeing something wrong with finalisation of weak arrays using the new scheme.  In Squeak method source access is done by default by opening a new read-only file for each method's source read (crazy, but that's not the issue).  So by default something that accesses lots of source ends up running out of file descriptors, which causes primOpen:writable: to fail.  The surrounding code then uses retryWithGC:until:forFileNamed: to do a GC to try and reclaim non-longer referenced files, close file descriptors and continue:

        StandardFileStream retryWithGC:[self primOpen: f writable: writeMode] 

until:[:id| id notNil] 

forFileNamed: fileName.

But in my tests I'm not seeing any files reclaimed.  I wonder whether the new finalisation code is failing to finalise weak arrays properly.  I wonder whether your weak tests work properly with the new scheme or not.

Anyway, I think I can reproduce the pathology in the simulator if I modify it to implement a small limit on the number of file descriptors.  I'm going to try that to shed light on the problem.  I can't easily debug in a running image because...I run out of file descriptors ;-)

On Tue, Jun 7, 2016 at 2:32 AM, Guille Polito <[hidden email]> wrote:

Hi All,



Since this morning, in Pharo #60065, Ephemeron support is in the
image. Most of the changes are infrastructural, so far transparent
for the users. It is important to notice that even while the support
is there, it is not enabled by default. Also, this required changes
in the virtual machine that are not yet distributed everywhere. For
the ones that would like more detail, I invite you to read the
following :)



* On the infrastructure side

  - There is support to create Ephemeric classes and load them from
monticello

  - There is a new finalization mechanism (by default disabled) that
will process Ephemerons using a finalization queue. This will avoid
scanning collections in look for weak objects to finalize ,as it
happens now with the WeakDependent mechanism in WeakArray.

  - System-Finalization features two new classes *Ephemeron* and
*EphemeronRegistry*. For the ones that want more details on
Ephemerons, you can read the associated paper [1], or the class
comment of Ephemeron:

I represent ephemeric key-value objects. Ephemerons are
key-value objects (subclasses of Association) with special
semantics during garbage collection.  My special behavior can
resumed as follows:



- The garbage collection will iterate my instances only if the key
is not referenced strongly by another object.

- Then, if no strong references to the key are found, then the
values of this ephemeron are hold weakly.

- Otherwise, the values are hold strongly.



In this implementation, an Ephemeron can hold more than one value,
which are all treated in the same manner. This ephemeron instance
knows its container, which allows the ephemeron to remove itself
from a container (such as a Dictionary) upon finalization.



!! Example usages



In general terms, do not use myself directly. Use instead an
Ephemeric container like EphemeronRegistry. An Ephemeron registry
will guarantee the collection of keys and values of the object
inside the Ephemeron.



Otherwise, if you want to use it, you can create an Ephemeron as
any association:



ephemeron := Ephemeron key: aKey value: aValue.

ephemeron container: aContainer.



!! Ephemeron Finalization



When an ephemeron's key is hold strongly just by the ephemeron
itself, the Ephemeron will be mourned (finalized). That means that
the VM will:

- put the Ephemeron in the mourning queue waiting for the image to
take care of mourning

- make the Ephemeron non ephemeric. That is, the ephemeron
instance cannot be reused.



On the image side, the finalization process will send the message
#mourn to an Ephemeron.  #mourn will #finalize the Ephemeron's
key, and remove the Ephemeron from it's container to allow its
collection during a subsequent garbage collection.



!! More Documentation



You can read the associated paper to understand better the
semantics of ephemerons:



[1]Ephemerons: A New Finalization Mechanism. Barry Hayes. OOPSLA
'97

  - WARNING: to be able to use ephemerons, you need to use the
*latestVm* that has several fixes for making ephemerons work, and
you need to enable ephemerons on the image side by evaluating:



            Smalltalk supportsQueueingFinalization: true.



  - With latest vm and ephemerons enabled, tests should be green,
otherwise they are skipped







* From the user point of view:



  - The Weak registries were not yet migrated to the new
finalization mechanism.

  - We expect nothing will change from the user point of view. Just
less memory leaks.

 

* Next steps (in order)

  1) Bless the latest vm as stable

  2) Enable queueing finalization by default

  3) Replace Weak Registry by Ephemeron Registry.





Informed by: Guille

--

_,,,^..^,,,_

best, Eliot

finalizationProcess
    "The finalization process arranges to send mourn to each element of the VM's finalization queue,
     which is accessed via primitiveFetchMourner.  The VM signals FinalizationSemaphore whenever
     the queue is non-empty.  This process loops, waiting on the semaphore, fetches the first element
     of the queue and then spawns a process at a higher priority to acually send the mourn messages.
     If an error occurs in the higher priority mourn loop process then this process will simply spawn
     another process, hence ensuring that errors in finalization methods don't break finalization.
 
     In addition this process also runs the old finalization scheme, supporting clients of the older,
     WeakRegistry based scheme.  Hopefully this will go away when all cleints have moved over."
    | throttle firstMourner |
    throttle := Semaphore new.
    [FinalizationSemaphore wait; initSignals.
     "Support the old scheme until things have changed over..."
     self doOldFinalization.
     [firstMourner := self primitiveFetchMourner.
      firstMourner notNil] whileTrue:
        [[throttle signal.
          self mournLoopWith: firstMourner] forkAt: Processor activePriority + 1.
         throttle wait]]

Doh!  So I wrote a naked block, not a loop !! So it did nothing.  Thank you!!  I think the compiler should warn about naked blocks.