Class var and class inst var caches in libraries

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Class var and class inst var caches in libraries

GLASS mailing list
Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
Yes, lazy initialization for shared variables in a multi-gem system is
not a good idea.

Since it sounds like you do not need the values persisted and shared
between gems, you should be using the class SessionTemps. Here's an
example from Collection class>>randomForPicking:

randomForPicking
   | random |
   random := SessionTemps current
     at: #'COLLECTION_RANDOM_FOR_PICKING'
     otherwise: nil.
   random == nil
     ifTrue: [ random := Random new.
       SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
random ].
   ^ random

SessionTemps is a subclass of SymbolDictionary and can be used for
storing session-specific globals.

If you have the need to sharing persistent values then an explicit
initialization in the class initialization method is the right answer.
The initialize method gets run during install and at least the
initialization is safe from commit conflicts.

Dale


On 10/04/2016 11:56 AM, monty via Glass wrote:
> Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
My mail client wrongly sent this response to Dale, not to the list, so I'm forwarding it

> Sent: Friday, October 07, 2016 at 3:05 PM
> From: monty <[hidden email]>
> To: [hidden email]
> Subject: Re: [Glass] Class var and class inst var caches in libraries
>
> Thanks, I think this will do, but I have one more question. If a class method lazily initializes a SessionTemp to the same value (using at:ifAbsentPut: for example), but that method is executed at roughly the same time in different processes (due to fork), is there a possibility that "SessionTemps current" itself could get corrupted? For example, that it could report an inaccurate #size, or that its internal state in some other way could get corrupted due to the concurrent update?
>
> > Sent: Tuesday, October 04, 2016 at 3:21 PM
> > From: "Dale Henrichs via Glass" <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Glass] Class var and class inst var caches in libraries
> >
> > Yes, lazy initialization for shared variables in a multi-gem system is
> > not a good idea.
> >
> > Since it sounds like you do not need the values persisted and shared
> > between gems, you should be using the class SessionTemps. Here's an
> > example from Collection class>>randomForPicking:
> >
> > randomForPicking
> >    | random |
> >    random := SessionTemps current
> >      at: #'COLLECTION_RANDOM_FOR_PICKING'
> >      otherwise: nil.
> >    random == nil
> >      ifTrue: [ random := Random new.
> >        SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
> > random ].
> >    ^ random
> >
> > SessionTemps is a subclass of SymbolDictionary and can be used for
> > storing session-specific globals.
> >
> > If you have the need to sharing persistent values then an explicit
> > initialization in the class initialization method is the right answer.
> > The initialize method gets run during install and at least the
> > initialization is safe from commit conflicts.
> >
> > Dale
> >
> >
> > On 10/04/2016 11:56 AM, monty via Glass wrote:
> > > Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
> > > _______________________________________________
> > > Glass mailing list
> > > [hidden email]
> > > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
> > _______________________________________________
> > Glass mailing list
> > [hidden email]
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
Monty,

If there is the chance for concurrent updates by different GsProcesses
running in the same vm, then the access should be protected by a
Semaphore ...

Instances of Semaphore cannot be persisted, but you can persist a
reference to a TransientSemaphore instance (created during class
initialization). The field values in TransientSemaphore are not
persisted while the TransientSemaphore itself _is_ persisted ...

Dale


On 10/7/16 12:12 PM, monty via Glass wrote:

> My mail client wrongly sent this response to Dale, not to the list, so I'm forwarding it
>
>> Sent: Friday, October 07, 2016 at 3:05 PM
>> From: monty <[hidden email]>
>> To: [hidden email]
>> Subject: Re: [Glass] Class var and class inst var caches in libraries
>>
>> Thanks, I think this will do, but I have one more question. If a class method lazily initializes a SessionTemp to the same value (using at:ifAbsentPut: for example), but that method is executed at roughly the same time in different processes (due to fork), is there a possibility that "SessionTemps current" itself could get corrupted? For example, that it could report an inaccurate #size, or that its internal state in some other way could get corrupted due to the concurrent update?
>>
>>> Sent: Tuesday, October 04, 2016 at 3:21 PM
>>> From: "Dale Henrichs via Glass" <[hidden email]>
>>> To: [hidden email]
>>> Subject: Re: [Glass] Class var and class inst var caches in libraries
>>>
>>> Yes, lazy initialization for shared variables in a multi-gem system is
>>> not a good idea.
>>>
>>> Since it sounds like you do not need the values persisted and shared
>>> between gems, you should be using the class SessionTemps. Here's an
>>> example from Collection class>>randomForPicking:
>>>
>>> randomForPicking
>>>     | random |
>>>     random := SessionTemps current
>>>       at: #'COLLECTION_RANDOM_FOR_PICKING'
>>>       otherwise: nil.
>>>     random == nil
>>>       ifTrue: [ random := Random new.
>>>         SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
>>> random ].
>>>     ^ random
>>>
>>> SessionTemps is a subclass of SymbolDictionary and can be used for
>>> storing session-specific globals.
>>>
>>> If you have the need to sharing persistent values then an explicit
>>> initialization in the class initialization method is the right answer.
>>> The initialize method gets run during install and at least the
>>> initialization is safe from commit conflicts.
>>>
>>> Dale
>>>
>>>
>>> On 10/04/2016 11:56 AM, monty via Glass wrote:
>>>> Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
>>>> _______________________________________________
>>>> Glass mailing list
>>>> [hidden email]
>>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>> _______________________________________________
>>> Glass mailing list
>>> [hidden email]
>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
>>>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
In reply to this post by GLASS mailing list

With a full gs server (rather than gem-limited glass) it is uncommon that someone attempts forks within a gem rather than doing work in a separate gem. I had developed a framework for easily forking work in separate synchronous or asynchronous gems. That approach was safer and faster (due to custom gem configurations). You might consider licensing more glass gems rather than forging a less traveled path risking transaction view surprises.

What of your code is unavoidably sensitive to a momentary size inconsistency with session temps? Perhaps more than one might start a long lazy-initialization process? That could be handled with another session temp to hold a semaphore or recording an initialization state flag, but that also needs pre-initialization.

Pre-initialization of variables avoids the structure changes risks discussed by lazy-initialization. Similarly, it is a good idea to avoid lazy-initialization in persistent shared variables due to commit conflicts between competing gems. Forks within a gem are not common though, so don't be too surprised if you start encountering problems that few had seen even if using semaphores properly.

Paul Baumann

On Oct 7, 2016 3:13 PM, "monty via Glass" <[hidden email]> wrote:
My mail client wrongly sent this response to Dale, not to the list, so I'm forwarding it

> Sent: Friday, October 07, 2016 at 3:05 PM
> From: monty <[hidden email]>
> To: [hidden email]
> Subject: Re: [Glass] Class var and class inst var caches in libraries
>
> Thanks, I think this will do, but I have one more question. If a class method lazily initializes a SessionTemp to the same value (using at:ifAbsentPut: for example), but that method is executed at roughly the same time in different processes (due to fork), is there a possibility that "SessionTemps current" itself could get corrupted? For example, that it could report an inaccurate #size, or that its internal state in some other way could get corrupted due to the concurrent update?
>
> > Sent: Tuesday, October 04, 2016 at 3:21 PM
> > From: "Dale Henrichs via Glass" <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Glass] Class var and class inst var caches in libraries
> >
> > Yes, lazy initialization for shared variables in a multi-gem system is
> > not a good idea.
> >
> > Since it sounds like you do not need the values persisted and shared
> > between gems, you should be using the class SessionTemps. Here's an
> > example from Collection class>>randomForPicking:
> >
> > randomForPicking
> >    | random |
> >    random := SessionTemps current
> >      at: #'COLLECTION_RANDOM_FOR_PICKING'
> >      otherwise: nil.
> >    random == nil
> >      ifTrue: [ random := Random new.
> >        SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
> > random ].
> >    ^ random
> >
> > SessionTemps is a subclass of SymbolDictionary and can be used for
> > storing session-specific globals.
> >
> > If you have the need to sharing persistent values then an explicit
> > initialization in the class initialization method is the right answer.
> > The initialize method gets run during install and at least the
> > initialization is safe from commit conflicts.
> >
> > Dale
> >
> >
> > On 10/04/2016 11:56 AM, monty via Glass wrote:
> > > Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
> > > _______________________________________________
> > > Glass mailing list
> > > [hidden email]
> > > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
> > _______________________________________________
> > Glass mailing list
> > [hidden email]
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list

Paul makes a good point about using separate gems instead of forking green threads ... with green threads you have to be very careful that an abort or commit in one thread does not affect ongoing work in another thread or persist data prematurely.

Take a look at GsExternalSession for code that supports executing code in a separate session ... probably not as advanced as Paul's fraemwork, but it is a good starting point ...

See the ServiceVM project[1] for an example of a framework that dedicates a single vm for performing "background tasks".

Dale

[1] https://github.com/GsDevKit/ServiceVM

On 10/7/16 3:13 PM, Paul Baumann via Glass wrote:

With a full gs server (rather than gem-limited glass) it is uncommon that someone attempts forks within a gem rather than doing work in a separate gem. I had developed a framework for easily forking work in separate synchronous or asynchronous gems. That approach was safer and faster (due to custom gem configurations). You might consider licensing more glass gems rather than forging a less traveled path risking transaction view surprises.

What of your code is unavoidably sensitive to a momentary size inconsistency with session temps? Perhaps more than one might start a long lazy-initialization process? That could be handled with another session temp to hold a semaphore or recording an initialization state flag, but that also needs pre-initialization.

Pre-initialization of variables avoids the structure changes risks discussed by lazy-initialization. Similarly, it is a good idea to avoid lazy-initialization in persistent shared variables due to commit conflicts between competing gems. Forks within a gem are not common though, so don't be too surprised if you start encountering problems that few had seen even if using semaphores properly.

Paul Baumann

On Oct 7, 2016 3:13 PM, "monty via Glass" <[hidden email]> wrote:
My mail client wrongly sent this response to Dale, not to the list, so I'm forwarding it

> Sent: Friday, October 07, 2016 at 3:05 PM
> From: monty <[hidden email]>
> To: [hidden email]
> Subject: Re: [Glass] Class var and class inst var caches in libraries
>
> Thanks, I think this will do, but I have one more question. If a class method lazily initializes a SessionTemp to the same value (using at:ifAbsentPut: for example), but that method is executed at roughly the same time in different processes (due to fork), is there a possibility that "SessionTemps current" itself could get corrupted? For example, that it could report an inaccurate #size, or that its internal state in some other way could get corrupted due to the concurrent update?
>
> > Sent: Tuesday, October 04, 2016 at 3:21 PM
> > From: "Dale Henrichs via Glass" <[hidden email]>
> > To: [hidden email]
> > Subject: Re: [Glass] Class var and class inst var caches in libraries
> >
> > Yes, lazy initialization for shared variables in a multi-gem system is
> > not a good idea.
> >
> > Since it sounds like you do not need the values persisted and shared
> > between gems, you should be using the class SessionTemps. Here's an
> > example from Collection class>>randomForPicking:
> >
> > randomForPicking
> >    | random |
> >    random := SessionTemps current
> >      at: #'COLLECTION_RANDOM_FOR_PICKING'
> >      otherwise: nil.
> >    random == nil
> >      ifTrue: [ random := Random new.
> >        SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
> > random ].
> >    ^ random
> >
> > SessionTemps is a subclass of SymbolDictionary and can be used for
> > storing session-specific globals.
> >
> > If you have the need to sharing persistent values then an explicit
> > initialization in the class initialization method is the right answer.
> > The initialize method gets run during install and at least the
> > initialization is safe from commit conflicts.
> >
> > Dale
> >
> >
> > On 10/04/2016 11:56 AM, monty via Glass wrote:
> > > Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
> > > _______________________________________________
> > > Glass mailing list
> > > [hidden email]
> > > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
> > _______________________________________________
> > Glass mailing list
> > [hidden email]
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
> >
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
In reply to this post by GLASS mailing list
> Sent: Friday, October 07, 2016 at 5:40 PM
> From: "Dale Henrichs via Glass" <[hidden email]>
> To: [hidden email]
> Subject: Re: [Glass] Class var and class inst var caches in libraries
>
> Monty,
>
> If there is the chance for concurrent updates by different GsProcesses
> running in the same vm, then the access should be protected by a
> Semaphore ...

Isn't this always the case with lazily initialized SessionTemps? If a user forks off multiple processes that send the message triggering SessionTemp lazy initialization at the same time, unless SessionTemp>>#at:ifAbsentPut: is atomic or reentrant, the dictionary's internal state could be corrupted--even if the #at:ifAbsentPut: args are identical. So this could be a source of rare bugs (which is unacceptable to me), and it's not obvious how library authors can make their libraries both conflict-free and #fork-safe on GS.

I'm leaning towards using explicit initialization for all immutable values assigned to class and class isnt vars (which will persist), and then for values that mutate, to avoid locks, lazy initialization with SessionTemps but with access guarded by a global transient lock. Persisting immutable values explicitly initialized in a class #initialize method will unfortunately disallow class reinitialization with multiple Gems (would cause conflicts).

> Instances of Semaphore cannot be persisted, but you can persist a
> reference to a TransientSemaphore instance (created during class
> initialization). The field values in TransientSemaphore are not
> persisted while the TransientSemaphore itself _is_ persisted ...
>
> Dale

I already use TransientRecursionLock to lock non-persistent collections stored in class/class inst vars that are mutated after creation to make them #fork safe (Mutex is used on Pharo/Squeak to do the same).

>
> On 10/7/16 12:12 PM, monty via Glass wrote:
> > My mail client wrongly sent this response to Dale, not to the list, so I'm forwarding it
> >
> >> Sent: Friday, October 07, 2016 at 3:05 PM
> >> From: monty <[hidden email]>
> >> To: [hidden email]
> >> Subject: Re: [Glass] Class var and class inst var caches in libraries
> >>
> >> Thanks, I think this will do, but I have one more question. If a class method lazily initializes a SessionTemp to the same value (using at:ifAbsentPut: for example), but that method is executed at roughly the same time in different processes (due to fork), is there a possibility that "SessionTemps current" itself could get corrupted? For example, that it could report an inaccurate #size, or that its internal state in some other way could get corrupted due to the concurrent update?
> >>
> >>> Sent: Tuesday, October 04, 2016 at 3:21 PM
> >>> From: "Dale Henrichs via Glass" <[hidden email]>
> >>> To: [hidden email]
> >>> Subject: Re: [Glass] Class var and class inst var caches in libraries
> >>>
> >>> Yes, lazy initialization for shared variables in a multi-gem system is
> >>> not a good idea.
> >>>
> >>> Since it sounds like you do not need the values persisted and shared
> >>> between gems, you should be using the class SessionTemps. Here's an
> >>> example from Collection class>>randomForPicking:
> >>>
> >>> randomForPicking
> >>>     | random |
> >>>     random := SessionTemps current
> >>>       at: #'COLLECTION_RANDOM_FOR_PICKING'
> >>>       otherwise: nil.
> >>>     random == nil
> >>>       ifTrue: [ random := Random new.
> >>>         SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
> >>> random ].
> >>>     ^ random
> >>>
> >>> SessionTemps is a subclass of SymbolDictionary and can be used for
> >>> storing session-specific globals.
> >>>
> >>> If you have the need to sharing persistent values then an explicit
> >>> initialization in the class initialization method is the right answer.
> >>> The initialize method gets run during install and at least the
> >>> initialization is safe from commit conflicts.
> >>>
> >>> Dale
> >>>
> >>>
> >>> On 10/04/2016 11:56 AM, monty via Glass wrote:
> >>>> Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
> >>>> _______________________________________________
> >>>> Glass mailing list
> >>>> [hidden email]
> >>>> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>> _______________________________________________
> >>> Glass mailing list
> >>> [hidden email]
> >>> http://lists.gemtalksystems.com/mailman/listinfo/glass
> >>>
> > _______________________________________________
> > Glass mailing list
> > [hidden email]
> > http://lists.gemtalksystems.com/mailman/listinfo/glass
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
In reply to this post by GLASS mailing list
> Sent: Friday, October 07, 2016 at 6:13 PM
> From: "Paul Baumann" <[hidden email]>
> To: monty <[hidden email]>
> Cc: [hidden email]
> Subject: Re: [Glass] Class var and class inst var caches in libraries
> With a full gs server (rather than gem-limited glass) it is uncommon that someone attempts forks within a gem rather than doing work in a separate gem. I had developed a framework for easily forking work in separate synchronous or asynchronous gems. That approach was safer and faster (due to custom gem configurations). You might consider licensing more glass gems rather than forging a less traveled path risking transaction view surprises.

The question was in the context of libraries with GS ports and how to eliminate write conflicts and achieve #fork safely with class/class inst behavior. The concern is for the users of my libraries.

> What of your code is unavoidably sensitive to a momentary size inconsistency with session temps? Perhaps more than one might start a long lazy-initialization process? That could be handled with another session temp to hold a semaphore or recording an initialization state flag, but that also needs pre-initialization.

The #size issue was the first thing that came to mind, there could be others issues. And a Dictionary confused about its size could in theory cause issues by itself, like during resizing/rehashing, but without studying the implementation this is just speculation.

> Pre-initialization of variables avoids the structure changes risks discussed by lazy-initialization. Similarly, it is a good idea to avoid lazy-initialization in persistent shared variables due to commit conflicts between competing gems. Forks within a gem are not common though, so don't be too surprised if you start encountering problems that few had seen even if using semaphores properly.
> Paul Baumann
>
> On Oct 7, 2016 3:13 PM, "monty via Glass" <[hidden email][mailto:[hidden email]]> wrote:My mail client wrongly sent this response to Dale, not to the list, so I'm forwarding it
>
> > Sent: Friday, October 07, 2016 at 3:05 PM
> > From: monty <[hidden email][mailto:[hidden email]]>
> > To: [hidden email][mailto:[hidden email]]
> > Subject: Re: [Glass] Class var and class inst var caches in libraries
> >
> > Thanks, I think this will do, but I have one more question. If a class method lazily initializes a SessionTemp to the same value (using at:ifAbsentPut: for example), but that method is executed at roughly the same time in different processes (due to fork), is there a possibility that "SessionTemps current" itself could get corrupted? For example, that it could report an inaccurate #size, or that its internal state in some other way could get corrupted due to the concurrent update?
> >
> > > Sent: Tuesday, October 04, 2016 at 3:21 PM
> > > From: "Dale Henrichs via Glass" <[hidden email][mailto:[hidden email]]>
> > > To: [hidden email][mailto:[hidden email]]
> > > Subject: Re: [Glass] Class var and class inst var caches in libraries
> > >
> > > Yes, lazy initialization for shared variables in a multi-gem system is
> > > not a good idea.
> > >
> > > Since it sounds like you do not need the values persisted and shared
> > > between gems, you should be using the class SessionTemps. Here's an
> > > example from Collection class>>randomForPicking:
> > >
> > > randomForPicking
> > > | random |
> > > random := SessionTemps current
> > > at: #'COLLECTION_RANDOM_FOR_PICKING'
> > > otherwise: nil.
> > > random == nil
> > > ifTrue: [ random := Random new.
> > > SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
> > > random ].
> > > ^ random
> > >
> > > SessionTemps is a subclass of SymbolDictionary and can be used for
> > > storing session-specific globals.
> > >
> > > If you have the need to sharing persistent values then an explicit
> > > initialization in the class initialization method is the right answer.
> > > The initialize method gets run during install and at least the
> > > initialization is safe from commit conflicts.
> > >
> > > Dale
> > >
> > >
> > > On 10/04/2016 11:56 AM, monty via Glass wrote:
> > > > Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
> > > > _______________________________________________
> > > > Glass mailing list
> > > > [hidden email][mailto:[hidden email]]
> > > > http://lists.gemtalksystems.com/mailman/listinfo/glass[http://lists.gemtalksystems.com/mailman/listinfo/glass]
> > >
> > > _______________________________________________
> > > Glass mailing list
> > > [hidden email][mailto:[hidden email]]
> > > http://lists.gemtalksystems.com/mailman/listinfo/glass[http://lists.gemtalksystems.com/mailman/listinfo/glass]
> > >
> _______________________________________________
> Glass mailing list
> [hidden email][mailto:[hidden email]]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
In reply to this post by GLASS mailing list
None of my libs actually use #fork or equivalent BTW.

>
> Sent: Friday, October 07, 2016 at 6:13 PM
> From: "Paul Baumann" <[hidden email]>
> To: monty <[hidden email]>
> Cc: [hidden email]
> Subject: Re: [Glass] Class var and class inst var caches in libraries
> With a full gs server (rather than gem-limited glass) it is uncommon that someone attempts forks within a gem rather than doing work in a separate gem. I had developed a framework for easily forking work in separate synchronous or asynchronous gems. That approach was safer and faster (due to custom gem configurations). You might consider licensing more glass gems rather than forging a less traveled path risking transaction view surprises.
> What of your code is unavoidably sensitive to a momentary size inconsistency with session temps? Perhaps more than one might start a long lazy-initialization process? That could be handled with another session temp to hold a semaphore or recording an initialization state flag, but that also needs pre-initialization.
> Pre-initialization of variables avoids the structure changes risks discussed by lazy-initialization. Similarly, it is a good idea to avoid lazy-initialization in persistent shared variables due to commit conflicts between competing gems. Forks within a gem are not common though, so don't be too surprised if you start encountering problems that few had seen even if using semaphores properly.
> Paul Baumann
>
> On Oct 7, 2016 3:13 PM, "monty via Glass" <[hidden email][mailto:[hidden email]]> wrote:My mail client wrongly sent this response to Dale, not to the list, so I'm forwarding it
>
> > Sent: Friday, October 07, 2016 at 3:05 PM
> > From: monty <[hidden email][mailto:[hidden email]]>
> > To: [hidden email][mailto:[hidden email]]
> > Subject: Re: [Glass] Class var and class inst var caches in libraries
> >
> > Thanks, I think this will do, but I have one more question. If a class method lazily initializes a SessionTemp to the same value (using at:ifAbsentPut: for example), but that method is executed at roughly the same time in different processes (due to fork), is there a possibility that "SessionTemps current" itself could get corrupted? For example, that it could report an inaccurate #size, or that its internal state in some other way could get corrupted due to the concurrent update?
> >
> > > Sent: Tuesday, October 04, 2016 at 3:21 PM
> > > From: "Dale Henrichs via Glass" <[hidden email][mailto:[hidden email]]>
> > > To: [hidden email][mailto:[hidden email]]
> > > Subject: Re: [Glass] Class var and class inst var caches in libraries
> > >
> > > Yes, lazy initialization for shared variables in a multi-gem system is
> > > not a good idea.
> > >
> > > Since it sounds like you do not need the values persisted and shared
> > > between gems, you should be using the class SessionTemps. Here's an
> > > example from Collection class>>randomForPicking:
> > >
> > > randomForPicking
> > >    | random |
> > >    random := SessionTemps current
> > >      at: #'COLLECTION_RANDOM_FOR_PICKING'
> > >      otherwise: nil.
> > >    random == nil
> > >      ifTrue: [ random := Random new.
> > >        SessionTemps current at: #'COLLECTION_RANDOM_FOR_PICKING' put:
> > > random ].
> > >    ^ random
> > >
> > > SessionTemps is a subclass of SymbolDictionary and can be used for
> > > storing session-specific globals.
> > >
> > > If you have the need to sharing persistent values then an explicit
> > > initialization in the class initialization method is the right answer.
> > > The initialize method gets run during install and at least the
> > > initialization is safe from commit conflicts.
> > >
> > > Dale
> > >
> > >
> > > On 10/04/2016 11:56 AM, monty via Glass wrote:
> > > > Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
> > > > _______________________________________________
> > > > Glass mailing list
> > > > [hidden email][mailto:[hidden email]]
> > > > http://lists.gemtalksystems.com/mailman/listinfo/glass[http://lists.gemtalksystems.com/mailman/listinfo/glass]
> > >
> > > _______________________________________________
> > > Glass mailing list
> > > [hidden email][mailto:[hidden email]]
> > > http://lists.gemtalksystems.com/mailman/listinfo/glass[http://lists.gemtalksystems.com/mailman/listinfo/glass]
> > >
> _______________________________________________
> Glass mailing list
> [hidden email][mailto:[hidden email]]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
In reply to this post by GLASS mailing list

Use application design that provides division of responsibility. Avoid having more than one gem updating the same value holder. Gem-specific value holders can be used, referenced in gem-specific dictionary (or object).  Any other gems can query committed assignments of others to find consensus. Somewhat like an rc collection implementation, and similar to chain of responsibility design pattern.

One technique is to add an instruction object to an RcQueue that a single manager processes in a separate dedicated gem. Many gems give instructions, and one manager processes them without conflicts. Gem-to-gem signaling can be part of that design if delay loops are not desired or timely. Nested transactions might come in handy for this now that gs has them. The commits and aborts used for queued work may not be ideal for some applications.

There are many tricks and techniques, the most appropriate depends on your application needs. Review the RC collections to see if any help you. Your application code needs to have a strategy to avoid conflicts though. Locking can be avoided for most code. It seems odd that your code has multiple gems updating the same class variables.

Paul Baumann

On Oct 4, 2016 2:56 PM, "monty via Glass" <[hidden email]> wrote:
Lots of libraries use caches stored in class vars and class isnt vars that are either explicitly or lazily initialized. But with multiple Gems accessing the same repo, this can produce write-write conflicts. In fact, simply sending #initialize to the same class from different Gems can cause them. What are the best ways to implement these caches in GS? Write-locks seem like overkill, so I'm interested in simple ways of making them non-persistent.
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list
In reply to this post by GLASS mailing list
Thanks Dale and Paul. I've given up on making my libs #fork-safe on GS because after looking at the source for some of the GS(DevKit) methods I would use, I don't think they can guarantee it.

Instead I'm just focusing on preventing write conflicts. My solution now is forcing explicit initialization of normally lazy-initialized class/class inst vars in a Metacello postLoad on GS. Since the values are mostly immutable, they can be safely persisted, and since the mutable values are all instances of a custom dictionary-like cache class that uses a TransientRecursionLock to guard access to an internal collection wrapped in a TransientValue on GS, these instances can be safely persisted too. In my tests this eliminated all write conflicts, so XMLParser, XMLParserHTML, XMLParserStAX, XPath, and XMLWriter should all be conflict-free during normal use.
 
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: Class var and class inst var caches in libraries

GLASS mailing list

Collections that use value holders, like Dictionary having Association instances, only have write conflicts on growth or when a value is updated by two gems. A recursion lock on growth (and perhaps removals) can be enough if more than one session is not assigning the same keys. Pre-allocation (using #new:) may remove the need for locks because there would be no resize. A lock on the association can be useful to avoid duplication of effort between sessions that might precede commit failure. KeyValueCollection is object-allocation efficient (faster for some uses) but would have more conflicts on the structures holding values.

On Oct 9, 2016 8:40 PM, "monty via Glass" <[hidden email]> wrote:
Thanks Dale and Paul. I've given up on making my libs #fork-safe on GS because after looking at the source for some of the GS(DevKit) methods I would use, I don't think they can guarantee it.

Instead I'm just focusing on preventing write conflicts. My solution now is forcing explicit initialization of normally lazy-initialized class/class inst vars in a Metacello postLoad on GS. Since the values are mostly immutable, they can be safely persisted, and since the mutable values are all instances of a custom dictionary-like cache class that uses a TransientRecursionLock to guard access to an internal collection wrapped in a TransientValue on GS, these instances can be safely persisted too. In my tests this eliminated all write conflicts, so XMLParser, XMLParserHTML, XMLParserStAX, XPath, and XMLWriter should all be conflict-free during normal use.
 
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass