Pavel's ChangeLog week of 2018-02-05

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Pavel's ChangeLog week of 2018-02-05

Pavel Krivanek-3
Hi,

among other less interesting things, I spent some time on existing
PosixSharedMemory project. It is a UFFI binding for the LibC methods
that provide support for the memory allocation between several
separate processes. I significantly improved the performance by
implementing the block access. Writing of 10MB byte array takes about
1 millisecond, reading of it from other image took me about 4
milliseconds. While serialization with Fuel is very fast, it opens
interesting possibilities.
To have a shared memory without synchronization tools is not very
useful so I wrote a basic UFFI interface for the POSIX named
semaphores. They are quite easy to use and work nicely with Pharo. The
VM can all wait on the semaphore or it can check the status of it
periodically in an image thread. It has two small disadvantages. It
requires to dynamically link the next library (pthread) and they must
be cleaned manually. I plan to look at System V alternative in future.
Now we should write a nice framework for inter-image communication on
top of it or/and adopt Seamless for it ;-)

Cheers,
-- Pavel

Reply | Threaded
Open this post in threaded view
|

Re: Pavel's ChangeLog week of 2018-02-05

Stephane Ducasse-3
Thanks Pavel this looks quite fast :).
Do you have a scenario in mind that could take advantage of this?

Stef

On Mon, Feb 12, 2018 at 10:33 AM, Pavel Krivanek
<[hidden email]> wrote:

> Hi,
>
> among other less interesting things, I spent some time on existing
> PosixSharedMemory project. It is a UFFI binding for the LibC methods
> that provide support for the memory allocation between several
> separate processes. I significantly improved the performance by
> implementing the block access. Writing of 10MB byte array takes about
> 1 millisecond, reading of it from other image took me about 4
> milliseconds. While serialization with Fuel is very fast, it opens
> interesting possibilities.
> To have a shared memory without synchronization tools is not very
> useful so I wrote a basic UFFI interface for the POSIX named
> semaphores. They are quite easy to use and work nicely with Pharo. The
> VM can all wait on the semaphore or it can check the status of it
> periodically in an image thread. It has two small disadvantages. It
> requires to dynamically link the next library (pthread) and they must
> be cleaned manually. I plan to look at System V alternative in future.
> Now we should write a nice framework for inter-image communication on
> top of it or/and adopt Seamless for it ;-)
>
> Cheers,
> -- Pavel
>

Reply | Threaded
Open this post in threaded view
|

Re: Pavel's ChangeLog week of 2018-02-05

Thierry Goubier
Le 14/02/2018 à 20:19, Stephane Ducasse a écrit :
> Thanks Pavel this looks quite fast :).
> Do you have a scenario in mind that could take advantage of this?

I'd be very interested to see that used with image segments for objects
migration between images running in different processes (or coupled with
a distributed shared memory implementation like [1]).

If latency is low enough, I think multi-window applications could be
developped as multi-process single window images (and we could get
scalability to thousands of cores, if the application design is right.
Even code synchronisation between images would be easy to do).

Thierry

[1] https://hal.archives-ouvertes.fr/hal-01679052

> Stef
>
> On Mon, Feb 12, 2018 at 10:33 AM, Pavel Krivanek
> <[hidden email]> wrote:
>> Hi,
>>
>> among other less interesting things, I spent some time on existing
>> PosixSharedMemory project. It is a UFFI binding for the LibC methods
>> that provide support for the memory allocation between several
>> separate processes. I significantly improved the performance by
>> implementing the block access. Writing of 10MB byte array takes about
>> 1 millisecond, reading of it from other image took me about 4
>> milliseconds. While serialization with Fuel is very fast, it opens
>> interesting possibilities.
>> To have a shared memory without synchronization tools is not very
>> useful so I wrote a basic UFFI interface for the POSIX named
>> semaphores. They are quite easy to use and work nicely with Pharo. The
>> VM can all wait on the semaphore or it can check the status of it
>> periodically in an image thread. It has two small disadvantages. It
>> requires to dynamically link the next library (pthread) and they must
>> be cleaned manually. I plan to look at System V alternative in future.
>> Now we should write a nice framework for inter-image communication on
>> top of it or/and adopt Seamless for it ;-)
>>
>> Cheers,
>> -- Pavel
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Pavel's ChangeLog week of 2018-02-05

Tudor Girba-2
In reply to this post by Pavel Krivanek-3
Hi Pavel,

This is very cool. Is the code available?

Cheers,
Doru


> On Feb 12, 2018, at 10:33 AM, Pavel Krivanek <[hidden email]> wrote:
>
> Hi,
>
> among other less interesting things, I spent some time on existing
> PosixSharedMemory project. It is a UFFI binding for the LibC methods
> that provide support for the memory allocation between several
> separate processes. I significantly improved the performance by
> implementing the block access. Writing of 10MB byte array takes about
> 1 millisecond, reading of it from other image took me about 4
> milliseconds. While serialization with Fuel is very fast, it opens
> interesting possibilities.
> To have a shared memory without synchronization tools is not very
> useful so I wrote a basic UFFI interface for the POSIX named
> semaphores. They are quite easy to use and work nicely with Pharo. The
> VM can all wait on the semaphore or it can check the status of it
> periodically in an image thread. It has two small disadvantages. It
> requires to dynamically link the next library (pthread) and they must
> be cleaned manually. I plan to look at System V alternative in future.
> Now we should write a nice framework for inter-image communication on
> top of it or/and adopt Seamless for it ;-)
>
> Cheers,
> -- Pavel
>

--
www.tudorgirba.com
www.feenk.com

"From an abstract enough point of view, any two things are similar."





Reply | Threaded
Open this post in threaded view
|

Re: Pavel's ChangeLog week of 2018-02-05

Marcus Denker-4
In reply to this post by Thierry Goubier


> On 14 Feb 2018, at 21:34, Thierry Goubier <[hidden email]> wrote:
>
> Le 14/02/2018 à 20:19, Stephane Ducasse a écrit :
>> Thanks Pavel this looks quite fast :).
>> Do you have a scenario in mind that could take advantage of this?
>
> I'd be very interested to see that used with image segments for objects migration between images running in different processes (or coupled with a distributed shared memory implementation like [1]).
>
> If latency is low enough, I think multi-window applications could be developped as multi-process single window images (and we could get scalability to thousands of cores, if the application design is right. Even code synchronisation between images would be easy to do).
>
>
Yes, that is the direction to explore :-)

This started with me and Pavel discussion about the general idea that it would be interesting if Objects would actually be “really recursive” (and the whole system would be such an Object, too).
And then asking the question: what is the minimal thing we need to explore that direction?

>
> [1] https://hal.archives-ouvertes.fr/hal-01679052
>

I will read that.

        Marcus


Reply | Threaded
Open this post in threaded view
|

Re: Pavel's ChangeLog week of 2018-02-05

Thierry Goubier
2018-02-15 11:31 GMT+01:00 Marcus Denker <[hidden email]>:

>
>
>> On 14 Feb 2018, at 21:34, Thierry Goubier <[hidden email]> wrote:
>>
>> Le 14/02/2018 à 20:19, Stephane Ducasse a écrit :
>>> Thanks Pavel this looks quite fast :).
>>> Do you have a scenario in mind that could take advantage of this?
>>
>> I'd be very interested to see that used with image segments for objects migration between images running in different processes (or coupled with a distributed shared memory implementation like [1]).
>>
>> If latency is low enough, I think multi-window applications could be developped as multi-process single window images (and we could get scalability to thousands of cores, if the application design is right. Even code synchronisation between images would be easy to do).
>>
>>
> Yes, that is the direction to explore :-)

When I worked on that sort of systems, one of the questions I had was:
what about the GC ? But yes, in today's context, focusing on process
based concurrency can be interesting.

> This started with me and Pavel discussion about the general idea that it would be interesting if Objects would actually be “really recursive” (and the whole system would be such an Object, too).

I'm not sure I get the concept. Would you be ready to explain ?

> And then asking the question: what is the minimal thing we need to explore that direction?

On a meta-level, I can understand that.

Thierry

>>
>> [1] https://hal.archives-ouvertes.fr/hal-01679052
>>
>
> I will read that.

I talked to Loic about that sort of ideas (his S-DSM could do even
more interesting things) but also of the fact we don't have the
resources and time to explore it :(

Thierry

>
>         Marcus
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Pavel's ChangeLog week of 2018-02-05

Marcus Denker-4


> On 15 Feb 2018, at 12:56, Thierry Goubier <[hidden email]> wrote:
>
> 2018-02-15 11:31 GMT+01:00 Marcus Denker <[hidden email]>:
>>
>>
>>> On 14 Feb 2018, at 21:34, Thierry Goubier <[hidden email]> wrote:
>>>
>>> Le 14/02/2018 à 20:19, Stephane Ducasse a écrit :
>>>> Thanks Pavel this looks quite fast :).
>>>> Do you have a scenario in mind that could take advantage of this?
>>>
>>> I'd be very interested to see that used with image segments for objects migration between images running in different processes (or coupled with a distributed shared memory implementation like [1]).
>>>
>>> If latency is low enough, I think multi-window applications could be developped as multi-process single window images (and we could get scalability to thousands of cores, if the application design is right. Even code synchronisation between images would be easy to do).
>>>
>>>
>> Yes, that is the direction to explore :-)
>
> When I worked on that sort of systems, one of the questions I had was:
> what about the GC ? But yes, in today's context, focusing on process
> based concurrency can be interesting.
>
>> This started with me and Pavel discussion about the general idea that it would be interesting if Objects would actually be “really recursive” (and the whole system would be such an Object, too).
>
> I'm not sure I get the concept. Would you be ready to explain ?

I have started to write something about it (I need to make it clearer for myself, too).
When it is in some presentable shape (currently just notes) I will forward it.

        Marcus