I attach the forwarded post I posted in pharo-users but if you want the summary its a collection of questions whether its possible to use memory mapped files shared memory with Pharo VM for IPC. I thought to forward also here since this may be a VM related too. Any help is appreciated
---------- Forwarded message --------- From: Dimitris Chloupis <[hidden email]> Date: Sun, Jan 17, 2016 at 6:41 PM Subject: Memory mapped files and Pharo (aka Unreal Engine 4 integration with Pharo) To: Any question about pharo is welcome <[hidden email]> An apology for the long post but this is not a simple issue and hence not a simple question. So I am looking into ways to integrate Pharo with Unreal to bring some nuclear powered graphics to Pharo and make us first class citizents to 2D and 3D world. https://www.unrealengine.com/what-is-unreal-engine-4 2) Will I be limited by the fact that the VM currently does not multithread or cannot use multithreading libraries ? ( I have no intention of using multithreading but some handling of process access to the data may be necessary to make sure data is safe from concurent modification) 3) Is there anything else I should be aware of as common pitfalls for such implementation ? |
Hi Dimitris,
On Sun, Jan 17, 2016 at 8:57 AM, Dimitris Chloupis <[hidden email]> wrote:
You should be able to map a file via the FFI calling mmap, and access it via an ExternalPointer. But that's crappy.
As you've noticed we intend to provide a threaded FFI that will allow you to use threaded libraries. But that's not really the issue. There must be some handshaking protocol for the two halves to communicate via shared memory without conflict. The threaded FFI should make it possible to call e.g. pthread_cond_wait to synchronise, but a lower-level test-and-set or conditional move facility would be nicer. See below.
Forgive me for diving down a bit before I answer your question, but I find being concrete helps. Spur has a segmented memory model. When one grows the heap the VM allocates memory via mmap and integrates it with the rest of the heap by using "bridge" objects at the end of each segment. A bridge object is two 64-bit words. It says "I am a bytes object" so that the GC will never look inside the object, and it lies about its size, saying "I am large enough to span to the start of the next segment". So when adding a new segment, the bridge in the segment before the new one is "shortened" to point to the start of the new segment, and a new bridge is added to the end of the segment so that it points to the start of the next segment in memory. The last segment's bridge has a zero length. Spur also provides pinning, as simple as a per-object flag that tells the GC not to move an object. So bridges are pinned, and hence compaction leaves bridges unmolested, but any object in old space can be pinned also. So an elegant way of providing shared memory in Spur would be to add a "map a file as a byte array" primitive. There are issues with this. The first 16 bytes of the mapped file would have to be used to construct the header for a ByteArray that would comprise the rest of the segment, excepting another 16 bytes at the end that would need to be a bridge. So if you wanted to share this with another application you'd probably want to allocate a file that was, say, 2k bytes bigger than needed, and use the first and last 1k bytes to hide the header and the bridge. Ah, better still would be to construct three objects in the segment, an initial ByteArray that stretches to 16 bytes before the second page, a ByteArray whose contents start at the beginning of the second page and reach all the way to the penultimate page, and then a ByteArray to reach from the end of the penultimate page to the bridge at the end of the mmapped file segment. All three objects could be pinned and be prevented from being GC'ed until the entire segment was released. That would give you a ByteArray (the middle of the three) whose contents were all but the first and last pages of the mmapped file, aligned on a file page boundary and whose length was a multiple of the page size. C++/C clients could then map the central portion and use e.g. pointers to access it. To implement this there would be an mmap-file-as-segment primitive and perhaps a release-segment primitive, or some magic in the GC to release the segment when the ByteArrays were no longer accessed. So then I could imagine test-and-set or conditional-move primitives on ByteArray that supported manipulating locks in a ByteArray, and by extension, in the shared file. I think this kind of approach would give you the fastest, most direct access to shared memory I can think of. Does this appeal?
_,,,^..^,,,_ best, Eliot |
Eliot as long as you integrate this, because my knowledge of VM is close to zero, and make it a standard part of the VM I am perfectly ok with it. I dont care about an optimal solution just a solution that works . All I want is the ability to do shared memory management from inside Pharo. I am very new to all this so I am not aware of the problems and limititations I will face. But I know that you are far better than me to provide a good implementation. On Sun, Jan 17, 2016 at 10:18 PM Eliot Miranda <[hidden email]> wrote:
|
Ok, well the old macintosh safari squeak VM plugin code did this. We setup a shared memory area (there are many ways to do that, some are very obsolete). So the VM would make the shared memory area, and pass that information back to the Safari plugin via a pipe. The Safari plugin would then get the magic token describing the shared memory and now we had both sharing the same memory area. Squeak supports a offscreen drawing area, so we just would draw into that, which was of course backing into the shared memory area. Note there is a problem here with BitBlt, some forms read one word off the end of the screen area. This doesn't cause a problem for the algorithm, but DOES cause a problem if your screen area is 4K and the next page frame is not yours. In any case once a draw cycle was done we sent a draw request over the pipe. Later this was changed to use a semaphore in the shared memory area so the VM couldn't overdrive the browser. So draw, signal pipe, wait. The safari plugin would get the request, draw, then signal the semaphore. When I was looking into this I found it difficult to understand as there are many ways to do it and no good examples. However the code was do a shm_open, ftruncate and mmap if (*plugin->sharedMemoryName == 0x00) { sharedMemIDIncremental++; sprintf(plugin->sharedMemoryName,"%i",42+plugin->pid+sharedMemIDIncremental); plugin->sharedMemoryfd = shm_open(plugin->sharedMemoryName,O_RDWR | O_CREAT,S_IRUSR+S_IWUSR); if (plugin->sharedMemoryfd < 0) { plugin->sharedMemoryfd = 0; perror("NP: shared memory shm_open failed\n"); return; } ftruncate(plugin->sharedMemoryfd,gWindowMaxLength*gWindowMaxLength*4+20); plugin->sharedMemoryBlock= mmap(0, (gWindowMaxLength*gWindowMaxLength*4)+20, PROT_READ | PROT_WRITE, MAP_SHARED, plugin->sharedMemoryfd,0); DPRINT("NP: shared memory mmap memory fd %i at %i \n", plugin->sharedMemoryfd,plugin->sharedMemoryBlock); if (plugin->sharedMemoryBlock == MAP_FAILED) { perror("NP: shared memory mmap failed %i\n"); plugin->sharedMemoryBlock = NULL; close(plugin->sharedMemoryfd); shm_unlink(plugin->sharedMemoryName); *plugin->sharedMemoryName = 0x00; plugin->sharedMemoryfd = 0; return; } } On Sun, Jan 17, 2016 at 1:12 PM, Dimitris Chloupis <[hidden email]> wrote:
=========================================================================== John M. McIntosh. Corporate Smalltalk Consulting Ltd https://www.linkedin.com/in/smalltalk =========================================================================== |
In reply to this post by kilon.alios
On Sun, Jan 17, 2016 at 1:12 PM, Dimitris Chloupis <[hidden email]> wrote:
Right. It would become a standard facility.
Good. But remember this and be prepared to nag when the threaded FFI arrives :-). And if there are volunteers out there looking for an interesting project that's not too big this is a candidate.
Good point.
_,,,^..^,,,_ best, Eliot |
Than you very much Eliot. I am both interested in this and threaded FFI, and I am sure I will need your wisdom with mmap in the future, so I will be around ;) On Sun, 17 Jan 2016 at 23:54, Eliot Miranda <[hidden email]> wrote:
|
Free forum by Nabble | Edit this page |