VM handle management

>
> Folks -
>
> [Warning: Long post. Please try to stay on topic and don't get into any
> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come up
> with these issues. This is a practical proposal that I'd like to implement].
>
> For the third time this month I ran into the situation that I was writing a
> plugin that would create an internal data structure (containing pointers
> etc) and where I wanted to return a handle to this structure and pass that
> back to the VM.
>
> Since I'm a lazy bastard, what I usually do is something like this in the
> support code:
>
> int sqNewFooBar(void) {
>  /* creates a new foobar and returns a handle */
>  FooBar *newFoo = malloc(1, sizeof(FooBar));
>
>  /* return a handle, sort of */
>  return (int) newFoo;
> }
>
> void sqDeleteFooBar(int handle) {
>  FooBar *theFoo = (FooBar*) handle;
>  free(handle);
> }
>
> I don't have to tell you how horrible this is and what all the problems with
> the approach are - I'm sure you're all well aware of it. The question is,
> how do we fix this?
>
> Here's my proposal. I say let's extend the VM with 4 new proxy functions,
> namely:
>
>  /* Registers a pointer and returns a handle.
>     The type argument ensures that the handle cannot be forged. */
>  OOP registerHandleType(void *ptr, char *type);
>
>  /* Unregister a handle */
>  void unregisterHandle(void *ptr);
>
>  /* Look up the pointer for a given handle/pair type */
>  void* lookupHandleType(OOP handle, char *type);
>
>  /* Ditto, just based on a stack index */
>  void* stackHandleValueType(int stackIndex, char *type);
>
> With these functions the support code can *always* deal with pointers, never
> with OOPs. The glue code does the translation between OOP and pointer. As a
> consequence the above support code becomes:
>
> FooBar *sqNewFooBar(void) {
>  /* creates a new foobar and returns it */
>  FooBar *newFoo = malloc(1, sizeof(FooBar));
>  return newFoo;
> }
>
> void sqDeleteFooBar(FooBar *theFoo) {
>  /* deletes the FooBar */
>  free(theFoo);
> }
>
> and the glue code in the plugin becomes:
>
> primitiveNewFooBar
>    "Creates a new foobar"
>    | foobar |
>    <var: #foobar type: 'FooBar*'>
>    foobar := self sqNewFooBar.
>    handle := interpreterProxy registerHandle: foobar type: 'FooBar';
>    interpreterProxy pop: interpreterProxy methodArgumentCount+1 thenPush:
> handle.
>
> primitiveDestroyFooBar
>    "Destroys a foobar"
>    | foobar |
>    <var: #foobar type: 'FooBar*'>
>    foobar := interpreterProxy lookupHandle: (interpreterProxy stackValue: 0)
> type: 'FooBar'.
>    "or simply:
>      foobar = interpreterProxy stackHandleValue: 0 type: 'FooBar'
>    "
>    foobar ifNotNil:[
>        interpreterProxy unregisterHandle: foobar.
>        self sqDestroy: foobar.
>    ].
>    interpreterProxy pop: interpreterProxy methodArgumentCount.
>
> What is interesting is that handles are completely opaque structures. They
> could be integers, byte arrays, whatever representation we choose, the only
> requirement is that the VM has a handleTable that can look up these
> pointers.
>
> The obvious advantages of such a scheme are:
>
> * It's safe. We don't expose a pointer but rather a handle that the VM knows
> about and that does not expose any dangerous internals.
>
> * It's secure. By passing the additional "type" argument you can ensure that
> only handles registered as that type are used. I.e., no passing a SocketPtr
> into the FilePlugin.
>
> * It's simple. I've written at least three handle management implementations
> and I'm getting sick and tired of it.
>
> * It allows sharing. For example, OSProcessPlugin likes to do manipulate and
> register / deregister file handles, and can now safely look up the internal
> structure from the handle. (there are some issues here but they are no worse
> than what OSProcess already does)
>
> If you agree, I'd like to implement this scheme and rewrite SocketPlugin and
> FilePlugin as examples for how to use such handles. They are already places
> that expose a ton of pretty risky information and it'd be a good example for
> how to utilize handles.
>
> Cheers,
>  - Andreas
>

> Then, for image side a handle is just an integer value - an index in that array.
> The requirement is that once handle registered, both pointers should
> remain valid,
> so you can always compare strings, to make sure that type is ok.
>
> I using a following scheme for making register/unregister to be always O(1):
> Each array entry is a
> [<any data>, nextFree ]
> pair, where nextFree field used for maintaining a free entries list.
> - if element is unused(or free) , then its nextFree value>  0 , and points to a
> nextFree element. If nextFree value = 0 , then it is last free element in array.
> - if nextFree value<  0 then this element is under use.
> Additionally, there is a variable (FirstFree), which points to the
> head of free elements list.
>
> Then, registering a new element is:
> 1. check a FirstFree value,
> 2. if its 0 , then grow an array and format new elements (by filling
> all nextFree values), updating FirstFree value.Goto 1.
> 3. if its a non-zero, then set FirstFree to a nextFree value of a
> given element, and set element's nextFree to -1, to indicate that it
> is under use.
>
> Unregistering is simple as well:
>    - record a FirstFree value into nextFree field of element to be freed
>    - set  FirstFree value to an index of element which is just freed
>
> In this way, registering a new handle, or unregistering , always takes
> O(1) time, except the cases
> when you need to grow array.
>
> On 24 March 2010 05:10, Andreas Raab<[hidden email]>  wrote:
>>
>> Folks -
>>
>> [Warning: Long post. Please try to stay on topic and don't get into any
>> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come up
>> with these issues. This is a practical proposal that I'd like to implement].
>>
>> For the third time this month I ran into the situation that I was writing a
>> plugin that would create an internal data structure (containing pointers
>> etc) and where I wanted to return a handle to this structure and pass that
>> back to the VM.
>>
>> Since I'm a lazy bastard, what I usually do is something like this in the
>> support code:
>>
>> int sqNewFooBar(void) {
>>   /* creates a new foobar and returns a handle */
>>   FooBar *newFoo = malloc(1, sizeof(FooBar));
>>
>>   /* return a handle, sort of */
>>   return (int) newFoo;
>> }
>>
>> void sqDeleteFooBar(int handle) {
>>   FooBar *theFoo = (FooBar*) handle;
>>   free(handle);
>> }
>>
>> I don't have to tell you how horrible this is and what all the problems with
>> the approach are - I'm sure you're all well aware of it. The question is,
>> how do we fix this?
>>
>> Here's my proposal. I say let's extend the VM with 4 new proxy functions,
>> namely:
>>
>>   /* Registers a pointer and returns a handle.
>>      The type argument ensures that the handle cannot be forged. */
>>   OOP registerHandleType(void *ptr, char *type);
>>
>>   /* Unregister a handle */
>>   void unregisterHandle(void *ptr);
>>
>>   /* Look up the pointer for a given handle/pair type */
>>   void* lookupHandleType(OOP handle, char *type);
>>
>>   /* Ditto, just based on a stack index */
>>   void* stackHandleValueType(int stackIndex, char *type);
>>
>> With these functions the support code can *always* deal with pointers, never
>> with OOPs. The glue code does the translation between OOP and pointer. As a
>> consequence the above support code becomes:
>>
>> FooBar *sqNewFooBar(void) {
>>   /* creates a new foobar and returns it */
>>   FooBar *newFoo = malloc(1, sizeof(FooBar));
>>   return newFoo;
>> }
>>
>> void sqDeleteFooBar(FooBar *theFoo) {
>>   /* deletes the FooBar */
>>   free(theFoo);
>> }
>>
>> and the glue code in the plugin becomes:
>>
>> primitiveNewFooBar
>>     "Creates a new foobar"
>>     | foobar |
>>     <var: #foobar type: 'FooBar*'>
>>     foobar := self sqNewFooBar.
>>     handle := interpreterProxy registerHandle: foobar type: 'FooBar';
>>     interpreterProxy pop: interpreterProxy methodArgumentCount+1 thenPush:
>> handle.
>>
>> primitiveDestroyFooBar
>>     "Destroys a foobar"
>>     | foobar |
>>     <var: #foobar type: 'FooBar*'>
>>     foobar := interpreterProxy lookupHandle: (interpreterProxy stackValue: 0)
>> type: 'FooBar'.
>>     "or simply:
>>       foobar = interpreterProxy stackHandleValue: 0 type: 'FooBar'
>>     "
>>     foobar ifNotNil:[
>>         interpreterProxy unregisterHandle: foobar.
>>         self sqDestroy: foobar.
>>     ].
>>     interpreterProxy pop: interpreterProxy methodArgumentCount.
>>
>> What is interesting is that handles are completely opaque structures. They
>> could be integers, byte arrays, whatever representation we choose, the only
>> requirement is that the VM has a handleTable that can look up these
>> pointers.
>>
>> The obvious advantages of such a scheme are:
>>
>> * It's safe. We don't expose a pointer but rather a handle that the VM knows
>> about and that does not expose any dangerous internals.
>>
>> * It's secure. By passing the additional "type" argument you can ensure that
>> only handles registered as that type are used. I.e., no passing a SocketPtr
>> into the FilePlugin.
>>
>> * It's simple. I've written at least three handle management implementations
>> and I'm getting sick and tired of it.
>>
>> * It allows sharing. For example, OSProcessPlugin likes to do manipulate and
>> register / deregister file handles, and can now safely look up the internal
>> structure from the handle. (there are some issues here but they are no worse
>> than what OSProcess already does)
>>
>> If you agree, I'd like to implement this scheme and rewrite SocketPlugin and
>> FilePlugin as examples for how to use such handles. They are already places
>> that expose a ton of pretty risky information and it'd be a good example for
>> how to utilize handles.
>>
>> Cheers,
>>   - Andreas
>>
>
>
>

>
> Folks -
>
> [Warning: Long post. Please try to stay on topic and don't get into any
> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come
> up with these issues. This is a practical proposal that I'd like to
> implement].
>
> For the third time this month I ran into the situation that I was
> writing a plugin that would create an internal data structure
> (containing pointers etc) and where I wanted to return a handle to this
> structure and pass that back to the VM.
>
> Since I'm a lazy bastard, what I usually do is something like this in
> the support code:
>
> int sqNewFooBar(void) {
>   /* creates a new foobar and returns a handle */
>   FooBar *newFoo = malloc(1, sizeof(FooBar));
>
>   /* return a handle, sort of */
>   return (int) newFoo;
> }

>
>
> Here's my proposal. I say let's extend the VM with 4 new proxy functions, namely:
>
>  /* Registers a pointer and returns a handle.
>     The type argument ensures that the handle cannot be forged. */
>  OOP registerHandleType(void *ptr, char *type);
>
>  /* Unregister a handle */
>  void unregisterHandle(void *ptr);
>
>  /* Look up the pointer for a given handle/pair type */
>  void* lookupHandleType(OOP handle, char *type);
>
>  /* Ditto, just based on a stack index */
>  void* stackHandleValueType(int stackIndex, char *type);

>
>
> On Tue, Mar 23, 2010 at 08:10:18PM -0700, Andreas Raab wrote:
>>
>> Folks -
>>
>> [Warning: Long post. Please try to stay on topic and don't get into any
>> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come
>> up with these issues. This is a practical proposal that I'd like to
>> implement].
>>
>> For the third time this month I ran into the situation that I was
>> writing a plugin that would create an internal data structure
>> (containing pointers etc) and where I wanted to return a handle to this
>> structure and pass that back to the VM.
>>
>> Since I'm a lazy bastard, what I usually do is something like this in
>> the support code:
>>
>> int sqNewFooBar(void) {
>>  /* creates a new foobar and returns a handle */
>>  FooBar *newFoo = malloc(1, sizeof(FooBar));
>>
>>  /* return a handle, sort of */
>>  return (int) newFoo;
>> }
>
> This seems like solving the wrong problem. If the problem is
> that pointers cannot be treated as sqInt or OOP values, then
> the solution is simple: Don't Do That. Declaring the types
> properly (in slang where necessary) results in simpler slang
> and simpler support code overall. That is my actual experience
> after working through quite a few 64/32 bit issues, not just
> a theoretical statement.

>
> On 24.03.2010, at 13:40, David T. Lewis wrote:
> >
> >
> > On Tue, Mar 23, 2010 at 08:10:18PM -0700, Andreas Raab wrote:
> >>
> >> Folks -
> >>
> >> [Warning: Long post. Please try to stay on topic and don't get into any
> >> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come
> >> up with these issues. This is a practical proposal that I'd like to
> >> implement].
> >>
> >> For the third time this month I ran into the situation that I was
> >> writing a plugin that would create an internal data structure
> >> (containing pointers etc) and where I wanted to return a handle to this
> >> structure and pass that back to the VM.
> >>
> >> Since I'm a lazy bastard, what I usually do is something like this in
> >> the support code:
> >>
> >> int sqNewFooBar(void) {
> >>  /* creates a new foobar and returns a handle */
> >>  FooBar *newFoo = malloc(1, sizeof(FooBar));
> >>
> >>  /* return a handle, sort of */
> >>  return (int) newFoo;
> >> }
> >
> > This seems like solving the wrong problem. If the problem is
> > that pointers cannot be treated as sqInt or OOP values, then
> > the solution is simple: Don't Do That. Declaring the types
> > properly (in slang where necessary) results in simpler slang
> > and simpler support code overall. That is my actual experience
> > after working through quite a few 64/32 bit issues, not just
> > a theoretical statement.
>
> No, that's not the problem. The problem is having to implement a handle
> registry over and over again, for many plugins. Whenever a plugin needs
> to store some dynamic state, you end up writing this again. I've seen
> this with Cairo surfaces, Freetype fonts, DBus connections, OpenGL
> contexts etc.