VM handle management

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

VM handle management

Andreas.Raab
 
Folks -

[Warning: Long post. Please try to stay on topic and don't get into any
SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come
up with these issues. This is a practical proposal that I'd like to
implement].

For the third time this month I ran into the situation that I was
writing a plugin that would create an internal data structure
(containing pointers etc) and where I wanted to return a handle to this
structure and pass that back to the VM.

Since I'm a lazy bastard, what I usually do is something like this in
the support code:

int sqNewFooBar(void) {
   /* creates a new foobar and returns a handle */
   FooBar *newFoo = malloc(1, sizeof(FooBar));

   /* return a handle, sort of */
   return (int) newFoo;
}

void sqDeleteFooBar(int handle) {
   FooBar *theFoo = (FooBar*) handle;
   free(handle);
}

I don't have to tell you how horrible this is and what all the problems
with the approach are - I'm sure you're all well aware of it. The
question is, how do we fix this?

Here's my proposal. I say let's extend the VM with 4 new proxy
functions, namely:

   /* Registers a pointer and returns a handle.
      The type argument ensures that the handle cannot be forged. */
   OOP registerHandleType(void *ptr, char *type);

   /* Unregister a handle */
   void unregisterHandle(void *ptr);

   /* Look up the pointer for a given handle/pair type */
   void* lookupHandleType(OOP handle, char *type);

   /* Ditto, just based on a stack index */
   void* stackHandleValueType(int stackIndex, char *type);

With these functions the support code can *always* deal with pointers,
never with OOPs. The glue code does the translation between OOP and
pointer. As a consequence the above support code becomes:

FooBar *sqNewFooBar(void) {
   /* creates a new foobar and returns it */
   FooBar *newFoo = malloc(1, sizeof(FooBar));
   return newFoo;
}

void sqDeleteFooBar(FooBar *theFoo) {
   /* deletes the FooBar */
   free(theFoo);
}

and the glue code in the plugin becomes:

primitiveNewFooBar
     "Creates a new foobar"
     | foobar |
     <var: #foobar type: 'FooBar*'>
     foobar := self sqNewFooBar.
     handle := interpreterProxy registerHandle: foobar type: 'FooBar';
     interpreterProxy pop: interpreterProxy methodArgumentCount+1
thenPush: handle.

primitiveDestroyFooBar
     "Destroys a foobar"
     | foobar |
     <var: #foobar type: 'FooBar*'>
     foobar := interpreterProxy lookupHandle: (interpreterProxy
stackValue: 0) type: 'FooBar'.
     "or simply:
       foobar = interpreterProxy stackHandleValue: 0 type: 'FooBar'
     "
     foobar ifNotNil:[
         interpreterProxy unregisterHandle: foobar.
         self sqDestroy: foobar.
     ].
     interpreterProxy pop: interpreterProxy methodArgumentCount.

What is interesting is that handles are completely opaque structures.
They could be integers, byte arrays, whatever representation we choose,
the only requirement is that the VM has a handleTable that can look up
these pointers.

The obvious advantages of such a scheme are:

* It's safe. We don't expose a pointer but rather a handle that the VM
knows about and that does not expose any dangerous internals.

* It's secure. By passing the additional "type" argument you can ensure
that only handles registered as that type are used. I.e., no passing a
SocketPtr into the FilePlugin.

* It's simple. I've written at least three handle management
implementations and I'm getting sick and tired of it.

* It allows sharing. For example, OSProcessPlugin likes to do manipulate
and register / deregister file handles, and can now safely look up the
internal structure from the handle. (there are some issues here but they
are no worse than what OSProcess already does)

If you agree, I'd like to implement this scheme and rewrite SocketPlugin
and FilePlugin as examples for how to use such handles. They are already
places that expose a ton of pretty risky information and it'd be a good
example for how to utilize handles.

Cheers,
   - Andreas
Reply | Threaded
Open this post in threaded view
|

Re: VM handle management

Igor Stasenko

A good rationalization.
So, as i understand, a VM keeps a list of (void*, char*)
pairs in an integer-based array.
Then, for image side a handle is just an integer value - an index in that array.
The requirement is that once handle registered, both pointers should
remain valid,
so you can always compare strings, to make sure that type is ok.

I using a following scheme for making register/unregister to be always O(1):
Each array entry is a
[ <any data>, nextFree ]
pair, where nextFree field used for maintaining a free entries list.
- if element is unused(or free) , then its nextFree value > 0 , and points to a
nextFree element. If nextFree value = 0 , then it is last free element in array.
- if nextFree value < 0 then this element is under use.
Additionally, there is a variable (FirstFree), which points to the
head of free elements list.

Then, registering a new element is:
1. check a FirstFree value,
2. if its 0 , then grow an array and format new elements (by filling
all nextFree values), updating FirstFree value.Goto 1.
3. if its a non-zero, then set FirstFree to a nextFree value of a
given element, and set element's nextFree to -1, to indicate that it
is under use.

Unregistering is simple as well:
  - record a FirstFree value into nextFree field of element to be freed
  - set  FirstFree value to an index of element which is just freed

In this way, registering a new handle, or unregistering , always takes
O(1) time, except the cases
when you need to grow array.

On 24 March 2010 05:10, Andreas Raab <[hidden email]> wrote:

>
> Folks -
>
> [Warning: Long post. Please try to stay on topic and don't get into any
> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come up
> with these issues. This is a practical proposal that I'd like to implement].
>
> For the third time this month I ran into the situation that I was writing a
> plugin that would create an internal data structure (containing pointers
> etc) and where I wanted to return a handle to this structure and pass that
> back to the VM.
>
> Since I'm a lazy bastard, what I usually do is something like this in the
> support code:
>
> int sqNewFooBar(void) {
>  /* creates a new foobar and returns a handle */
>  FooBar *newFoo = malloc(1, sizeof(FooBar));
>
>  /* return a handle, sort of */
>  return (int) newFoo;
> }
>
> void sqDeleteFooBar(int handle) {
>  FooBar *theFoo = (FooBar*) handle;
>  free(handle);
> }
>
> I don't have to tell you how horrible this is and what all the problems with
> the approach are - I'm sure you're all well aware of it. The question is,
> how do we fix this?
>
> Here's my proposal. I say let's extend the VM with 4 new proxy functions,
> namely:
>
>  /* Registers a pointer and returns a handle.
>     The type argument ensures that the handle cannot be forged. */
>  OOP registerHandleType(void *ptr, char *type);
>
>  /* Unregister a handle */
>  void unregisterHandle(void *ptr);
>
>  /* Look up the pointer for a given handle/pair type */
>  void* lookupHandleType(OOP handle, char *type);
>
>  /* Ditto, just based on a stack index */
>  void* stackHandleValueType(int stackIndex, char *type);
>
> With these functions the support code can *always* deal with pointers, never
> with OOPs. The glue code does the translation between OOP and pointer. As a
> consequence the above support code becomes:
>
> FooBar *sqNewFooBar(void) {
>  /* creates a new foobar and returns it */
>  FooBar *newFoo = malloc(1, sizeof(FooBar));
>  return newFoo;
> }
>
> void sqDeleteFooBar(FooBar *theFoo) {
>  /* deletes the FooBar */
>  free(theFoo);
> }
>
> and the glue code in the plugin becomes:
>
> primitiveNewFooBar
>    "Creates a new foobar"
>    | foobar |
>    <var: #foobar type: 'FooBar*'>
>    foobar := self sqNewFooBar.
>    handle := interpreterProxy registerHandle: foobar type: 'FooBar';
>    interpreterProxy pop: interpreterProxy methodArgumentCount+1 thenPush:
> handle.
>
> primitiveDestroyFooBar
>    "Destroys a foobar"
>    | foobar |
>    <var: #foobar type: 'FooBar*'>
>    foobar := interpreterProxy lookupHandle: (interpreterProxy stackValue: 0)
> type: 'FooBar'.
>    "or simply:
>      foobar = interpreterProxy stackHandleValue: 0 type: 'FooBar'
>    "
>    foobar ifNotNil:[
>        interpreterProxy unregisterHandle: foobar.
>        self sqDestroy: foobar.
>    ].
>    interpreterProxy pop: interpreterProxy methodArgumentCount.
>
> What is interesting is that handles are completely opaque structures. They
> could be integers, byte arrays, whatever representation we choose, the only
> requirement is that the VM has a handleTable that can look up these
> pointers.
>
> The obvious advantages of such a scheme are:
>
> * It's safe. We don't expose a pointer but rather a handle that the VM knows
> about and that does not expose any dangerous internals.
>
> * It's secure. By passing the additional "type" argument you can ensure that
> only handles registered as that type are used. I.e., no passing a SocketPtr
> into the FilePlugin.
>
> * It's simple. I've written at least three handle management implementations
> and I'm getting sick and tired of it.
>
> * It allows sharing. For example, OSProcessPlugin likes to do manipulate and
> register / deregister file handles, and can now safely look up the internal
> structure from the handle. (there are some issues here but they are no worse
> than what OSProcess already does)
>
> If you agree, I'd like to implement this scheme and rewrite SocketPlugin and
> FilePlugin as examples for how to use such handles. They are already places
> that expose a ton of pretty risky information and it'd be a good example for
> how to utilize handles.
>
> Cheers,
>  - Andreas
>



--
Best regards,
Igor Stasenko AKA sig.
Reply | Threaded
Open this post in threaded view
|

Re: VM handle management

Andreas.Raab
 
On 3/23/2010 8:56 PM, Igor Stasenko wrote:
> A good rationalization.
> So, as i understand, a VM keeps a list of (void*, char*)
> pairs in an integer-based array.

Yes, something along these lines. I hadn't specifically thought how to
implement it, but this would certainly work.

Cheers,
   - Andreas

> Then, for image side a handle is just an integer value - an index in that array.
> The requirement is that once handle registered, both pointers should
> remain valid,
> so you can always compare strings, to make sure that type is ok.
>
> I using a following scheme for making register/unregister to be always O(1):
> Each array entry is a
> [<any data>, nextFree ]
> pair, where nextFree field used for maintaining a free entries list.
> - if element is unused(or free) , then its nextFree value>  0 , and points to a
> nextFree element. If nextFree value = 0 , then it is last free element in array.
> - if nextFree value<  0 then this element is under use.
> Additionally, there is a variable (FirstFree), which points to the
> head of free elements list.
>
> Then, registering a new element is:
> 1. check a FirstFree value,
> 2. if its 0 , then grow an array and format new elements (by filling
> all nextFree values), updating FirstFree value.Goto 1.
> 3. if its a non-zero, then set FirstFree to a nextFree value of a
> given element, and set element's nextFree to -1, to indicate that it
> is under use.
>
> Unregistering is simple as well:
>    - record a FirstFree value into nextFree field of element to be freed
>    - set  FirstFree value to an index of element which is just freed
>
> In this way, registering a new handle, or unregistering , always takes
> O(1) time, except the cases
> when you need to grow array.
>
> On 24 March 2010 05:10, Andreas Raab<[hidden email]>  wrote:
>>
>> Folks -
>>
>> [Warning: Long post. Please try to stay on topic and don't get into any
>> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come up
>> with these issues. This is a practical proposal that I'd like to implement].
>>
>> For the third time this month I ran into the situation that I was writing a
>> plugin that would create an internal data structure (containing pointers
>> etc) and where I wanted to return a handle to this structure and pass that
>> back to the VM.
>>
>> Since I'm a lazy bastard, what I usually do is something like this in the
>> support code:
>>
>> int sqNewFooBar(void) {
>>   /* creates a new foobar and returns a handle */
>>   FooBar *newFoo = malloc(1, sizeof(FooBar));
>>
>>   /* return a handle, sort of */
>>   return (int) newFoo;
>> }
>>
>> void sqDeleteFooBar(int handle) {
>>   FooBar *theFoo = (FooBar*) handle;
>>   free(handle);
>> }
>>
>> I don't have to tell you how horrible this is and what all the problems with
>> the approach are - I'm sure you're all well aware of it. The question is,
>> how do we fix this?
>>
>> Here's my proposal. I say let's extend the VM with 4 new proxy functions,
>> namely:
>>
>>   /* Registers a pointer and returns a handle.
>>      The type argument ensures that the handle cannot be forged. */
>>   OOP registerHandleType(void *ptr, char *type);
>>
>>   /* Unregister a handle */
>>   void unregisterHandle(void *ptr);
>>
>>   /* Look up the pointer for a given handle/pair type */
>>   void* lookupHandleType(OOP handle, char *type);
>>
>>   /* Ditto, just based on a stack index */
>>   void* stackHandleValueType(int stackIndex, char *type);
>>
>> With these functions the support code can *always* deal with pointers, never
>> with OOPs. The glue code does the translation between OOP and pointer. As a
>> consequence the above support code becomes:
>>
>> FooBar *sqNewFooBar(void) {
>>   /* creates a new foobar and returns it */
>>   FooBar *newFoo = malloc(1, sizeof(FooBar));
>>   return newFoo;
>> }
>>
>> void sqDeleteFooBar(FooBar *theFoo) {
>>   /* deletes the FooBar */
>>   free(theFoo);
>> }
>>
>> and the glue code in the plugin becomes:
>>
>> primitiveNewFooBar
>>     "Creates a new foobar"
>>     | foobar |
>>     <var: #foobar type: 'FooBar*'>
>>     foobar := self sqNewFooBar.
>>     handle := interpreterProxy registerHandle: foobar type: 'FooBar';
>>     interpreterProxy pop: interpreterProxy methodArgumentCount+1 thenPush:
>> handle.
>>
>> primitiveDestroyFooBar
>>     "Destroys a foobar"
>>     | foobar |
>>     <var: #foobar type: 'FooBar*'>
>>     foobar := interpreterProxy lookupHandle: (interpreterProxy stackValue: 0)
>> type: 'FooBar'.
>>     "or simply:
>>       foobar = interpreterProxy stackHandleValue: 0 type: 'FooBar'
>>     "
>>     foobar ifNotNil:[
>>         interpreterProxy unregisterHandle: foobar.
>>         self sqDestroy: foobar.
>>     ].
>>     interpreterProxy pop: interpreterProxy methodArgumentCount.
>>
>> What is interesting is that handles are completely opaque structures. They
>> could be integers, byte arrays, whatever representation we choose, the only
>> requirement is that the VM has a handleTable that can look up these
>> pointers.
>>
>> The obvious advantages of such a scheme are:
>>
>> * It's safe. We don't expose a pointer but rather a handle that the VM knows
>> about and that does not expose any dangerous internals.
>>
>> * It's secure. By passing the additional "type" argument you can ensure that
>> only handles registered as that type are used. I.e., no passing a SocketPtr
>> into the FilePlugin.
>>
>> * It's simple. I've written at least three handle management implementations
>> and I'm getting sick and tired of it.
>>
>> * It allows sharing. For example, OSProcessPlugin likes to do manipulate and
>> register / deregister file handles, and can now safely look up the internal
>> structure from the handle. (there are some issues here but they are no worse
>> than what OSProcess already does)
>>
>> If you agree, I'd like to implement this scheme and rewrite SocketPlugin and
>> FilePlugin as examples for how to use such handles. They are already places
>> that expose a ton of pretty risky information and it'd be a good example for
>> how to utilize handles.
>>
>> Cheers,
>>   - Andreas
>>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: VM handle management

johnmci
In reply to this post by Andreas.Raab
 

On 2010-03-23, at 8:10 PM, Andreas Raab wrote:

> If you agree, I'd like to implement this scheme and rewrite SocketPlugin and FilePlugin as examples for how to use such handles. They are already places that expose a ton of pretty risky information and it'd be a good example for how to utilize handles.
>
> Cheers,
>  - Andreas

This seems ok.
Don't forget the IPV6 support
Having a file primitive that let's you know if a file exists and if it is read or read/write or write would be helpful.


--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================





smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: VM handle management

David T. Lewis
In reply to this post by Andreas.Raab
 
On Tue, Mar 23, 2010 at 08:10:18PM -0700, Andreas Raab wrote:

>
> Folks -
>
> [Warning: Long post. Please try to stay on topic and don't get into any
> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come
> up with these issues. This is a practical proposal that I'd like to
> implement].
>
> For the third time this month I ran into the situation that I was
> writing a plugin that would create an internal data structure
> (containing pointers etc) and where I wanted to return a handle to this
> structure and pass that back to the VM.
>
> Since I'm a lazy bastard, what I usually do is something like this in
> the support code:
>
> int sqNewFooBar(void) {
>   /* creates a new foobar and returns a handle */
>   FooBar *newFoo = malloc(1, sizeof(FooBar));
>
>   /* return a handle, sort of */
>   return (int) newFoo;
> }

This seems like solving the wrong problem. If the problem is
that pointers cannot be treated as sqInt or OOP values, then
the solution is simple: Don't Do That. Declaring the types
properly (in slang where necessary) results in simpler slang
and simpler support code overall. That is my actual experience
after working through quite a few 64/32 bit issues, not just
a theoretical statement.

In practice, I don't think I have ever actually encounted a problem
due to accidentally passing a bogus handle to a plugin. It is definitely
possible, but it does not seem to be a practical concern. So to me,
adding a protocol for registering handles feels like adding complexity
with not much real benefit.

That said, the approach should work fine and if there is a use
case where it is solving real problems, then go for it.

If you want to test performance, some of the unit tests for
CommandShell will generate thousands of file handles, so that
might be a good smoke test. Web servers will presumably also
create and destroy handles as a high rate over long periods
of time.

Dave

p.s. I would certainly like to have a way to have OSProcess work
with the win32 HANDLE registry, so any solution that supports this
is very welcome :)

Reply | Threaded
Open this post in threaded view
|

Re: VM handle management

Bert Freudenberg
In reply to this post by Andreas.Raab

On 24.03.2010, at 04:10, Andreas Raab wrote:

>
>
> Here's my proposal. I say let's extend the VM with 4 new proxy functions, namely:
>
>  /* Registers a pointer and returns a handle.
>     The type argument ensures that the handle cannot be forged. */
>  OOP registerHandleType(void *ptr, char *type);
>
>  /* Unregister a handle */
>  void unregisterHandle(void *ptr);
>
>  /* Look up the pointer for a given handle/pair type */
>  void* lookupHandleType(OOP handle, char *type);
>
>  /* Ditto, just based on a stack index */
>  void* stackHandleValueType(int stackIndex, char *type);

Love it. Should have proposed something like this the third time *I* implemented another ad-hoc handle scheme ;)

- Bert -

Reply | Threaded
Open this post in threaded view
|

Re: VM handle management

Bert Freudenberg
In reply to this post by David T. Lewis

On 24.03.2010, at 13:40, David T. Lewis wrote:

>
>
> On Tue, Mar 23, 2010 at 08:10:18PM -0700, Andreas Raab wrote:
>>
>> Folks -
>>
>> [Warning: Long post. Please try to stay on topic and don't get into any
>> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come
>> up with these issues. This is a practical proposal that I'd like to
>> implement].
>>
>> For the third time this month I ran into the situation that I was
>> writing a plugin that would create an internal data structure
>> (containing pointers etc) and where I wanted to return a handle to this
>> structure and pass that back to the VM.
>>
>> Since I'm a lazy bastard, what I usually do is something like this in
>> the support code:
>>
>> int sqNewFooBar(void) {
>>  /* creates a new foobar and returns a handle */
>>  FooBar *newFoo = malloc(1, sizeof(FooBar));
>>
>>  /* return a handle, sort of */
>>  return (int) newFoo;
>> }
>
> This seems like solving the wrong problem. If the problem is
> that pointers cannot be treated as sqInt or OOP values, then
> the solution is simple: Don't Do That. Declaring the types
> properly (in slang where necessary) results in simpler slang
> and simpler support code overall. That is my actual experience
> after working through quite a few 64/32 bit issues, not just
> a theoretical statement.

No, that's not the problem. The problem is having to implement a handle registry over and over again, for many plugins. Whenever a plugin needs to store some dynamic state, you end up writing this again. I've seen this with Cairo surfaces, Freetype fonts, DBus connections, OpenGL contexts etc.

- Bert -


Reply | Threaded
Open this post in threaded view
|

Re: VM handle management

David T. Lewis
 
On Wed, Mar 24, 2010 at 03:31:39PM +0100, Bert Freudenberg wrote:

>
> On 24.03.2010, at 13:40, David T. Lewis wrote:
> >
> >
> > On Tue, Mar 23, 2010 at 08:10:18PM -0700, Andreas Raab wrote:
> >>
> >> Folks -
> >>
> >> [Warning: Long post. Please try to stay on topic and don't get into any
> >> SmartSyntaxInterpreterPlugin / FFI syntax digressions that always come
> >> up with these issues. This is a practical proposal that I'd like to
> >> implement].
> >>
> >> For the third time this month I ran into the situation that I was
> >> writing a plugin that would create an internal data structure
> >> (containing pointers etc) and where I wanted to return a handle to this
> >> structure and pass that back to the VM.
> >>
> >> Since I'm a lazy bastard, what I usually do is something like this in
> >> the support code:
> >>
> >> int sqNewFooBar(void) {
> >>  /* creates a new foobar and returns a handle */
> >>  FooBar *newFoo = malloc(1, sizeof(FooBar));
> >>
> >>  /* return a handle, sort of */
> >>  return (int) newFoo;
> >> }
> >
> > This seems like solving the wrong problem. If the problem is
> > that pointers cannot be treated as sqInt or OOP values, then
> > the solution is simple: Don't Do That. Declaring the types
> > properly (in slang where necessary) results in simpler slang
> > and simpler support code overall. That is my actual experience
> > after working through quite a few 64/32 bit issues, not just
> > a theoretical statement.
>
> No, that's not the problem. The problem is having to implement a handle
> registry over and over again, for many plugins. Whenever a plugin needs
> to store some dynamic state, you end up writing this again. I've seen
> this with Cairo surfaces, Freetype fonts, DBus connections, OpenGL
> contexts etc.

Ah, I see. That sounds fine then.

Dave