[NB] NBExternalArray not working for complex types?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[NB] NBExternalArray not working for complex types?

Max Leske
Hi guys

I have a problem with NBExternalArray. AFAICT the generated assembly should be able to handle arbitrary elements in the array (as long as they are all of the same type). The operations required to read / write an element are given by the external type specified for the collection (e.g ‘char’ or ‘int’). All of this works wonderfully for char, int, NBExternalAddress etc. But I’m trying to get it working for strings. So here is the code that I would expect to work:

arrayClass := NBExternalArray anonymousSubclassInitElementType: 'String'.
array := arrayClass new: 1.
array at: 1 put: ‘foo'.
array at: 1. “—> should procude ‘foo’, but produces arbitrary strings”

I can make it work with NBExternalAddress of course but I don’t want to if I can avoid it.

I’d appreciate it if someone (Nicolai?) could take a look at NBExternalArray>>emitRead / emitWrite and see if there’s an easy solution. Especially for strings I don’t think it should be too hard since all that’s needed is already present in the string handling facilities.

Cheers,
Max
Reply | Threaded
Open this post in threaded view
|

Re: [NB] NBExternalArray not working for complex types?

Nicolai Hess
Hi Max

2015-04-26 19:29 GMT+02:00 Max Leske <[hidden email]>:
Hi guys

I have a problem with NBExternalArray. AFAICT the generated assembly should be able to handle arbitrary elements in the array (as long as they are all of the same type). The operations required to read / write an element are given by the external type specified for the collection (e.g ‘char’ or ‘int’). All of this works wonderfully for char, int, NBExternalAddress etc. But I’m trying to get it working for strings. So here is the code that I would expect to work:

arrayClass := NBExternalArray anonymousSubclassInitElementType: 'String'.
array := arrayClass new: 1.
array at: 1 put: ‘foo'.
array at: 1. “—> should procude ‘foo’, but produces arbitrary strings”

interesting

 

I can make it work with NBExternalAddress of course but I don’t want to if I can avoid it.

How would you use NBExternalAddress in this case?
 

I’d appreciate it if someone (Nicolai?) could take a look at NBExternalArray>>emitRead / emitWrite and see if there’s an easy solution. Especially for strings I don’t think it should be too hard since all that’s needed is already present in the string handling facilities.


Somehow it only works half the way, either write the String / or read the String.


this one:
arrayClass := NBExternalArray ofType: 'String'.
array := arrayClass new: 1.
array at: 1 put: 'foo'.
array at:1

reads the data back to a string object, but I think the data was written wrong.


this one:
arrayClass := NBExternalArray ofType: 'char*'.
array := arrayClass new: 1.
array at: 1 put: 'foo'.

at least writes the right data, but now array at:1 contains an external address and it is not automatically converted back to a string.
You can confirm that the data is written correctly by explicitly creating a string from the external address.
(array at:1) readString -> 'foo'

Now, why is the String data written wrong in the first case?
(Maybe this has something to do with #pointerArity. I think NBExternalString should have pointerArity 1.
I changed this for NBExternalString, but it seems this does not make a difference - and may break other things).
It looks like some "double conversion" ST-String -> char*, that is, it converts a String as NBExternalString
and uses this to do the conversion again:

I played it bit with the code (NB makes really fun, no irony, it really makes fun, it is a nice piece of work).
and created a new NBExternalArray subclass StringArray:

NBExternalArray subclass: #StringArray
    instanceVariableNames: ''
    classVariableNames: ''
    category: 'NativeBoost-Core-Objects'

define the class initialize method

initialize
    self initElementType: #String

and overwrite emitWrite, the same code as in NBExternalArray, but replace

        function: ' oop ( oop value , uint32 index , void * data , ' , self class elementType , ' value )'
with
        function: ' oop ( oop value , uint32 index , void * data , char* value )'

now, this works:
arrayClass := StringArray.
array := arrayClass new: 1.
array at: 1 put: 'foo'.
(array at:1) -> 'foo'

So, maybe the default function prototype in NBExternalArray>>#emitWrite is responsible for the double
conversion or the pointer arity.

I don't fully understand what is going on :),  but  maybe this helps.

nicolai


 

Cheers,
Max

Reply | Threaded
Open this post in threaded view
|

Re: [NB] NBExternalArray not working for complex types?

Henrik Sperre Johansen
In reply to this post by Max Leske

> On 26 Apr 2015, at 7:29 , Max Leske <[hidden email]> wrote:
>
> Hi guys
>
> I have a problem with NBExternalArray. AFAICT the generated assembly should be able to handle arbitrary elements in the array (as long as they are all of the same type). The operations required to read / write an element are given by the external type specified for the collection (e.g ‘char’ or ‘int’). All of this works wonderfully for char, int, NBExternalAddress etc. But I’m trying to get it working for strings. So here is the code that I would expect to work:
>
> arrayClass := NBExternalArray anonymousSubclassInitElementType: 'String'.
> array := arrayClass new: 1.
> array at: 1 put: ‘foo'.
> array at: 1. “—> should procude ‘foo’, but produces arbitrary strings”
>
> I can make it work with NBExternalAddress of course but I don’t want to if I can avoid it.
>
> I’d appreciate it if someone (Nicolai?) could take a look at NBExternalArray>>emitRead / emitWrite and see if there’s an easy solution. Especially for strings I don’t think it should be too hard since all that’s needed is already present in the string handling facilities.
>
> Cheers,
> Max

There's a solution, but I doubt you could call it simple.
NBExternalArray is meant to hold const sized objects, so is allocated with the proper amount of slots up front.
If you want it to hold pointers to variably sized objects (like strings), since we can't pin the objects, they need to be copied (well, would have to do that anyways for Strings, which need it'd need to do dynamic allocation (and freeing when appropriate), so emitWrite would become quite a bit more complicated, + necessary object finalization would have to occur.

You *can* work around it, by using an element type that is externally allocated, but it adds quite a bit of boilerplate, here's the workspace PoC code:

string := 'herru€'.
bytes := ZnUTF8Encoder new encodeString: string.
mem := NativeBoost allocate: bytes size + 1.
NativeBoost memCopy: bytes to: mem size: bytes size.

arrClass := NBExternalArray anonymousSubclassInitElementType: 'char *'.
arr := arrClass new: 1.
arr at: 1 put: mem.
ZnUTF8Encoder new decodeBytes: (arr at:1) readString asByteArray

Cheers,
Henry
Reply | Threaded
Open this post in threaded view
|

Re: [NB] NBExternalArray not working for complex types?

Max Leske
In reply to this post by Nicolai Hess

On 27 Apr 2015, at 10:42, Nicolai Hess <[hidden email]> wrote:

Hi Max

2015-04-26 19:29 GMT+02:00 Max Leske <[hidden email]>:
Hi guys

I have a problem with NBExternalArray. AFAICT the generated assembly should be able to handle arbitrary elements in the array (as long as they are all of the same type). The operations required to read / write an element are given by the external type specified for the collection (e.g ‘char’ or ‘int’). All of this works wonderfully for char, int, NBExternalAddress etc. But I’m trying to get it working for strings. So here is the code that I would expect to work:

arrayClass := NBExternalArray anonymousSubclassInitElementType: 'String'.
array := arrayClass new: 1.
array at: 1 put: ‘foo'.
array at: 1. “—> should procude ‘foo’, but produces arbitrary strings”

interesting

 

I can make it work with NBExternalAddress of course but I don’t want to if I can avoid it.

How would you use NBExternalAddress in this case?

By using 'char *’ or 'NBExternalAddress' as type and then converting back to a String by sending #readString to the address. My current solution acutally does that but it seems that the allocated memory is being freed sometimes without my doing, so I’m trying to let NBExternalAddress do as much as possible for me.

 

I’d appreciate it if someone (Nicolai?) could take a look at NBExternalArray>>emitRead / emitWrite and see if there’s an easy solution. Especially for strings I don’t think it should be too hard since all that’s needed is already present in the string handling facilities.


Somehow it only works half the way, either write the String / or read the String.

exactly…



this one:
arrayClass := NBExternalArray ofType: 'String'.
array := arrayClass new: 1.
array at: 1 put: 'foo'.
array at:1

reads the data back to a string object, but I think the data was written wrong.


this one:
arrayClass := NBExternalArray ofType: 'char*'.
array := arrayClass new: 1.
array at: 1 put: 'foo'.

at least writes the right data, but now array at:1 contains an external address and it is not automatically converted back to a string.
You can confirm that the data is written correctly by explicitly creating a string from the external address.
(array at:1) readString -> 'foo'

Now, why is the String data written wrong in the first case?
(Maybe this has something to do with #pointerArity. I think NBExternalString should have pointerArity 1.
I changed this for NBExternalString, but it seems this does not make a difference - and may break other things).
It looks like some "double conversion" ST-String -> char*, that is, it converts a String as NBExternalString
and uses this to do the conversion again:

I played it bit with the code (NB makes really fun, no irony, it really makes fun, it is a nice piece of work).
and created a new NBExternalArray subclass StringArray:

NBExternalArray subclass: #StringArray
    instanceVariableNames: ''
    classVariableNames: ''
    category: 'NativeBoost-Core-Objects'

define the class initialize method

initialize
    self initElementType: #String

and overwrite emitWrite, the same code as in NBExternalArray, but replace

        function: ' oop ( oop value , uint32 index , void * data , ' , self class elementType , ' value )'
with
        function: ' oop ( oop value , uint32 index , void * data , char* value )’

Smart! If that works conistently (I’ll test it) we should be able to find the problem.


now, this works:
arrayClass := StringArray.
array := arrayClass new: 1.
array at: 1 put: 'foo'.
(array at:1) -> 'foo'

So, maybe the default function prototype in NBExternalArray>>#emitWrite is responsible for the double
conversion or the pointer arity.

I don't fully understand what is going on :),  but  maybe this helps.

Thanks a lot Nicolai, I really appreciate your help.

Cheers,
Max


nicolai


 

Cheers,
Max


Reply | Threaded
Open this post in threaded view
|

Re: [NB] NBExternalArray not working for complex types?

Max Leske
In reply to this post by Henrik Sperre Johansen

> On 27 Apr 2015, at 16:14, Henrik Johansen <[hidden email]> wrote:
>
>
>> On 26 Apr 2015, at 7:29 , Max Leske <[hidden email]> wrote:
>>
>> Hi guys
>>
>> I have a problem with NBExternalArray. AFAICT the generated assembly should be able to handle arbitrary elements in the array (as long as they are all of the same type). The operations required to read / write an element are given by the external type specified for the collection (e.g ‘char’ or ‘int’). All of this works wonderfully for char, int, NBExternalAddress etc. But I’m trying to get it working for strings. So here is the code that I would expect to work:
>>
>> arrayClass := NBExternalArray anonymousSubclassInitElementType: 'String'.
>> array := arrayClass new: 1.
>> array at: 1 put: ‘foo'.
>> array at: 1. “—> should procude ‘foo’, but produces arbitrary strings”
>>
>> I can make it work with NBExternalAddress of course but I don’t want to if I can avoid it.
>>
>> I’d appreciate it if someone (Nicolai?) could take a look at NBExternalArray>>emitRead / emitWrite and see if there’s an easy solution. Especially for strings I don’t think it should be too hard since all that’s needed is already present in the string handling facilities.
>>
>> Cheers,
>> Max
>
> There's a solution, but I doubt you could call it simple.
> NBExternalArray is meant to hold const sized objects, so is allocated with the proper amount of slots up front.
> If you want it to hold pointers to variably sized objects (like strings), since we can't pin the objects, they need to be copied (well, would have to do that anyways for Strings, which need it'd need to do dynamic allocation (and freeing when appropriate), so emitWrite would become quite a bit more complicated, + necessary object finalization would have to occur.

Right. So space is allocated for the constant sized objects or the pointers but not for the variable sized objects… That explains what I’m seeing with the memory being used sometimes. I guess allocation and finalization can easily be done in a dedicated subclass, so I’ll try that.

Thanks a lot Henry!

Cheers,
Max

>
> You *can* work around it, by using an element type that is externally allocated, but it adds quite a bit of boilerplate, here's the workspace PoC code:
>
> string := 'herru€'.
> bytes := ZnUTF8Encoder new encodeString: string.
> mem := NativeBoost allocate: bytes size + 1.
> NativeBoost memCopy: bytes to: mem size: bytes size.
>
> arrClass := NBExternalArray anonymousSubclassInitElementType: 'char *'.
> arr := arrClass new: 1.
> arr at: 1 put: mem.
> ZnUTF8Encoder new decodeBytes: (arr at:1) readString asByteArray
>
> Cheers,
> Henry