Transfer of binary data between images

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Transfer of binary data between images

Herbert König
Hi all,

I want to spread an algorithm over several images. For that I need to
transfer some objects.

For other purposes I already read and write objects via JSON files. To
change this to UDP is straightforward. But the conversion to JSON and
back takes considerable time, biggest chunk of data is an array with floats.

Is there a fast and easy way to transfer objects binary between images?
I can transfer only the array of floats (maybe a float array) if that
saves time.


Thanks,


Herbert




Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Alan Pinch
I think ASN1 inside of the Cryptography package may have some speed and
now supports Floats, ScaledDecimals and Fractions, I think. I would love
to hear your real world comparison to json, as I am starting to adapt
STON to my needs.


On 10/20/2017 06:18 AM, Herbert König wrote:

> Hi all,
>
> I want to spread an algorithm over several images. For that I need to
> transfer some objects.
>
> For other purposes I already read and write objects via JSON files. To
> change this to UDP is straightforward. But the conversion to JSON and
> back takes considerable time, biggest chunk of data is an array with
> floats.
>
> Is there a fast and easy way to transfer objects binary between
> images? I can transfer only the array of floats (maybe a float array)
> if that saves time.
>
>
> Thanks,
>
>
> Herbert
>
>
>
>

--
Thank you for your consideration,
Alan


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Edgar De Cleene
In reply to this post by Herbert König
Yes.
Serialized ReferenceStream .obj files (see attached)
You could transfer huge complex objects (complete 6000+ .html pages of
swiki) between Squeak and Cuis and with some extra work to Pharo too
Or you could convert your JSON to .obj with the attached .mcz and see if
more convenient tranfer JSON or .obj yourself



On 10/20/17, 07:18, "Herbert König" <[hidden email]> wrote:

> Hi all,

I want to spread an algorithm over several images. For that I need to
>
transfer some objects.

For other purposes I already read and write objects
> via JSON files. To
change this to UDP is straightforward. But the conversion
> to JSON and
back takes considerable time, biggest chunk of data is an array
> with floats.

Is there a fast and easy way to transfer objects binary between
> images?
I can transfer only the array of floats (maybe a float array) if that
>
saves time.


Thanks,


Herbert




ObjectCompatibleENH.1.cs (1K) Download Attachment
JSON-edc.mcz (21K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Kjell Godo
What about Fuel?


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

David T. Lewis
On Fri, Oct 20, 2017 at 06:33:28PM +0000, Kjell Godo wrote:
> What about Fuel?

I thought I had already suggested that, but I guess it was in a different discussion:

 http://lists.squeakfoundation.org/pipermail/squeak-dev/2017-October/date.html

Yes, Fuel is a good choice for serializing between images. ReferenceStream works well,
but some things do not work with non-positionable streams, whereas Fuel seems to do
a good job with any kind of stream (such as streaming on an OS pipe between two
images).

The Magma serializer is another candidate.

Dave


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Herbert König
In reply to this post by Herbert König
Thanks all,

first I'll try Edgar's way. Nothing new to load/learn and I used
ReferenceStream earlier.


Cheers,


Herbert


Am 20.10.2017 um 12:18 schrieb Herbert König:

> Hi all,
>
> I want to spread an algorithm over several images. For that I need to
> transfer some objects.
>
> For other purposes I already read and write objects via JSON files. To
> change this to UDP is straightforward. But the conversion to JSON and
> back takes considerable time, biggest chunk of data is an array with
> floats.
>
> Is there a fast and easy way to transfer objects binary between
> images? I can transfer only the array of floats (maybe a float array)
> if that saves time.
>
>
> Thanks,
>
>
> Herbert
>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Hannes Hirzel
And noteworthy in addition is that in case you do not have cycles in
your net of objects a DataStream is just fine.

http://wiki.squeak.org/squeak/4310

--Hannes

On 10/21/17, Herbert König <[hidden email]> wrote:

> Thanks all,
>
> first I'll try Edgar's way. Nothing new to load/learn and I used
> ReferenceStream earlier.
>
>
> Cheers,
>
>
> Herbert
>
>
> Am 20.10.2017 um 12:18 schrieb Herbert König:
>> Hi all,
>>
>> I want to spread an algorithm over several images. For that I need to
>> transfer some objects.
>>
>> For other purposes I already read and write objects via JSON files. To
>> change this to UDP is straightforward. But the conversion to JSON and
>> back takes considerable time, biggest chunk of data is an array with
>> floats.
>>
>> Is there a fast and easy way to transfer objects binary between
>> images? I can transfer only the array of floats (maybe a float array)
>> if that saves time.
>>
>>
>> Thanks,
>>
>>
>> Herbert
>>
>>
>>
>>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Herbert König
Hi Hannes,

Am Sat, 21 Oct 2017 15:05:19 +0200
schrieb "H. Hirzel" <[hidden email]>:

> And noteworthy in addition is that in case you do not have cycles in
> your net of objects a DataStream is just fine.
>
> http://wiki.squeak.org/squeak/4310

Which I don't have so maybe this is even faster because it doesn't
employ any logic to deal with cycles.

Thanks,

Herbert


>
> --Hannes
>
> On 10/21/17, Herbert König <[hidden email]> wrote:
> > Thanks all,
> >
> > first I'll try Edgar's way. Nothing new to load/learn and I used
> > ReferenceStream earlier.
> >
> >
> > Cheers,
> >
> >
> > Herbert
> >
> >
> > Am 20.10.2017 um 12:18 schrieb Herbert König:
> >> Hi all,
> >>
> >> I want to spread an algorithm over several images. For that I need
> >> to transfer some objects.
> >>
> >> For other purposes I already read and write objects via JSON
> >> files. To change this to UDP is straightforward. But the
> >> conversion to JSON and back takes considerable time, biggest chunk
> >> of data is an array with floats.
> >>
> >> Is there a fast and easy way to transfer objects binary between
> >> images? I can transfer only the array of floats (maybe a float
> >> array) if that saves time.
> >>
> >>
> >> Thanks,
> >>
> >>
> >> Herbert
> >>
> >>
> >>
> >>
> >
> >
> >
>


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Alan Pinch
There reason I suggested ASN1 is that it is crosee-platofrm, if that is
your requirement. I believe it is effectively binary, for your needs of
lots of Floats and I just added support for them too. If you need that...


On 10/21/2017 11:38 AM, Herbert König wrote:

> Hi Hannes,
>
> Am Sat, 21 Oct 2017 15:05:19 +0200
> schrieb "H. Hirzel" <[hidden email]>:
>
>> And noteworthy in addition is that in case you do not have cycles in
>> your net of objects a DataStream is just fine.
>>
>> http://wiki.squeak.org/squeak/4310
> Which I don't have so maybe this is even faster because it doesn't
> employ any logic to deal with cycles.
>
> Thanks,
>
> Herbert
>
>
>> --Hannes
>>
>> On 10/21/17, Herbert König <[hidden email]> wrote:
>>> Thanks all,
>>>
>>> first I'll try Edgar's way. Nothing new to load/learn and I used
>>> ReferenceStream earlier.
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Herbert
>>>
>>>
>>> Am 20.10.2017 um 12:18 schrieb Herbert König:
>>>> Hi all,
>>>>
>>>> I want to spread an algorithm over several images. For that I need
>>>> to transfer some objects.
>>>>
>>>> For other purposes I already read and write objects via JSON
>>>> files. To change this to UDP is straightforward. But the
>>>> conversion to JSON and back takes considerable time, biggest chunk
>>>> of data is an array with floats.
>>>>
>>>> Is there a fast and easy way to transfer objects binary between
>>>> images? I can transfer only the array of floats (maybe a float
>>>> array) if that saves time.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Herbert
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>

--
Thank you for your consideration,
Alan


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Kjell Godo
Is JSON the preferred way to transfer data between different 
     Smalltalk platforms?
     like Dolphin S8 visualWorks GnuSmalltalk Smalltalk/X Amber etc?
i would like a way for all the different things to talk to each other
     why should they all be seperate?

I am assuming JSON would be best for going between languages like 
     Smalltalk - Lisp
     ( it is the easiest to get access to everything has a Package for it ).

Are there any Packages that are designed to link different languages together?
     like Smalltalk - Racket - CommonLisp - SWIProlog etc?
     i am assuming that sockets + JSON are the easiest quickest way
     or DCom or what was that Linux d-Bus is d-Bus also on Windows?easy?
     is COM also on Linux?does anyone use it?hard to use? 
          Dolphin and VisialWork do COM
     is anyone doing this sort of thing?who would just know?


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

David T. Lewis
On Sat, Oct 21, 2017 at 04:33:21PM +0000, Kjell Godo wrote:

> Is JSON the preferred way to transfer data between different
>      Smalltalk platforms?
>      like Dolphin S8 visualWorks GnuSmalltalk Smalltalk/X Amber etc?
> i would like a way for all the different things to talk to each other
>      why should they all be seperate?
>
> I am assuming JSON would be best for going between languages like
>      Smalltalk - Lisp
>      ( it is the easiest to get access to everything has a Package for it ).
>
> Are there any Packages that are designed to link different languages
> together?
>      like Smalltalk - Racket - CommonLisp - SWIProlog etc?
>      i am assuming that sockets + JSON are the easiest quickest way
>      or DCom or what was that Linux d-Bus is d-Bus also on Windows?easy?
>      is COM also on Linux?does anyone use it?hard to use?
>           Dolphin and VisialWork do COM
>      is anyone doing this sort of thing?who would just know?

There is a SOAP implementation by Masashi Umezawa:

  http://wiki.squeak.org/squeak/1399

You can find it in the SqueakMap package loader (SoapCore, SoapCoreClient,
SoapCoreServer, SoapOpera).

I believe that the home page is here:

  http://www.mars.dti.ne.jp/~umejava/smalltalk/soapOpera/soapCore.html

I do not know if it still works in current Squeak, or if the package is
still being maintained. But I know that it used to work, so it might be
worth a look.

Dave


Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Herbert König
In reply to this post by Kjell Godo

Hi,

I used JSON with the addition of carrying the class (JSON being Javascript born has no concept of classes) because it requires only very little work after loading the JSON package. (And because I've been doing some web application where I needed JSON anyway.)

Each class needs on the class side:

constructFromJson: j
    ^ self new constructFromJson: j

and on the instance side:

jsonWriteOn: aStream
    Json renderInstanceVariables:
            (self class allInstVarNames collect: [:ea| ea asSymbol])
        of: self
        on: aStream

and:

constructFromJson: j
    j keysAndValuesDo: [:key :value|
        self instVarNamed: key put: value]

Simple, stupid and good enough for several of my applications.

This should work across dialects if the destination image has the required classes in the same shape. The @Classname designator is a Smalltalk specific extension to JSON. I think it is not good enough to transfer object trees with cyclic dependencies. Other languages with a different ideas of how a class is constructed it might need more work.

JSON excerpt:

[@IndividualGenom{"halfNum":77,"fitness":2.364011719201257,"fitnesses":null,"geneValues":[-0.7875619210225705,-1.1191086825115324,2.2987547782235476,-2.1996847933965746,3.3254889274350594,-1.7481778033261461 ......

Cheers,

Herbert

Am 21.10.2017 um 18:33 schrieb Kjell Godo:
Is JSON the preferred way to transfer data between different 
     Smalltalk platforms?
     like Dolphin S8 visualWorks GnuSmalltalk Smalltalk/X Amber etc?
i would like a way for all the different things to talk to each other
     why should they all be seperate?

I am assuming JSON would be best for going between languages like 
     Smalltalk - Lisp
     ( it is the easiest to get access to everything has a Package for it ).

Are there any Packages that are designed to link different languages together?
     like Smalltalk - Racket - CommonLisp - SWIProlog etc?
     i am assuming that sockets + JSON are the easiest quickest way
     or DCom or what was that Linux d-Bus is d-Bus also on Windows?easy?
     is COM also on Linux?does anyone use it?hard to use? 
          Dolphin and VisialWork do COM
     is anyone doing this sort of thing?who would just know?



    



Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

Herbert König
In reply to this post by Herbert König
Hi,

DataStream it is. On an Array of objects

JSON  write avg. 6 s  read avg.  1s    for 1500 KB

ReferenceStream write avg. 1.9 s read avg.  0.4 s  for 689 KB

DataStream write avg. 1.55 s read avg. 0.13s for 698 KB

Cheers,

Herbert

Am 21.10.2017 um 17:38 schrieb Herbert König:

> Hi Hannes,
>
> Am Sat, 21 Oct 2017 15:05:19 +0200
> schrieb "H. Hirzel" <[hidden email]>:
>
>> And noteworthy in addition is that in case you do not have cycles in
>> your net of objects a DataStream is just fine.
>>
>> http://wiki.squeak.org/squeak/4310
> Which I don't have so maybe this is even faster because it doesn't
> employ any logic to deal with cycles.
>
> Thanks,
>
> Herbert
>
>
>> --Hannes
>>
>> On 10/21/17, Herbert König <[hidden email]> wrote:
>>> Thanks all,
>>>
>>> first I'll try Edgar's way. Nothing new to load/learn and I used
>>> ReferenceStream earlier.
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Herbert
>>>
>>>
>>> Am 20.10.2017 um 12:18 schrieb Herbert König:
>>>> Hi all,
>>>>
>>>> I want to spread an algorithm over several images. For that I need
>>>> to transfer some objects.
>>>>
>>>> For other purposes I already read and write objects via JSON
>>>> files. To change this to UDP is straightforward. But the
>>>> conversion to JSON and back takes considerable time, biggest chunk
>>>> of data is an array with floats.
>>>>
>>>> Is there a fast and easy way to transfer objects binary between
>>>> images? I can transfer only the array of floats (maybe a float
>>>> array) if that saves time.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> Herbert
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>


bpi
Reply | Threaded
Open this post in threaded view
|

Re: Transfer of binary data between images

bpi
By the way, this would be a great library to port for serialization and interfacing to other programming languages:
https://avro.apache.org/docs/current/index.html

It gets quite a bit of praise in the book Designing Data-Intensive Applications (https://dataintensive.net) which I can highly recommend.

Cheers,
Bernhard

> Am 22.10.2017 um 17:11 schrieb Herbert König <[hidden email]>:
>
> Hi,
>
> DataStream it is. On an Array of objects
>
> JSON  write avg. 6 s  read avg.  1s    for 1500 KB
>
> ReferenceStream write avg. 1.9 s read avg.  0.4 s  for 689 KB
>
> DataStream write avg. 1.55 s read avg. 0.13s for 698 KB
>
> Cheers,
>
> Herbert
>
> Am 21.10.2017 um 17:38 schrieb Herbert König:
>> Hi Hannes,
>>
>> Am Sat, 21 Oct 2017 15:05:19 +0200
>> schrieb "H. Hirzel" <[hidden email]>:
>>
>>> And noteworthy in addition is that in case you do not have cycles in
>>> your net of objects a DataStream is just fine.
>>>
>>> http://wiki.squeak.org/squeak/4310
>> Which I don't have so maybe this is even faster because it doesn't
>> employ any logic to deal with cycles.
>>
>> Thanks,
>>
>> Herbert
>>
>>
>>> --Hannes
>>>
>>> On 10/21/17, Herbert König <[hidden email]> wrote:
>>>> Thanks all,
>>>>
>>>> first I'll try Edgar's way. Nothing new to load/learn and I used
>>>> ReferenceStream earlier.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>>
>>>> Herbert
>>>>
>>>>
>>>> Am 20.10.2017 um 12:18 schrieb Herbert König:
>>>>> Hi all,
>>>>>
>>>>> I want to spread an algorithm over several images. For that I need
>>>>> to transfer some objects.
>>>>>
>>>>> For other purposes I already read and write objects via JSON
>>>>> files. To change this to UDP is straightforward. But the
>>>>> conversion to JSON and back takes considerable time, biggest chunk
>>>>> of data is an array with floats.
>>>>>
>>>>> Is there a fast and easy way to transfer objects binary between
>>>>> images? I can transfer only the array of floats (maybe a float
>>>>> array) if that saves time.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>> Herbert
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>
>