Smalltalk › Pharo › Pharo Smalltalk Users

Serializing to/from Json

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

17 messages Options

larrry

Serializing to/from Json

Hi,

I'm hoping to use CouchDB for storage on a seaside application. It would be lovely if I didn't have to serialize/de-serialize by hand to json for all my persistent objects, and could use instead something like a smalltalk version of Google's Gson library. Is there anything like that?

I've tried the JSON library bundled with SCouchDB and it seems to automatically handle simple objects (eg, where the inst vars reference Strings or Numbers), but not handle more complex objects. Also, I haven't seen anything that de-serializes from json.

Thanks much!

Larry

fstephany

Re: Serializing to/from Json

Hi Larry,

You can maybe try with Magritte ?

http://forum.world.st/Magritte-gt-Json-td3727820.html should be a good
starting point :)

On 29/08/11 21:27, Larry White wrote:

> Hi,
>
> I'm hoping to use CouchDB for storage on a seaside application. It would
> be lovely if I didn't have to serialize/de-serialize by hand to json for
> all my persistent objects, and could use instead something like a
> smalltalk version of Google's Gson library. Is there anything like that?
>
> I've tried the JSON library bundled with SCouchDB and it seems to
> automatically handle simple objects (eg, where the inst vars reference
> Strings or Numbers), but not handle more complex objects. Also, I
> haven't seen anything that de-serializes from json.
>
> Thanks much!
>
> Larry

Marcus Denker-4

Re: Serializing to/from Json

In reply to this post by larrry

On Aug 29, 2011, at 9:58 PM, Francois Stephany wrote:

>
> On 29/08/11 21:27, Larry White wrote:
>> Hi,
>>
>> I'm hoping to use CouchDB for storage on a seaside application. It would
>> be lovely if I didn't have to serialize/de-serialize by hand to json for
>> all my persistent objects, and could use instead something like a
>> smalltalk version of Google's Gson library. Is there anything like that?
>>
>> I've tried the JSON library bundled with SCouchDB and it seems to
>> automatically handle simple objects (eg, where the inst vars reference
>> Strings or Numbers), but not handle more complex objects. Also, I
>> haven't seen anything that de-serializes from json.

Does it need to be JSON?

There is a new binary serializer:

http://rmod.lille.inria.fr/web/pier/software/Fuel

Marcus

--
Marcus Denker -- http://marcusdenker.de

drush66

Re: Serializing to/from Json

On Tue, Aug 30, 2011 at 10:23 AM, Marcus Denker <[hidden email]> wrote:
>
> Does it need to be JSON?

Since CouchDB stores data in json - I guess the answer is very likely
yes, since just storing serialized blobs inside CouchDB would loose
all querying and indexing available. In that case some key-value store
might be a better choice than CouchDB

Davorin Rusevljan
http://www.cloud208.com/

http://www.cloud208.com/

Sven Van Caekenberghe

Re: Serializing to/from Json

In reply to this post by larrry

Hi Larry,

On 29 Aug 2011, at 21:27, Larry White wrote:

> I've tried the JSON library bundled with SCouchDB and it seems to automatically handle simple objects (eg, where the inst vars reference Strings or Numbers), but not handle more complex objects. Also, I haven't seen anything that de-serializes from json.

There is a JSON encoder/decoder in Seaside, it is pretty standalone, see this recent thread:

http://forum.world.st/Standalone-JSON-Package-tp3759705p3759705.html

This one works very well (I am using it all the time), but only handles standard JSON types (follows the spec).

Now, JSON objects don't hold their type or class as Smalltalk (or Java objects) do, they are just Dictionaries. There is no automagic solution to this, apart from encoding the type/class as a property, but will most probably not be portable across languages (java.lang.Integer for Java, Integer for Smalltalk), one of the strengths of JSON.

So you will have to write code to convert your Person (or whatever) objects to/from Dictionaries and use JSON.

Sven

Bernat Romagosa

Re: Serializing to/from Json

Without having tried them, I see there are two CouchDB projects in squeaksource, they may handle your needs:

http://www.squeaksource.com/SCouchDB.html

http://www.squeaksource.com/CouchDB.html

Cheers,

2011/8/30 Sven Van Caekenberghe <[hidden email]>

Hi Larry,

On 29 Aug 2011, at 21:27, Larry White wrote:

> I've tried the JSON library bundled with SCouchDB and it seems to automatically handle simple objects (eg, where the inst vars reference Strings or Numbers), but not handle more complex objects. Also, I haven't seen anything that de-serializes from json.

There is a JSON encoder/decoder in Seaside, it is pretty standalone, see this recent thread:

http://forum.world.st/Standalone-JSON-Package-tp3759705p3759705.html

This one works very well (I am using it all the time), but only handles standard JSON types (follows the spec).

Now, JSON objects don't hold their type or class as Smalltalk (or Java objects) do, they are just Dictionaries. There is no automagic solution to this, apart from encoding the type/class as a property, but will most probably not be portable across languages (java.lang.Integer for Java, Integer for Smalltalk), one of the strengths of JSON.

So you will have to write code to convert your Person (or whatever) objects to/from Dictionaries and use JSON.

Sven

--
Bernat Romagosa.

larrry

Re: Serializing to/from Json

Wow. Thanks for all the replies.

@Bernat. I am using the SCouchDB library, but will take a look at the other one. Thanks.

@Sven. I'll also take a look at that Json library. Thank you.
@Markus. It has to Json, for exactly the reason that @Davorin suggests.

@Francois. I'm looking at Magritte as well, but it seems a bit overwhelming for a Smalltalk noob. I may give it a try on my next project.

Apologies for the naivete of the original question. It wasn't 'til I tried writing my own that I realized (doh) that you can't write something like Gson in Java without augmenting the model with type information. I assume that's how Magritte does it, since it seems to be all about meta data.

On Tue, Aug 30, 2011 at 5:30 AM, Bernat Romagosa <[hidden email]> wrote:

Without having tried them, I see there are two CouchDB projects in squeaksource, they may handle your needs:

http://www.squeaksource.com/SCouchDB.html

http://www.squeaksource.com/CouchDB.html

Cheers,

2011/8/30 Sven Van Caekenberghe <[hidden email]>

Hi Larry,

On 29 Aug 2011, at 21:27, Larry White wrote:

> I've tried the JSON library bundled with SCouchDB and it seems to automatically handle simple objects (eg, where the inst vars reference Strings or Numbers), but not handle more complex objects. Also, I haven't seen anything that de-serializes from json.

There is a JSON encoder/decoder in Seaside, it is pretty standalone, see this recent thread:

http://forum.world.st/Standalone-JSON-Package-tp3759705p3759705.html

This one works very well (I am using it all the time), but only handles standard JSON types (follows the spec).

Now, JSON objects don't hold their type or class as Smalltalk (or Java objects) do, they are just Dictionaries. There is no automagic solution to this, apart from encoding the type/class as a property, but will most probably not be portable across languages (java.lang.Integer for Java, Integer for Smalltalk), one of the strengths of JSON.

So you will have to write code to convert your Person (or whatever) objects to/from Dictionaries and use JSON.

Sven

--
Bernat Romagosa.

Mariano Martinez Peck

Re: Serializing to/from Json

In reply to this post by drush66

On Tue, Aug 30, 2011 at 9:40 AM, Davorin Rusevljan <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 10:23 AM, Marcus Denker <[hidden email]> wrote:
>
> Does it need to be JSON?

Since CouchDB stores data in json - I guess the answer is very likely
yes, since just storing serialized blobs inside CouchDB would loose
all querying and indexing available.

I don't understand. What is the difference from the querying and indexing point of view between serializing with JSON or a binary serializer.

--
Mariano
http://marianopeck.wordpress.com

Mariano Martinez Peck

Re: Serializing to/from Json

BTW....I did experiments between Fuel and Riak, maybe that helps in something: http://marianopeck.wordpress.com/2011/08/16/when-fuel-meets-riak-nosql-database/

On Tue, Aug 30, 2011 at 3:27 PM, Mariano Martinez Peck <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 9:40 AM, Davorin Rusevljan <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 10:23 AM, Marcus Denker <[hidden email]> wrote:
>
> Does it need to be JSON?

Since CouchDB stores data in json - I guess the answer is very likely
yes, since just storing serialized blobs inside CouchDB would loose
all querying and indexing available.

I don't understand. What is the difference from the querying and indexing point of view between serializing with JSON or a binary serializer.

--
Mariano
http://marianopeck.wordpress.com

--
Mariano
http://marianopeck.wordpress.com

Sven Van Caekenberghe

Re: Serializing to/from Json

In reply to this post by Mariano Martinez Peck

On 30 Aug 2011, at 16:27, Mariano Martinez Peck wrote:

> I don't understand. What is the difference from the querying and indexing point of view between serializing with JSON or a binary serializer.

It could be (not knowing the specific database) that the db understands the contents and can query on the properties, which would make huge differences in performance.

Say you store:

1 -> { "id" : 1, "name" : "one" }
2 -> { "id" : 2, "name" : "two" }
…
99999 -> …

You could then ask the db for the object where name = "one hundred" without reading all of them (but the db would have to do that of course). Maybe even indexes are possible.

Sven

larrry

Re: Serializing to/from Json

In reply to this post by Mariano Martinez Peck

Couch stores data as Json. Its query capabilities are based on json pair names.

I'm just trying it out but I believe that, for example, if a User object was a blob, you couldn't issue a query like "fetch user with username = 'foo'". You would have to read all the users into Smalltalk and do a select:

Anyway that's my impression of how Couch works.

On Tue, Aug 30, 2011 at 10:27 AM, Mariano Martinez Peck <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 9:40 AM, Davorin Rusevljan <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 10:23 AM, Marcus Denker <[hidden email]> wrote:
>
> Does it need to be JSON?

Since CouchDB stores data in json - I guess the answer is very likely
yes, since just storing serialized blobs inside CouchDB would loose
all querying and indexing available.

I don't understand. What is the difference from the querying and indexing point of view between serializing with JSON or a binary serializer.

--
Mariano
http://marianopeck.wordpress.com

drush66

Re: Serializing to/from Json

In reply to this post by Mariano Martinez Peck

On Tue, Aug 30, 2011 at 4:27 PM, Mariano Martinez Peck
<[hidden email]> wrote:
> I don't understand. What is the difference from the querying and indexing
> point of view between serializing with JSON or a binary serializer.

let's say you have a class:

Object subclass: #MyClass
instanceVariableNames: 'a b'
classVariableNames: ''
poolDictionaries: ''
category: 'Example'

and you serialize it into the json to something like

{
'_id': 'lkjjfdjdjdj',
'a': 'value of a',
'b': 'value of b'
}

than you can ask CouchDB to fetch you all object where for instance a is 'Aha!'

while if you serialize it as:

{
'_id': 'lkjjfdjdjdj',
'fuel_blob': 'AAXsdfkljijS3SM .....'
}

CouchDB has no idea how to look into the a, and select appropriate objects.

Davorin Rusevljan
http://www.cloud208.com/

http://www.cloud208.com/

Mariano Martinez Peck

Re: Serializing to/from Json

On Tue, Aug 30, 2011 at 3:40 PM, Davorin Rusevljan <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 4:27 PM, Mariano Martinez Peck
<[hidden email]> wrote:
> I don't understand. What is the difference from the querying and indexing
> point of view between serializing with JSON or a binary serializer.

let's say you have a class:

Object subclass: #MyClass
instanceVariableNames: 'a b'
classVariableNames: ''
poolDictionaries: ''
category: 'Example'

and you serialize it into the json to something like

{
'_id': 'lkjjfdjdjdj',
'a': 'value of a',
'b': 'value of b'
}

than you can ask CouchDB to fetch you all object where for instance a is 'Aha!'

while if you serialize it as:

{
'_id': 'lkjjfdjdjdj',
'fuel_blob': 'AAXsdfkljijS3SM .....'
}

CouchDB has no idea how to look into the a, and select appropriate objects.

Interesting. Now I understand. So I guess the same happens with all the rest of NoSQL databases? I am right no assume that if I want to be able to do queries without bringing the whole graph into memory, I need to store the data in JSON ?

thanks!

--
Mariano
http://marianopeck.wordpress.com

drush66

Re: Serializing to/from Json

On Tue, Aug 30, 2011 at 4:43 PM, Mariano Martinez Peck
<[hidden email]> wrote:
>
> Interesting. Now I understand. So I guess the same happens with all the rest
> of NoSQL databases? I am right no assume that if I want to be able to do
> queries without bringing the whole graph into memory, I need to store the
> data in JSON ?

Not necessarily, CouchDB and MongoDB are based on the json, but some
are more close to pure key value stores, where value is usually just
some blob and db does not care much about. In combination with such
databases Fuel is a great fit. I _think_ TokyTyran is such db, and
memcached.

Davorin

http://www.cloud208.com/

Jan van de Sandt

Re: Serializing to/from Json

In reply to this post by Mariano Martinez Peck

Well, you can also store binary contents in CouchDB [1]. You can attach zero or more binary attachments to a Json document in CouchDB.

You could use this in combination with Fuel. Add the properties you want to search for to the Json document and leave all the other data in binary format.

Jan.

[1] http://guide.couchdb.org/editions/1/en/api.html#attachments

On Tue, Aug 30, 2011 at 4:43 PM, Mariano Martinez Peck <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 3:40 PM, Davorin Rusevljan <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 4:27 PM, Mariano Martinez Peck
<[hidden email]> wrote:
> I don't understand. What is the difference from the querying and indexing
> point of view between serializing with JSON or a binary serializer.

let's say you have a class:

Object subclass: #MyClass
instanceVariableNames: 'a b'
classVariableNames: ''
poolDictionaries: ''
category: 'Example'

and you serialize it into the json to something like

{
'_id': 'lkjjfdjdjdj',
'a': 'value of a',
'b': 'value of b'
}

than you can ask CouchDB to fetch you all object where for instance a is 'Aha!'

while if you serialize it as:

{
'_id': 'lkjjfdjdjdj',
'fuel_blob': 'AAXsdfkljijS3SM .....'
}

CouchDB has no idea how to look into the a, and select appropriate objects.

Interesting. Now I understand. So I guess the same happens with all the rest of NoSQL databases? I am right no assume that if I want to be able to do queries without bringing the whole graph into memory, I need to store the data in JSON ?

thanks!

--
Mariano
http://marianopeck.wordpress.com

drush66

Re: Serializing to/from Json

In reply to this post by Mariano Martinez Peck

On Tue, Aug 30, 2011 at 4:43 PM, Mariano Martinez Peck
<[hidden email]> wrote:
>I am right no assume that if I want to be able to do
> queries without bringing the whole graph into memory, I need to store the
> data in JSON ?

and some other issues might be connected. For instance, I think you
could persuade CouchDB to do replication of only some records based on
the json data.

Davorin

http://www.cloud208.com/

Mariano Martinez Peck

Re: Serializing to/from Json

Ok. I see. Thanks a lot guys for the info, I am learning a lot :)

On Tue, Aug 30, 2011 at 3:54 PM, Davorin Rusevljan <[hidden email]> wrote:

On Tue, Aug 30, 2011 at 4:43 PM, Mariano Martinez Peck
<[hidden email]> wrote:
>I am right no assume that if I want to be able to do
> queries without bringing the whole graph into memory, I need to store the
> data in JSON ?

and some other issues might be connected. For instance, I think you
could persuade CouchDB to do replication of only some records based on
the json data.

Davorin

--
Mariano
http://marianopeck.wordpress.com