NeoJSON Mappings

Hi,

I'm happily mapping my objects to JSON. But in order to reduce the payload I'm looking for a way to map my objects differently depending on whether it is the root object of the mapping or not.

Let's say I have two classes with the following instvars between parenthesis:
* ClassA(id, name)
* ClassB(id, classA)

I'm using #neoJsonMapping: to define the mappings and (NeoJsonWriter toString: classB) to get the JSON string.

If in the Class B mappings I reference classA (by instVar or accessor) I get the whole ClassA instance serialized as JSON. Which is not what I want. I'd like to have only the id of classA.

I know I can define a custom mapping using for:customDo: , but I'd like to know if there is another way to do that.

And this relates to another point, that is how to define a schema for a whole hierarchy of objects. All my objects are descendant of a common superclass.

There is no way I can do the following on different levels:
super neoJsonMapping: mapper.
mapper for: self do: [:mapping | mapping mapInstVars: #(foo) ]

Because on every call it overwrites the mapping for the schema of the current call, ending up mapping only the instVar 'foo'.

I can work my way around this by writing more, refactoring and so on. But I would like to know if Sven or somebody else who uses NeoJSON have some practice patterns that would like to share with the rest of us, mere mortals.

Best regards!


Esteban.

Additionally... it is really easy to f*ck up the image if you have circular references... :)

As fast as NeoJSON is, it also fills the memory really fast. From a regular 50MB to 512MB in 2 seconds :).

Regards!

Hello Esteban,

On 18 Mar 2013, at 00:48, Esteban A. Maringolo <[hidden email]> wrote:

Thanks for using NeoJSON and for giving feedback.

NeoJSON is actually two things: a standard JSON parser and encoder using roughly Dictionary for maps and some regular Collection for lists.

The mapping layer allows for an efficient conversion of JSON directly to and from Smalltalk objects, bypassing an intermediate Dictionary/Array representation.

There is however a problem: the basic JSON spec is deliberately limited and vague about certain aspects, and even non existing for other aspects such as typing, references or cycles. There exist numerous ways to add those to JSON, some very complicated ones, often coming from statically typed languages. Many of those kill the human readability/writability of JSON. The problem of how to encode a type or class in JSON across languages is actually unsolved. That is why I decided not to implement any support for that.

With NeoJSON mappings your are supposed to know upfront what types your JSON represents/contains. This works fine in many cases but certainly not all.

Circular references will always be a problem.

Structure sharing is extra complicated, it requires both reader and writer support with an objet table.

STON, which is very similar but incompatible with JSON, has support for all these aspects, right out of the box, no mapping required. If you do not have to interact with foreign systems, that is an option.

Support for one of the many semantics built on top of JSON will have to be built by hand. Hopefully the mapping layer can help there. But like you said: you'll have to play with custom mappings. The mapping system is pretty simple to understand.

I know this is much help, but I hope it makes things a bit clearer.

Regards,

Sven

PS: I also have to confess that although I am using NeoJSON in various important projects myself, we usually stick with Dictionaries/Collections without doing any mapping. Only in very specific cases do we actually convert to specific Smalltalk objects.

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

Hi Sven,

2013/3/18 Sven Van Caekenberghe-2 [via Smalltalk]
> There is however a problem: the basic JSON spec is deliberately limited
> and vague about certain aspects, and even non existing for other aspects
> such as typing, references or cycles. There exist numerous ways to add those
> to JSON, some very complicated ones, often coming from statically typed
> languages. Many of those kill the human readability/writability of JSON. The
> problem of how to encode a type or class in JSON across languages is
> actually unsolved. That is why I decided not to implement any support for
> that.

> With NeoJSON mappings your are supposed to know upfront what types your
> JSON represents/contains. This works fine in many cases but certainly not
> all.

In this case I know it. And I Just need the writter and not the
reader, altough it might be useful in the future.

> Circular references will always be a problem.
> Structure sharing is extra complicated, it requires both reader and writer
> support with an objet table.

Yes, this happens with any serializer without global state or object table :)

> STON, which is very similar but incompatible with JSON, has support for
> all these aspects, right out of the box, no mapping required. If you do not
> have to interact with foreign systems, that is an option.

In this case JSON is mandatory. The consumer of the data is a mobile device.
The other alternative is, god forbid, XML, which isn't friendly in
terms of bandwidth.

> Support for one of the many semantics built on top of JSON will have to be
> built by hand. Hopefully the mapping layer can help there. But like you
> said: you'll have to play with custom mappings. The mapping system is pretty
> simple to understand.

Yes, but sometimes I feel the need to be able to map instVars/accesors
to encoder/decoders without having to use an schema (defined by a
symbol). I haven't found the case where it is not an extra step.

> PS: I also have to confess that although I am using NeoJSON in various
> important projects myself, we usually stick with Dictionaries/Collections
> without doing any mapping. Only in very specific cases do we actually
> convert to specific Smalltalk objects.

Same happens here so far, I produce JSON, and get JSON back, but then
I manually pick the JSON values (deserialized as Dictionaries/Arrays)
and assign them to my objects.

I don't know what the guys from SmalltalkHub are using to
serialize/deserialize JSON.

Thanks for your support. I'll keep with using custom mappings for
almost everything that references an object which isn't "simple"
(string, dates, numbers, etc.).

Best regards,

--
Esteban.

Hi Sven,

2013/3/18 Sven Van Caekenberghe-2 [via Smalltalk]
> There is however a problem: the basic JSON spec is deliberately limited
> and vague about certain aspects, and even non existing for other aspects
> such as typing, references or cycles. There exist numerous ways to add those
> to JSON, some very complicated ones, often coming from statically typed
> languages. Many of those kill the human readability/writability of JSON. The
> problem of how to encode a type or class in JSON across languages is
> actually unsolved. That is why I decided not to implement any support for
> that.

> With NeoJSON mappings your are supposed to know upfront what types your
> JSON represents/contains. This works fine in many cases but certainly not
> all.

In this case I know it. And I Just need the writter and not the
reader, altough it might be useful in the future.

> Circular references will always be a problem.
> Structure sharing is extra complicated, it requires both reader and writer
> support with an objet table.

Yes, this happens with any serializer without global state or object table :)

> STON, which is very similar but incompatible with JSON, has support for
> all these aspects, right out of the box, no mapping required. If you do not
> have to interact with foreign systems, that is an option.

In this case JSON is mandatory. The consumer of the data is a mobile device.
The other alternative is, god forbid, XML, which isn't friendly in
terms of bandwidth.

> Support for one of the many semantics built on top of JSON will have to be
> built by hand. Hopefully the mapping layer can help there. But like you
> said: you'll have to play with custom mappings. The mapping system is pretty
> simple to understand.

Yes, but sometimes I feel the need to be able to map instVars/accesors
to encoder/decoders without having to use an schema (defined by a
symbol). I haven't found the case where it is not an extra step.

> PS: I also have to confess that although I am using NeoJSON in various
> important projects myself, we usually stick with Dictionaries/Collections
> without doing any mapping. Only in very specific cases do we actually
> convert to specific Smalltalk objects.

Same happens here so far, I produce JSON, and get JSON back, but then
I manually pick the JSON values (deserialized as Dictionaries/Arrays)
and assign them to my objects.

I don't know what the guys from SmalltalkHub are using to
serialize/deserialize JSON.

Thanks for your support. I'll keep with using custom mappings for
almost everything that references an object which isn't "simple"
(string, dates, numbers, etc.).

Best regards,

--
Esteban.

On 18 Mar 2013, at 19:46, Sven Van Caekenberghe <[hidden email]> wrote:

> NeoJSON is actually two things: a standard JSON parser and encoder using roughly Dictionary for maps and some regular Collection for lists.

That sentence is incomplete. Make that:

NeoJSON is actually two things: first, a standard JSON parser and encoder using roughly Dictionary for maps and some regular Collection for lists, and second a mapping system on top of that.

Additionally, while writing, the key message sent to each object is #neoJsonOn: neoJSONWriter - you could also overwrite that, check the implementors.

For reading, the key message is #readFrom: neoJSONReader sent to mappings (see #nextAs:).

But I guess you already know all this.

Sven


--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

Yes, I find out that.
But I wanted to have the JSON serialization to be context-dependant.

And that's what I achieved by means of the custom mappings.

Again, kudos for making it so fast and memory-savvy, being stream based with no internal transformations makes it almost a basic printOn: aStream.

One question that remains is... how to map the same accesor/instVar to more than one attribute.
eg.
neoJsonMapping: mapper

  mapper for: self do: [:mapping |
    mapping mapAccessor: #reference. "full"
    (mapping mapAccessor: #reference to: 'reference-id) valueSchema: #ReferenceId "just its ID"
  ].

   mapper for: #ReferenceId customDo: [:mapping |
      mapper encoder: [:obj | obj id printString ]
      "..."
   ]

If I do that, I end up having the full reference twice.


Regards!

On 19 Mar 2013, at 13:14, "Esteban A. Maringolo" <[hidden email]> wrote:

> Yes, I find out that.
> But I wanted to have the JSON serialization to be context-dependant.
>
> And that's what I achieved by means of the custom mappings.
>
> Again, kudos for making it so fast and memory-savvy, being stream based with
> no internal transformations makes it almost a basic printOn: aStream.

Thanks, that was indeed the idea.

I think I understand what you want, and I know that you tried to explain the model (or a minimal version of it), but if you are writing out JSON according to some externally defined format (with the ideas and sharing), would it be possible to give me the specification of that format ? Maybe off list ?

In any case, I will think about the issue you raised.

Sven

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

Hi Esteban,

I have been thinking about you ;-)

On 19 Mar 2013, at 15:26, Sven Van Caekenberghe <[hidden email]> wrote:

> I think I understand what you want, and I know that you tried to explain the model (or a minimal version of it), but if you are writing out JSON according to some externally defined format (with the ideas and sharing), would it be possible to give me the specification of that format ? Maybe off list ?

Without knowing exactly what your target is, it is hard for me, below is how I understand your problem and how I would solve it.

> In any case, I will think about the issue you raised.

There are book objects, with id, title and author. The first time you write them out, you write all the details. If you encounter the same book again in the same encoding session, you only write out the id.

To do this, you need a custom writer subclass that can keep track of books seen, in the scope of the encoding session (incidentally, this is how you would do a writer that knows when it goes into cycles - but again, this has all been done in STON).

Furthermore, to dynamically decide the encoding you have to overwrite #neoJsonOn: This is done here by delegating to the custom writer. There the decision is made to take the normal route (super) or to just write the id. The class side of book still have a normal mapping.

Of course, a production quality implementation will be some more work.

I hope that this is more or less what you were looking for.

Regards,

Sven

----

TestCase subclass: #NeoJSONBookTests
        instanceVariableNames: ''
        classVariableNames: ''
        poolDictionaries: ''
        category: 'Neo-JSON-Example'!

!NeoJSONBookTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:52'!
testMultiple
        | book1 book2 |
        (book1 := NeoJSONBook new)
                id: 'B101';
                title: 'The Hobbit';
                author: 'JJR Tolkien'.
        (book2 := NeoJSONBook new)
                id: 'B102';
                title: 'Lord Of The Rings';
                author: 'JJR Tolkien'.
        NeoJSONCustomWriter toString: { book1. book1. book2. book1. book2 }! !

!NeoJSONBookTests methodsFor: 'testing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:44'!
testOne
        | book |
        (book := NeoJSONBook new)
                id: 'B101';
                title: 'The Hobbit';
                author: 'JJR Tolkien'.
        NeoJSONCustomWriter toString: book.! !

Object subclass: #NeoJSONBook
        instanceVariableNames: 'id title author'
        classVariableNames: ''
        poolDictionaries: ''
        category: 'Neo-JSON-Example'!

!NeoJSONBook methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
author
        ^ author! !

!NeoJSONBook methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
author: anObject
        author := anObject! !

!NeoJSONBook methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
id
        ^ id! !

!NeoJSONBook methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
id: anObject
        id := anObject! !

!NeoJSONBook methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:39'!
neoJsonOn: neoJSONWriter
        neoJSONWriter writeBook: self! !

!NeoJSONBook methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
title
        ^ title! !

!NeoJSONBook methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
title: anObject
        title := anObject! !

"-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- "!

NeoJSONBook class
        instanceVariableNames: ''!

!NeoJSONBook class methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:41'!
neoJsonMapping: mapper
        mapper mapAllInstVarsFor: self! !

NeoJSONWriter subclass: #NeoJSONCustomWriter
        instanceVariableNames: 'books'
        classVariableNames: ''
        poolDictionaries: ''
        category: 'Neo-JSON-Example'!

!NeoJSONCustomWriter methodsFor: 'initialize-release' stamp: 'SvenVanCaekenberghe 3/24/2013 17:52'!
initialize
        super initialize.
        books := Set new! !


!NeoJSONCustomWriter methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
books
        ^ books! !

!NeoJSONCustomWriter methodsFor: 'accessing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:35'!
books: anObject
        books := anObject! !


!NeoJSONCustomWriter methodsFor: 'writing' stamp: 'SvenVanCaekenberghe 3/24/2013 17:53'!
writeBook: aBook
        (self books includes: aBook id)
                ifTrue: [
                        self writeMapStreamingDo: [ :jsonMapWriter |
                                jsonMapWriter writeKey: 'id' value: aBook id ] ]
                ifFalse: [
                        books add: aBook id.
                        super writeObject: aBook ]! !

----




--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

Hi Sven,

You think about me and I'm not even kind enought to give you a prompt response. :)
Please apologize, I wasn't getting the mails from the pharo-users list.

What you propose isn't exactly what I need, but it was a good excercise on how to build a simple serializer just by subclassing the mapper. :)

I was needing to have both the full object serialization AND its id. So far I'm able to do it.

I'll be full of questions.
But until now I'm happily using your Websockets and NeoJSON. :)