Smalltalk › Pharo › Pharo Smalltalk Developers

NeoJSON: parsing mapped nil fields

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

5 messages Options

Skip Lentz-2

NeoJSON: parsing mapped nil fields

Hi all,

When parsing with NeoJSON, I’m experiencing some problems.

If I parse a JSON representation using NeoJSONReader>>next (i.e. not according to a schema), it parses nil fields just fine.

But when I explicitly set a value schema, it won’t parse it in the case it is nil (or “null” in JSON). Is there a way to let it return nil in the case the parser encounters “null”, and otherwise serialize it as a value schema (e.g. a Point class).

Taking the example from the paper (https://github.com/svenvc/docs/blob/master/neo/neo-json-paper.md):

(NeoJSONReader on: ’null' readStream)
    mapInstVarsFor: Point;
    nextAs: Point.

Is there a way to adapt the above snippet so that it evaluates to ‘nil’ instead of resulting in an error? I ask because I am parsing an API response which sometimes contains a “null” field, and sometimes a JSON object for that field.

Thanks.

Sven Van Caekenberghe-2

Re: [Pharo-users] NeoJSON: parsing mapped nil fields

Hi,

> On 22 Dec 2015, at 17:40, Skip Lentz <[hidden email]> wrote:
>
> Hi all,
>
> When parsing with NeoJSON, I’m experiencing some problems.
>
> If I parse a JSON representation using NeoJSONReader>>next (i.e. not according to a schema), it parses nil fields just fine.
> But when I explicitly set a value schema, it won’t parse it in the case it is nil (or “null” in JSON). Is there a way to let it return nil in the case the parser encounters “null”, and otherwise serialize it as a value schema (e.g. a Point class).
>
> Taking the example from the paper (https://github.com/svenvc/docs/blob/master/neo/neo-json-paper.md):
> (NeoJSONReader on: ’null' readStream)
> mapInstVarsFor: Point;
> nextAs: Point.
>
> Is there a way to adapt the above snippet so that it evaluates to ‘nil’ instead of resulting in an error? I ask because I am parsing an API response which sometimes contains a “null” field, and sometimes a JSON object for that field.
>
> Thanks.

NeoJSON does have a bit of a split personality.

When using #next it parses whatever is there and returns it as either primitives, a map or a list, nested to any depth.

When you use mappings and #nextAs: it tries to map a static type structure on what is coming in. As such it is not really prepared for dynamic surprises like objects being replaced by null. Do also note that the typing info is totally absent from the JSON itself, unlike STON for example. NeoJSON also tries (and succeeds) in maintaining a stream based parser that does not generate intermediate structures except your own domain models.

But your request does make sense and I will think about it. It might be possible. It has been a while that I really thought about the implementation.

Sven

Skip Lentz-2

Re: [Pharo-users] NeoJSON: parsing mapped nil fields

Hi, thanks for replying! I can share some ideas.

> On Dec 23, 2015, at 12:09 AM, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Hi,
>
> NeoJSON does have a bit of a split personality.

Yes, this is what I noticed. Apart from this, I think it’s an awesome library, and I thank you a lot for making it.
It might be good to not make a distinction between #nextAs: and #next, and only rely on #nextAs:. Then #next might be rewritten as such:

NeoJSONReader>>next

^ self nextAs: self mapClass

Don’t know if that makes sense or not. It might alleviate this split personality, as you put it.

> When you use mappings and #nextAs: it tries to map a static type structure on what is coming in. As such it is not really prepared for dynamic surprises like objects being replaced by null.

Yes, in the general case, once you’ve sent #nextAs: to the reader with your value schema, you’re “stuck” with that exact schema.

While working on the bindings for the GitHub API, there are some JSON representations which send a “type” field (e.g. “type” : “commit”).
I would like it to be possible to change the class schema to that of a subclass of the original class when detecting a combination of fields.

That is, I specify a combination of fields, and if any is present, then change the class of the current object to a subclass. By specifying combinations of fields, you don't rely on the order in which they appear in the JSON.

Maybe in this way it is possible to accomplish without giving up the nice property of NeoJSON not making intermediate structures? Sounds tough to do without turning it into a hack though..

> Do also note that the typing info is totally absent from the JSON itself, unlike STON for example. NeoJSON also tries (and succeeds) in maintaining a stream based parser that does not generate intermediate structures except your own domain models.
>
> But your request does make sense and I will think about it. It might be possible. It has been a while that I really thought about the implementation.

Alright, I’m willing to help if needed. We can discuss in this e-mail thread.

Sven Van Caekenberghe-2

Re: [Pharo-users] NeoJSON: parsing mapped nil fields

Hi,

(Other comments in line).

I added a new feature:

===
Name: Neo-JSON-Core-SvenVanCaekenberghe.31
Author: SvenVanCaekenberghe
Time: 30 December 2015, 12:31:59.303349 pm
UUID: fb235526-3c04-4e5f-a543-9a7e9eaaac2a
Ancestors: Neo-JSON-Core-SvenVanCaekenberghe.30

New #allowNil option to Object mapping so that null values are accepted and returned as nil when reading a schema using #nextAs: (off by default)

Added NeoJSONObjectMapping>>#allowNil
Added NeoJSONReaderTests>>#testAllowNil
===
Name: Neo-JSON-Tests-SvenVanCaekenberghe.31
Author: SvenVanCaekenberghe
Time: 30 December 2015, 12:32:13.455009 pm
UUID: 650b4ecf-c9d9-40a5-a988-e4e3801faa42
Ancestors: Neo-JSON-Tests-SvenVanCaekenberghe.30

New #allowNil option to Object mapping so that null values are accepted and returned as nil when reading a schema using #nextAs: (off by default)

Added NeoJSONObjectMapping>>#allowNil
Added NeoJSONReaderTests>>#testAllowNil
===

From the unit test, this allows you to do as follows:

self
assert: ((NeoJSONReader on: 'null' readStream)
mapInstVarsFor: Point;
for: Point do: [ :mapping | mapping allowNil ];
nextAs: Point)
equals: nil.

self
assert: ((NeoJSONReader on: '[ { "x" : 1, "y" : 2 }, null, { "x" : 3, "y" : -1 } ]' readStream)
mapInstVarsFor: Point;
for: Point do: [ :mapping | mapping allowNil ];
for: #ArrayOfPoints customDo: [ :mapping | mapping listOfElementSchema: Point ];
nextAs: #ArrayOfPoints)
equals: { 1 @ 2. nil. 3 @ -1 }.

I decided to make this a per mapping option, not a per reader option.

> On 23 Dec 2015, at 11:19, Skip Lentz <[hidden email]> wrote:
>
> Hi, thanks for replying! I can share some ideas.
>
>> On Dec 23, 2015, at 12:09 AM, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> Hi,
>>
>> NeoJSON does have a bit of a split personality.
>
> Yes, this is what I noticed. Apart from this, I think it’s an awesome library, and I thank you a lot for making it.
> It might be good to not make a distinction between #nextAs: and #next, and only rely on #nextAs:. Then #next might be rewritten as such:
>
> NeoJSONReader>>next
>
> ^ self nextAs: self mapClass
>
> Don’t know if that makes sense or not. It might alleviate this split personality, as you put it.

No, that would not help ;-)

The two approaches are fundamentally different: either you accept anything that is present (dynamic mode), returning lists (Arrays) and maps (Dictionaries), or you accept a schema that is predefined (static mode).

>> When you use mappings and #nextAs: it tries to map a static type structure on what is coming in. As such it is not really prepared for dynamic surprises like objects being replaced by null.
>
> Yes, in the general case, once you’ve sent #nextAs: to the reader with your value schema, you’re “stuck” with that exact schema.
>
> While working on the bindings for the GitHub API, there are some JSON representations which send a “type” field (e.g. “type” : “commit”).
> I would like it to be possible to change the class schema to that of a subclass of the original class when detecting a combination of fields.
>
> That is, I specify a combination of fields, and if any is present, then change the class of the current object to a subclass. By specifying combinations of fields, you don't rely on the order in which they appear in the JSON.
>
> Maybe in this way it is possible to accomplish without giving up the nice property of NeoJSON not making intermediate structures? Sounds tough to do without turning it into a hack though..

I know what you mean, there are 2 common approaches (some would call it a wrong way to use JSON, but that is another discussion):

{ "type":"Point", "x":1, "y":2 }

or

{ "type":"Point", "value": {"x":1, "y":2} }

(Type could be class, or whatever).

The first case is bad, you have to read everything before you can make a decision, which IMO kills the advantage of mapping while trying to avoid intermediate structures (i.e. you could do the mapping yourself afterwards).

The second case is a bit better, but not much. Both top level elements could have switched place, in which case you already have to read the value, not knowing what to map it to.

You could try experimenting with Custom mappings, they allow arbitrary conversions, but I doubt if you could really prevent intermediate structure creation. But maybe I just don't see it right, and it is possible after all.

If the returned JSON is highly dynamic and variable, I would just use dynamic mode and then in a second pass convert the maps/lists to domain objects myself. Or someone should write a generic maps/lists to domain model objects mapper (independent of JSON).

>> Do also note that the typing info is totally absent from the JSON itself, unlike STON for example. NeoJSON also tries (and succeeds) in maintaining a stream based parser that does not generate intermediate structures except your own domain models.
>>
>> But your request does make sense and I will think about it. It might be possible. It has been a while that I really thought about the implementation.
>
> Alright, I’m willing to help if needed. We can discuss in this e-mail thread.

Thanks for the feedback/discussion.

Regards,

Sven

Skip Lentz-2

Re: [Pharo-users] NeoJSON: parsing mapped nil fields

Hi Sven,

On Dec 30, 2015, at 12:51 PM, Sven Van Caekenberghe <[hidden email]> wrote:

I added a new feature:

Thank you for this new feature!

From the unit test, this allows you to do as follows:

self
assert: ((NeoJSONReader on: 'null' readStream)
            mapInstVarsFor: Point;
            for: Point do: [ :mapping | mapping allowNil ];
            nextAs: Point)
equals: nil.

self
assert: ((NeoJSONReader on: '[ { "x" : 1, "y" : 2 }, null, { "x" : 3, "y" : -1 } ]' readStream)
            mapInstVarsFor: Point;
            for: Point do: [ :mapping | mapping allowNil ];
            for: #ArrayOfPoints customDo: [ :mapping | mapping listOfElementSchema: Point ];
            nextAs: #ArrayOfPoints)
equals: { 1 @ 2. nil. 3 @ -1 }.

I decided to make this a per mapping option, not a per reader option.

Maybe there should be an ensureNotNil option too, which sets allowNil to false?

This allows for changing it back to the default mode when reusing the same reader.

I tried making this a per _property mapping_ option, but it became a bit of a hack so I stopped doing that.

No, that would not help ;-)

The two approaches are fundamentally different: either you accept anything that is present (dynamic mode), returning lists (Arrays) and maps (Dictionaries), or you accept a schema that is predefined (static mode).

When you use mappings and #nextAs: it tries to map a static type structure on what is coming in. As such it is not really prepared for dynamic surprises like objects being replaced by null.

Yes, in the general case, once you’ve sent #nextAs: to the reader with your value schema, you’re “stuck” with that exact schema.

While working on the bindings for the GitHub API, there are some JSON representations which send a “type” field (e.g. “type” : “commit”).
I would like it to be possible to change the class schema to that of a subclass of the original class when detecting a combination of fields.

That is, I specify a combination of fields, and if any is present, then change the class of the current object to a subclass. By specifying combinations of fields, you don't rely on the order in which they appear in the JSON.

Maybe in this way it is possible to accomplish without giving up the nice property of NeoJSON not making intermediate structures? Sounds tough to do without turning it into a hack though..

I know what you mean, there are 2 common approaches (some would call it a wrong way to use JSON, but that is another discussion):

{ "type":"Point", "x":1, "y":2 }

or

{ "type":"Point", "value": {"x":1, "y":2} }

(Type could be class, or whatever).

The first case is bad, you have to read everything before you can make a decision, which IMO kills the advantage of mapping while trying to avoid intermediate structures (i.e. you could do the mapping yourself afterwards).

The second case is a bit better, but not much. Both top level elements could have switched place, in which case you already have to read the value, not knowing what to map it to.

Exactly, this is what I meant.

You could try experimenting with Custom mappings, they allow arbitrary conversions, but I doubt if you could really prevent intermediate structure creation. But maybe I just don't see it right, and it is possible after all.

One could perhaps use adoptInstance:, based on some conditions (e.g. “if the json contains this combination of keys”, “if the type field has the value “commit””).

But on the other hand this solution would make NeoJSON less simple, and the simplicity of the library is a very nice property. It would be stupid to throw that property away for a solution that doesn't even cover all of the cases.

If the returned JSON is highly dynamic and variable, I would just use dynamic mode and then in a second pass convert the maps/lists to domain objects myself. Or someone should write a generic maps/lists to domain model objects mapper (independent of JSON).

Yes this might be the best solution in this case.

Best wishes for 2016 :),

Skip