Json encoding

Vitor

 

First clarification: There are two different serialization formats, json and STON. The content of a json file can only be number, Boolean, string, array, dictionary. STON is an extended form based on json, designed to represent (almost) any Smalltalk object.

 

Second clarification: What are you trying to do? Do you want to serialize data for your own use, to read back into your own Pharo app? Or do you want to export data in json format, so that other users, not using necessarily using Pharo, can import it? For the first case, STON is a very flexible and convenient system. For the second case, you must stick to strict json, using only the classes allowed in json.

 

Coming to your specific examples, Date is not a class allowed in json. This is not a limitation of NeoJson or STON, it is a limitation of the definition of json. If you want to include a Date in a json file, you must turn it into an instance of a permitted class. So you could write:

 

STON toJsonString: Date today asString.

 

The only complication here is, if the json is read by other users, they must understand the format generated by Date>>asString.

 

If you are using STON to serialise objects for your own use, you may want to exclude some instvars (e.g. block closures). This is described in the class comments to the STON-Core package. You need to include a class side message #stonAllInstVarNames in the class, which lists the names of the instvars that are to be serialized.

 

If you want to go deeper into STON, I think Sven has an article on his website telling the whole story. This maybe enough to get you started.

 

HTH

 

Peter Kenny

 

From: Pharo-users <[hidden email]> On Behalf Of Vitor Medina Cruz
Sent: 20 March 2020 01:20
To: Any question about pharo is welcome <[hidden email]>
Subject: [Pharo-users] Json encoding

 

Hello,

 

I know two projects of json encoding/decoding — NeoJson and STON.

 

In Java I have two most used ones too: Gson and Jackson. Using those I can simply pass any object and it they can convert to a json string, the former can't deal with cycles, the latter can with some little config.

 

NeoJson seems to be limited to primitives, for example, in Pharo 8 I can't run

 

NeoJSONWriter toString: (Date today)

 

Since I got:

 

NeoJSONMappingNotFound: No mapping found for Date in NeoJSONWriter

 

 

STON works fine with 

 

STON toString: (Date today).

 

but fail with:

 

STON toJsonString: (Date today).

 

Also, STON fails if I try to serialize an object which contains an instance variable pointing to a block closure. I didn't find out how to ignore these instance variable as it is unnecessary for what I need at the moment.

 

So, which lib for json or some object/string encoding/decoding should I be using on Pharo? Is there something better than those libs? Am I doing something wrong? My experience tells this should be simple, but it is not.

 

Thanks in advance,

Vitor

Vitor

 

First clarification: There are two different serialization formats, json and STON. The content of a json file can only be number, Boolean, string, array, dictionary. STON is an extended form based on json, designed to represent (almost) any Smalltalk object.

 

Second clarification: What are you trying to do? Do you want to serialize data for your own use, to read back into your own Pharo app? Or do you want to export data in json format, so that other users, not using necessarily using Pharo, can import it? For the first case, STON is a very flexible and convenient system. For the second case, you must stick to strict json, using only the classes allowed in json.

 

Coming to your specific examples, Date is not a class allowed in json. This is not a limitation of NeoJson or STON, it is a limitation of the definition of json. If you want to include a Date in a json file, you must turn it into an instance of a permitted class. So you could write:

 

STON toJsonString: Date today asString.

 

The only complication here is, if the json is read by other users, they must understand the format generated by Date>>asString.

 

If you are using STON to serialise objects for your own use, you may want to exclude some instvars (e.g. block closures). This is described in the class comments to the STON-Core package. You need to include a class side message #stonAllInstVarNames in the class, which lists the names of the instvars that are to be serialized.

 

If you want to go deeper into STON, I think Sven has an article on his website telling the whole story. This maybe enough to get you started.

 

HTH

 

Peter Kenny

 

From: Pharo-users <[hidden email]> On Behalf Of Vitor Medina Cruz
Sent: 20 March 2020 01:20
To: Any question about pharo is welcome <[hidden email]>
Subject: [Pharo-users] Json encoding

 

Hello,

 

I know two projects of json encoding/decoding — NeoJson and STON.

 

In Java I have two most used ones too: Gson and Jackson. Using those I can simply pass any object and it they can convert to a json string, the former can't deal with cycles, the latter can with some little config.

 

NeoJson seems to be limited to primitives, for example, in Pharo 8 I can't run

 

NeoJSONWriter toString: (Date today)

 

Since I got:

 

NeoJSONMappingNotFound: No mapping found for Date in NeoJSONWriter

 

 

STON works fine with 

 

STON toString: (Date today).

 

but fail with:

 

STON toJsonString: (Date today).

 

Also, STON fails if I try to serialize an object which contains an instance variable pointing to a block closure. I didn't find out how to ignore these instance variable as it is unnecessary for what I need at the moment.

 

So, which lib for json or some object/string encoding/decoding should I be using on Pharo? Is there something better than those libs? Am I doing something wrong? My experience tells this should be simple, but it is not.

 

Thanks in advance,

Vitor

I think you have to think harder about this ;-)

It would indeed be relatively easy to allow any object to be written out to JSON, falling back to just writing out its instance variables. You could hack that today by overwriting/implementing #neoJsonOn:

But you would never be able to read that back, because you don't known the types/classes (in general).

Say you have a Person object with a dateOfBirth in it, a Date.

{ "name":"Joe","dateOfBirth":"19900407" }

You don't know this is a Person, nor that dateOfBirth is a Date. You have to tell the JSON reader/parser that explicitly. And even if you tell it to start with Person, no reflection in Pharo tells you that dateOfBirth is a Date.

In statically typed languages you do know that.

That is why NeoJSON has mappings (section 5 of the document). These provide that missing information. But this mechanism is not perfect (it can become quite complex with deep nesting or variant types).

The unit tests show many usages of mappings.

But even so, the original JSON is not self describing.

This is why STON exists.

It was a design choice to restrict which classes can automatically be converted to JSON without an explicit mapping to the primitive JSON types.

> On 20 Mar 2020, at 21:47, Vitor Medina Cruz <[hidden email]> wrote:
>
> Thanks Sven,
>
> One of the pages provided one solution I need: to ignore certain instance variable while encoding.
>
> I, however, do not understand the problem for Pharo to produce json for arbitrary objects just like any Java API — I thought it would even be simpler.
>
> " In Pharo/Smalltalk, contrary to Java, we are not used to full type descriptions. "
>
> Well, but you need to? Java APIs use reflection to do this job, when looking inside an object you can tell it's type. Every API will do a best effort to navigate through object nesting and, very often, it will find known primitive objects  down the road. So complex objects are created from less complex ones, that area created eventually from from primitives. If it cannot deal with the object serialization, it raises an error and generally there are mapping features to deal with those border cases. I was surprised to see that was not the case with STON and NeoJSON (well, it appears that STON tries to do it for it's format)
>
> The serialization is also simple, you do something like:
>
> ston forClass: MyComplexObject desserialize: aJsonString
>
>
> And well, that is it.
>
> On Fri, Mar 20, 2020 at 8:46 AM PBKResearch <[hidden email]> wrote:
> Vitor
>
> 
>
> First clarification: There are two different serialization formats, json and STON. The content of a json file can only be number, Boolean, string, array, dictionary. STON is an extended form based on json, designed to represent (almost) any Smalltalk object.
>
> 
>
> Second clarification: What are you trying to do? Do you want to serialize data for your own use, to read back into your own Pharo app? Or do you want to export data in json format, so that other users, not using necessarily using Pharo, can import it? For the first case, STON is a very flexible and convenient system. For the second case, you must stick to strict json, using only the classes allowed in json.
>
> 
>
> Coming to your specific examples, Date is not a class allowed in json. This is not a limitation of NeoJson or STON, it is a limitation of the definition of json. If you want to include a Date in a json file, you must turn it into an instance of a permitted class. So you could write:
>
> 
>
> STON toJsonString: Date today asString.
>
> 
>
> The only complication here is, if the json is read by other users, they must understand the format generated by Date>>asString.
>
> 
>
> If you are using STON to serialise objects for your own use, you may want to exclude some instvars (e.g. block closures). This is described in the class comments to the STON-Core package. You need to include a class side message #stonAllInstVarNames in the class, which lists the names of the instvars that are to be serialized.
>
> 
>
> If you want to go deeper into STON, I think Sven has an article on his website telling the whole story. This maybe enough to get you started.
>
> 
>
> HTH
>
> 
>
> Peter Kenny
>
> 
>
> From: Pharo-users <[hidden email]> On Behalf Of Vitor Medina Cruz
> Sent: 20 March 2020 01:20
> To: Any question about pharo is welcome <[hidden email]>
> Subject: [Pharo-users] Json encoding
>
> 
>
> Hello,
>
> 
>
> I know two projects of json encoding/decoding — NeoJson and STON.
>
> 
>
> In Java I have two most used ones too: Gson and Jackson. Using those I can simply pass any object and it they can convert to a json string, the former can't deal with cycles, the latter can with some little config.
>
> 
>
> NeoJson seems to be limited to primitives, for example, in Pharo 8 I can't run
>
> 
>
> NeoJSONWriter toString: (Date today)
>
> 
>
> Since I got:
>
> 
>
> NeoJSONMappingNotFound: No mapping found for Date in NeoJSONWriter
>
> 
>
> 
>
> STON works fine with
>
> 
>
> STON toString: (Date today).
>
> 
>
> but fail with:
>
> 
>
> STON toJsonString: (Date today).
>
> 
>
> Also, STON fails if I try to serialize an object which contains an instance variable pointing to a block closure. I didn't find out how to ignore these instance variable as it is unnecessary for what I need at the moment.
>
> 
>
> So, which lib for json or some object/string encoding/decoding should I be using on Pharo? Is there something better than those libs? Am I doing something wrong? My experience tells this should be simple, but it is not.
>
> 
>
> Thanks in advance,
>
> Vitor
>

Victor,

But again, have you read everything ? Looked at the unit tests ?

A mapping can be specified either globally on the class side, or per reader/writer.

Have a look a NeoJSONTestObject1, NeoJSONTestObject2 and NeoJSONTestObject3 and how they are used in tests.

You can perfectly do a #mapAllInstVarsFor, #map[All]InstVars or #map[All]InstVars: or you can work based on properties/accessors. And there is a lot more (and it gets complicated in special cases).

I know that the DSL is not perfect, it just kind of evolved.

One of the design goals of NeoJSON is that you can work with mappings/types without creating intermediate structures (so it is not JSON->Dictionary->MyObject, but JSON->MyObject).

For example, there is

NeoJSONTestObject1 class>>#neoJsonMapping: mapper
  mapper for: self do: [ :mapping |
    mapping mapInstVars: #(id name).
    (mapping mapInstVar: #timestamp to: 'created-at') valueSchema: DateAndTime.
    (mapping mapInstVar: #points) valueSchema: #ArrayOfPoints.
    (mapping mapInstVar: #bytes) valueSchema: ByteArray ].
  mapper for: DateAndTime customDo: [ :mapping |
    mapping decoder: [ :string | DateAndTime fromString: string ].
    mapping encoder: [ :dateAndTime | dateAndTime printString ] ].
  mapper for: #ArrayOfPoints customDo: [ :mapping |
    mapping listOfElementSchema: Point ]. 
  mapper mapAllInstVarsFor: Point.
  mapper for: ByteArray customDo: [ :mapping |
      mapping listOfType: ByteArray ]

which already covers a lot. You could also specify this mapping directly on either a reader or a writer:

neoJsonReader
  for: NeoJSONTestObject1 do: [ :mapping |
    mapping mapInstVars: #(id name).
    (mapping mapInstVar: #timestamp to: 'created-at') valueSchema: DateAndTime.
    (mapping mapInstVar: #points) valueSchema: #ArrayOfPoints.
    (mapping mapInstVar: #bytes) valueSchema: ByteArray ];
   for: DateAndTime customDo: [ :mapping |
     mapping decoder: [ :string | DateAndTime fromString: string ].
     mapping encoder: [ :dateAndTime | dateAndTime printString ] ];
   for: #ArrayOfPoints customDo: [ :mapping |
     mapping listOfElementSchema: Point ];
   mapAllInstVarsFor: Point;
   for: ByteArray customDo: [ :mapping |
     mapping listOfType: ByteArray ]

In an example like the above, the whole mapping is in one place, but you could distribute it over class side methods. It would even be possible to have a special utility object that (automagically) constructs/fills in these mappings - several users done this already.

Again, for JSON there cannot be one way to encode types like Dates.

STON can make such decisions because it stands on its own.

> On 21 Mar 2020, at 00:43, Vitor Medina Cruz <[hidden email]> wrote:
>
> Oh! I understand, desserialize is, indeed, a problem because you don't have the types.....
>
> I find very hard to follow the mapping instructions, why can't I just tell the class of my instvars?
>
> jsonReader := NeoJSONReader new configure: MyCustomClass
>                                                       withInstVarMapping: {#name -> String. #dateOrBirth -> Date}
>
> jsonRreader fromJson: { "name":"Joe","dateOfBirth":"19900407"}
>                   toClass: MyCustomClass
>
> ???
>
>
> Well, I am going to experiment with STON for now, I used NeoJson some time ago, but for simpler stuff.
>
> Ignoring the instvar worked.
>
> Thanks!!
>
>
>
>
>
>
> On Fri, Mar 20, 2020 at 7:41 PM Sven Van Caekenberghe <[hidden email]> wrote:
> I think you have to think harder about this ;-)
>
> It would indeed be relatively easy to allow any object to be written out to JSON, falling back to just writing out its instance variables. You could hack that today by overwriting/implementing #neoJsonOn:
>
> But you would never be able to read that back, because you don't known the types/classes (in general).
>
> Say you have a Person object with a dateOfBirth in it, a Date.
>
> { "name":"Joe","dateOfBirth":"19900407" }
>
> You don't know this is a Person, nor that dateOfBirth is a Date. You have to tell the JSON reader/parser that explicitly. And even if you tell it to start with Person, no reflection in Pharo tells you that dateOfBirth is a Date.
>
> In statically typed languages you do know that.
>
> That is why NeoJSON has mappings (section 5 of the document). These provide that missing information. But this mechanism is not perfect (it can become quite complex with deep nesting or variant types).
>
> The unit tests show many usages of mappings.
>
> But even so, the original JSON is not self describing.
>
> This is why STON exists.
>
> It was a design choice to restrict which classes can automatically be converted to JSON without an explicit mapping to the primitive JSON types.
>
> > On 20 Mar 2020, at 21:47, Vitor Medina Cruz <[hidden email]> wrote:
> >
> > Thanks Sven,
> >
> > One of the pages provided one solution I need: to ignore certain instance variable while encoding.
> >
> > I, however, do not understand the problem for Pharo to produce json for arbitrary objects just like any Java API — I thought it would even be simpler.
> >
> > " In Pharo/Smalltalk, contrary to Java, we are not used to full type descriptions. "
> >
> > Well, but you need to? Java APIs use reflection to do this job, when looking inside an object you can tell it's type. Every API will do a best effort to navigate through object nesting and, very often, it will find known primitive objects  down the road. So complex objects are created from less complex ones, that area created eventually from from primitives. If it cannot deal with the object serialization, it raises an error and generally there are mapping features to deal with those border cases. I was surprised to see that was not the case with STON and NeoJSON (well, it appears that STON tries to do it for it's format)
> >
> > The serialization is also simple, you do something like:
> >
> > ston forClass: MyComplexObject desserialize: aJsonString
> >
> >
> > And well, that is it.
> >
> > On Fri, Mar 20, 2020 at 8:46 AM PBKResearch <[hidden email]> wrote:
> > Vitor
> >
> > 
> >
> > First clarification: There are two different serialization formats, json and STON. The content of a json file can only be number, Boolean, string, array, dictionary. STON is an extended form based on json, designed to represent (almost) any Smalltalk object.
> >
> > 
> >
> > Second clarification: What are you trying to do? Do you want to serialize data for your own use, to read back into your own Pharo app? Or do you want to export data in json format, so that other users, not using necessarily using Pharo, can import it? For the first case, STON is a very flexible and convenient system. For the second case, you must stick to strict json, using only the classes allowed in json.
> >
> > 
> >
> > Coming to your specific examples, Date is not a class allowed in json. This is not a limitation of NeoJson or STON, it is a limitation of the definition of json. If you want to include a Date in a json file, you must turn it into an instance of a permitted class. So you could write:
> >
> > 
> >
> > STON toJsonString: Date today asString.
> >
> > 
> >
> > The only complication here is, if the json is read by other users, they must understand the format generated by Date>>asString.
> >
> > 
> >
> > If you are using STON to serialise objects for your own use, you may want to exclude some instvars (e.g. block closures). This is described in the class comments to the STON-Core package. You need to include a class side message #stonAllInstVarNames in the class, which lists the names of the instvars that are to be serialized.
> >
> > 
> >
> > If you want to go deeper into STON, I think Sven has an article on his website telling the whole story. This maybe enough to get you started.
> >
> > 
> >
> > HTH
> >
> > 
> >
> > Peter Kenny
> >
> > 
> >
> > From: Pharo-users <[hidden email]> On Behalf Of Vitor Medina Cruz
> > Sent: 20 March 2020 01:20
> > To: Any question about pharo is welcome <[hidden email]>
> > Subject: [Pharo-users] Json encoding
> >
> > 
> >
> > Hello,
> >
> > 
> >
> > I know two projects of json encoding/decoding — NeoJson and STON.
> >
> > 
> >
> > In Java I have two most used ones too: Gson and Jackson. Using those I can simply pass any object and it they can convert to a json string, the former can't deal with cycles, the latter can with some little config.
> >
> > 
> >
> > NeoJson seems to be limited to primitives, for example, in Pharo 8 I can't run
> >
> > 
> >
> > NeoJSONWriter toString: (Date today)
> >
> > 
> >
> > Since I got:
> >
> > 
> >
> > NeoJSONMappingNotFound: No mapping found for Date in NeoJSONWriter
> >
> > 
> >
> > 
> >
> > STON works fine with
> >
> > 
> >
> > STON toString: (Date today).
> >
> > 
> >
> > but fail with:
> >
> > 
> >
> > STON toJsonString: (Date today).
> >
> > 
> >
> > Also, STON fails if I try to serialize an object which contains an instance variable pointing to a block closure. I didn't find out how to ignore these instance variable as it is unnecessary for what I need at the moment.
> >
> > 
> >
> > So, which lib for json or some object/string encoding/decoding should I be using on Pharo? Is there something better than those libs? Am I doing something wrong? My experience tells this should be simple, but it is not.
> >
> > 
> >
> > Thanks in advance,
> >
> > Vitor
> >
>
>