[ ANN ] Neo Universal Binary JSON

Hi,

I just published an implementation of Universal Binary JSON (UBJSON) for Pharo.

https://github.com/svenvc/NeoUniversalBinaryJSON

Universal Binary JSON (UBJSON) is a computer data interchange format. It is a binary form directly imitating JSON, but requiring fewer bytes of data. It aims to achieve the generality of JSON, combined with being easier and more efficient to process than JSON.

http://ubjson.org
https://en.wikipedia.org/wiki/UBJSON

The size/speed/efficiency differences are minor for typical JSON payloads, especially compared with compacted JSON. The implementation is simpler, though, as there is no string escaping and no number parsing.

UBJSON is making a larger difference when dealing with arrays containing numbers. Especially with ByteArrays, UBJSON makes a huge difference, since these are essentially stored natively.

Sven

Excellent!

Thanks Sven for this as well as for all your other contributions to the community.


Noury

Sven

Excellent. Can the same idea be extended to STON - or is it there already?

Peter Kenny

-----Original Message-----
From: Pharo-users <[hidden email]> On Behalf Of Sven
Van Caekenberghe
Sent: 09 October 2019 13:29
To: Any question about pharo is welcome <[hidden email]>
Subject: [Pharo-users] [ ANN ] Neo Universal Binary JSON

Hi,

I just published an implementation of Universal Binary JSON (UBJSON) for
Pharo.

https://github.com/svenvc/NeoUniversalBinaryJSON

Universal Binary JSON (UBJSON) is a computer data interchange format. It is
a binary form directly imitating JSON, but requiring fewer bytes of data. It
aims to achieve the generality of JSON, combined with being easier and more
efficient to process than JSON.

http://ubjson.org
https://en.wikipedia.org/wiki/UBJSON

The size/speed/efficiency differences are minor for typical JSON payloads,
especially compared with compacted JSON. The implementation is simpler,
though, as there is no string escaping and no number parsing.

UBJSON is making a larger difference when dealing with arrays containing
numbers. Especially with ByteArrays, UBJSON makes a huge difference, since
these are essentially stored natively.

Sven

> On 9 Oct 2019, at 14:49, PBKResearch <[hidden email]> wrote:
>
> Sven
>
> Excellent. Can the same idea be extended to STON - or is it there already?

I did not (yet) consider that, I will think about it.

However, Pharo contains two serialisation formats out of the box, STON, which is a textual format focusing on domain model objects, and FUEL, which is a fast binary format that can serialise special system objects as well.

> On 9 Oct 2019, at 14:29, Sven Van Caekenberghe <[hidden email]> wrote:
>
> The size/speed/efficiency differences are minor for typical JSON payloads, especially compared with compacted JSON. The implementation is simpler, though, as there is no string escaping and no number parsing.
>
> UBJSON is making a larger difference when dealing with arrays containing numbers. Especially with ByteArrays, UBJSON makes a huge difference, since these are essentially stored natively.

Here are some details/benchmarks.


1. A typical JSON payload


data := ZnConstants httpStatusCodes associations
          collect: [ :each | { #code->each key. #reason->each value } asDictionary ].

[ NeoUBJSONWriter toByteArray: data ] bench. "'11108.114 per second'"

bytes := NeoUBJSONWriter toByteArray: data.

bytes size. "2290"

[ NeoUBJSONReader fromByteArray: bytes ] bench. "'4412.670 per second'"

[ NeoJSONWriter toString: data ] bench. "'11542.783 per second'"

json := NeoJSONWriter toString: data.

json size. "2358"

[ NeoJSONReader fromString: json ] bench. "'4814.711 per second'"


2. A 1K integer array


data := (1 to: 1024) asArray.

[ NeoUBJSONWriter toByteArray: data ] bench. "'6945.444 per second'"

bytes := NeoUBJSONWriter toByteArray: data.

bytes size. "2822"

[ NeoUBJSONReader fromByteArray: bytes ] bench. "'3280.632 per second'"

[ NeoJSONWriter toString: data ] bench. "'4523.095 per second'"

json := NeoJSONWriter toString: data.

json size. "4014"

[ NeoJSONReader fromString: json ] bench. "'1253.749 per second'"


3. A 1K byte array


data := ByteArray new: 1024 streamContents: [ :out |
          1024 timesRepeat: [ out nextPut: 256 atRandom - 1 ] ].

[ NeoUBJSONWriter toByteArray: data ] bench. "'538493.501 per second'"

bytes := NeoUBJSONWriter toByteArray: data.

bytes size. "1031"

[ NeoUBJSONReader fromByteArray: bytes ] bench. "'216269.200 per second'"

[ NeoJSONWriter toString: data ] bench. "'4579.084 per second'"

json := NeoJSONWriter toString: data.

json size. "3686"

[ NeoJSONReader fromString: json ] bench. "'1297.362 per second'"


Right now, only ByteArray got a highly optimised implementation. In the future, FloatArray and IntegerArray could receive the same treatment.

Sven

PS:

Note that UBJSON is not the same as BSON, nor CBOR, nor MessagePack, which all exist in the same space.