SimpleLiteralArray Spec

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

SimpleLiteralArray Spec

Sven Van Caekenberghe
OK, just to get started, let try to write a spec for SimpleLiteralArray, this is based on JSON (and my own STON, not ready for public consumption). It will soon become clear that this is slightly harder than expected, IMO a writer is needed as well.

S y n t a x

value
  primitive-value
  list
  nil
primitive-value
  number
  true
  false
  symbol
  string
  classname
list
  #()
  #( elements )
elements
  value
  value , elements
string
  ''
  ' chars '
chars
  char
  char chars
char
  any-printable-ASCII-character-
    except-"-or-\
  \"
  \\
  \/
  \b
  \f
  \n
  \r
  \t
  \u four-hex-digits
symbol
  # chars-limited
  # ' chars '
chars-limited
  char-limited
  char-limited chars-limited
char-limited
  a-z A-Z 0-9 - _ . /
classname
  uppercase-alpha-char alphanumeric-char
number
  int
  int frac
  int exp
  int frac exp
int
  digit
  digit1-9 digits
  - digit
  - digit1-9 digits
frac
  . digits
exp
  e digits
digits
  digit
  digit digits
e
  e
  e+
  e-
  E
  E+
  E-


Questions remain:

- is the escaping OK ?
- what about symbols without a leading # ?
- what should be allowed in unquoted symbols ?
- what should be allowed for classnames ?
- what about namespaces ?
- do we allow infinite precision integers ?
- maybe the toplevel should always be an array ?


Now: shoot!


Sven


--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill





Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

Dale Henrichs
Sven,

I know that having the parser produce dictionaries (i.e., dictionary _syntax_) is very convenient for the consumer of the parsed output ...

When javascript evaluates/parses JSON, you will note that you get real objects back instead of dictionaries and you can send messages to those objects, so I would think that rather than inventing a dictionary syntax, you invent a STON-object syntax then we can then send setter/getter style messages to these boys... then we'd  have _objects_, arrays, strings, numbers and booleans for the STON format ... a field could be optionally included in the STON object called _class_ which would optionally allow the parser to produce instances of that class with all of the fields filled in ...

JSON on steroids!

Anyway ... just a thought.

Dale

----- Original Message -----
| From: "Sven Van Caekenberghe" <[hidden email]>
| To: "Pharo Development" <[hidden email]>
| Sent: Tuesday, April 24, 2012 4:26:12 AM
| Subject: [Pharo-project] SimpleLiteralArray Spec
|
| OK, just to get started, let try to write a spec for
| SimpleLiteralArray, this is based on JSON (and my own STON, not
| ready for public consumption). It will soon become clear that this
| is slightly harder than expected, IMO a writer is needed as well.
|
| S y n t a x
|
| value
|   primitive-value
|   list
|   nil
| primitive-value
|   number
|   true
|   false
|   symbol
|   string
|   classname
| list
|   #()
|   #( elements )
| elements
|   value
|   value , elements
| string
|   ''
|   ' chars '
| chars
|   char
|   char chars
| char
|   any-printable-ASCII-character-
|     except-"-or-\
|   \"
|   \\
|   \/
|   \b
|   \f
|   \n
|   \r
|   \t
|   \u four-hex-digits
| symbol
|   # chars-limited
|   # ' chars '
| chars-limited
|   char-limited
|   char-limited chars-limited
| char-limited
|   a-z A-Z 0-9 - _ . /
| classname
|   uppercase-alpha-char alphanumeric-char
| number
|   int
|   int frac
|   int exp
|   int frac exp
| int
|   digit
|   digit1-9 digits
|   - digit
|   - digit1-9 digits
| frac
|   . digits
| exp
|   e digits
| digits
|   digit
|   digit digits
| e
|   e
|   e+
|   e-
|   E
|   E+
|   E-
|
|
| Questions remain:
|
| - is the escaping OK ?
| - what about symbols without a leading # ?
| - what should be allowed in unquoted symbols ?
| - what should be allowed for classnames ?
| - what about namespaces ?
| - do we allow infinite precision integers ?
| - maybe the toplevel should always be an array ?
|
|
| Now: shoot!
|
|
| Sven
|
|
| --
| Sven Van Caekenberghe
| http://stfx.eu
| Smalltalk is the Red Pill
|
|
|
|
|
|

Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

NorbertHartl
What is STON and STON-object syntax?

Norbert

Am 24.04.2012 um 18:18 schrieb Dale Henrichs:

> Sven,
>
> I know that having the parser produce dictionaries (i.e., dictionary _syntax_) is very convenient for the consumer of the parsed output ...
>
> When javascript evaluates/parses JSON, you will note that you get real objects back instead of dictionaries and you can send messages to those objects, so I would think that rather than inventing a dictionary syntax, you invent a STON-object syntax then we can then send setter/getter style messages to these boys... then we'd  have _objects_, arrays, strings, numbers and booleans for the STON format ... a field could be optionally included in the STON object called _class_ which would optionally allow the parser to produce instances of that class with all of the fields filled in ...
>
> JSON on steroids!
>
> Anyway ... just a thought.
>
> Dale
>
> ----- Original Message -----
> | From: "Sven Van Caekenberghe" <[hidden email]>
> | To: "Pharo Development" <[hidden email]>
> | Sent: Tuesday, April 24, 2012 4:26:12 AM
> | Subject: [Pharo-project] SimpleLiteralArray Spec
> |
> | OK, just to get started, let try to write a spec for
> | SimpleLiteralArray, this is based on JSON (and my own STON, not
> | ready for public consumption). It will soon become clear that this
> | is slightly harder than expected, IMO a writer is needed as well.
> |
> | S y n t a x
> |
> | value
> |   primitive-value
> |   list
> |   nil
> | primitive-value
> |   number
> |   true
> |   false
> |   symbol
> |   string
> |   classname
> | list
> |   #()
> |   #( elements )
> | elements
> |   value
> |   value , elements
> | string
> |   ''
> |   ' chars '
> | chars
> |   char
> |   char chars
> | char
> |   any-printable-ASCII-character-
> |     except-"-or-\
> |   \"
> |   \\
> |   \/
> |   \b
> |   \f
> |   \n
> |   \r
> |   \t
> |   \u four-hex-digits
> | symbol
> |   # chars-limited
> |   # ' chars '
> | chars-limited
> |   char-limited
> |   char-limited chars-limited
> | char-limited
> |   a-z A-Z 0-9 - _ . /
> | classname
> |   uppercase-alpha-char alphanumeric-char
> | number
> |   int
> |   int frac
> |   int exp
> |   int frac exp
> | int
> |   digit
> |   digit1-9 digits
> |   - digit
> |   - digit1-9 digits
> | frac
> |   . digits
> | exp
> |   e digits
> | digits
> |   digit
> |   digit digits
> | e
> |   e
> |   e+
> |   e-
> |   E
> |   E+
> |   E-
> |
> |
> | Questions remain:
> |
> | - is the escaping OK ?
> | - what about symbols without a leading # ?
> | - what should be allowed in unquoted symbols ?
> | - what should be allowed for classnames ?
> | - what about namespaces ?
> | - do we allow infinite precision integers ?
> | - maybe the toplevel should always be an array ?
> |
> |
> | Now: shoot!
> |
> |
> | Sven
> |
> |
> | --
> | Sven Van Caekenberghe
> | http://stfx.eu
> | Smalltalk is the Red Pill
> |
> |
> |
> |
> |
> |
>


Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

Sean P. DeNigris
Administrator
Norbert Hartl wrote
What is STON and STON-object syntax?
Smalltalk Object Notation?
http://ss3.gemstone.com/ss/STON.html
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

STON (was Re: SimpleLiteralArray Spec)

Sven Van Caekenberghe
In reply to this post by NorbertHartl
Norbert, Dale, et al,


On 24 Apr 2012, at 18:43, Norbert Hartl wrote:

> What is STON and STON-object syntax?

Arhh, it is not yet finished, I did it for myself, not for trying to convince the world, since that it too hard anyway.

This is the work-in-progress paper:




The code lives here: http://ss3.gemstone.com/ss/STON

It is a slight variation of JSON, adding some Smalltalk concepts, as well as a solution to the class problem.

But again, I am not trying to convince anyone to use this, I did it as a proof of concept for myself, and I kind of like it.

On the other hand, JSON is so ubiquitous that I wonder a bit, why bother ?

Sven


--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill





ston.pdf (179K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

Dale Henrichs
In reply to this post by Dale Henrichs
Sven,

I see that you have already gone beyond a simple literal array style syntax and it looks like you've headed into YAML territory? ... I'm seeing lots of []... the nice thing about YAML is that they've eliminated a lot of the excess syntactic characters which makes it very readable.

The real test for readability would be a variety of sample documents...

Dale

----- Original Message -----
| From: "Dale Henrichs" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 9:18:26 AM
| Subject: Re: [Pharo-project] SimpleLiteralArray Spec
|
| Sven,
|
| I know that having the parser produce dictionaries (i.e., dictionary
| _syntax_) is very convenient for the consumer of the parsed output
| ...
|
| When javascript evaluates/parses JSON, you will note that you get
| real objects back instead of dictionaries and you can send messages
| to those objects, so I would think that rather than inventing a
| dictionary syntax, you invent a STON-object syntax then we can then
| send setter/getter style messages to these boys... then we'd  have
| _objects_, arrays, strings, numbers and booleans for the STON format
| ... a field could be optionally included in the STON object called
| _class_ which would optionally allow the parser to produce instances
| of that class with all of the fields filled in ...
|
| JSON on steroids!
|
| Anyway ... just a thought.
|
| Dale
|
| ----- Original Message -----
| | From: "Sven Van Caekenberghe" <[hidden email]>
| | To: "Pharo Development" <[hidden email]>
| | Sent: Tuesday, April 24, 2012 4:26:12 AM
| | Subject: [Pharo-project] SimpleLiteralArray Spec
| |
| | OK, just to get started, let try to write a spec for
| | SimpleLiteralArray, this is based on JSON (and my own STON, not
| | ready for public consumption). It will soon become clear that this
| | is slightly harder than expected, IMO a writer is needed as well.
| |
| | S y n t a x
| |
| | value
| |   primitive-value
| |   list
| |   nil
| | primitive-value
| |   number
| |   true
| |   false
| |   symbol
| |   string
| |   classname
| | list
| |   #()
| |   #( elements )
| | elements
| |   value
| |   value , elements
| | string
| |   ''
| |   ' chars '
| | chars
| |   char
| |   char chars
| | char
| |   any-printable-ASCII-character-
| |     except-"-or-\
| |   \"
| |   \\
| |   \/
| |   \b
| |   \f
| |   \n
| |   \r
| |   \t
| |   \u four-hex-digits
| | symbol
| |   # chars-limited
| |   # ' chars '
| | chars-limited
| |   char-limited
| |   char-limited chars-limited
| | char-limited
| |   a-z A-Z 0-9 - _ . /
| | classname
| |   uppercase-alpha-char alphanumeric-char
| | number
| |   int
| |   int frac
| |   int exp
| |   int frac exp
| | int
| |   digit
| |   digit1-9 digits
| |   - digit
| |   - digit1-9 digits
| | frac
| |   . digits
| | exp
| |   e digits
| | digits
| |   digit
| |   digit digits
| | e
| |   e
| |   e+
| |   e-
| |   E
| |   E+
| |   E-
| |
| |
| | Questions remain:
| |
| | - is the escaping OK ?
| | - what about symbols without a leading # ?
| | - what should be allowed in unquoted symbols ?
| | - what should be allowed for classnames ?
| | - what about namespaces ?
| | - do we allow infinite precision integers ?
| | - maybe the toplevel should always be an array ?
| |
| |
| | Now: shoot!
| |
| |
| | Sven
| |
| |
| | --
| | Sven Van Caekenberghe
| | http://stfx.eu
| | Smalltalk is the Red Pill
| |
| |
| |
| |
| |
| |
|
|

Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Dale Henrichs
In reply to this post by Sven Van Caekenberghe
Sven,

I looked at your sample document in the pdf:

TestDomainObject {
  #created : DateAndTime [ '2012-02-14T16:40:15+01:00' ],
  #modified : DateAndTime [ '2012-02-14T16:40:18+01:00' ],
  #integer : 39581,
  #float : 73.84789359463944,
  #description : 'This is a test',
  #color : #green,
  #tags : [
    #two,
    #beta,
    #medium
  ],
  #bytes : ByteArray [ 'afabfdf61d030f43eb67960c0ae9f39f' ],
  #boolean : false
}

And this is very readable (to me)... not as many '[]' as I inferred from reading the tests:).

I take back my YAML comparison ... this has the nice flavor of Smalltalk ...

Dale

----- Original Message -----
| From: "Sven Van Caekenberghe" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 10:06:22 AM
| Subject: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
|
| Norbert, Dale, et al,
|
|
| On 24 Apr 2012, at 18:43, Norbert Hartl wrote:
|
| > What is STON and STON-object syntax?
|
| Arhh, it is not yet finished, I did it for myself, not for trying to
| convince the world, since that it too hard anyway.
|
| This is the work-in-progress paper:
|
|
|
|
| The code lives here: http://ss3.gemstone.com/ss/STON
|
| It is a slight variation of JSON, adding some Smalltalk concepts, as
| well as a solution to the class problem.
|
| But again, I am not trying to convince anyone to use this, I did it
| as a proof of concept for myself, and I kind of like it.
|
| On the other hand, JSON is so ubiquitous that I wonder a bit, why
| bother ?
|
| Sven
|
|
| --
| Sven Van Caekenberghe
| http://stfx.eu
| Smalltalk is the Red Pill
|
|
|
|
|

Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Dale Henrichs
Sven,

For tODE I am very interested having something with this kind of potential ... shipping "real" ST objects back and forth between Amber and Pharo or GemStone has some real appeal...

So Goran, the STON format looks somewhat similar to Tirade ... what would Tirade look like for the same data set?

Dale

----- Original Message -----
| From: "Dale Henrichs" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 10:22:36 AM
| Subject: Re: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
|
| Sven,
|
| I looked at your sample document in the pdf:
|
| TestDomainObject {
|   #created : DateAndTime [ '2012-02-14T16:40:15+01:00' ],
|   #modified : DateAndTime [ '2012-02-14T16:40:18+01:00' ],
|   #integer : 39581,
|   #float : 73.84789359463944,
|   #description : 'This is a test',
|   #color : #green,
|   #tags : [
|     #two,
|     #beta,
|     #medium
|   ],
|   #bytes : ByteArray [ 'afabfdf61d030f43eb67960c0ae9f39f' ],
|   #boolean : false
| }
|
| And this is very readable (to me)... not as many '[]' as I inferred
| from reading the tests:).
|
| I take back my YAML comparison ... this has the nice flavor of
| Smalltalk ...
|
| Dale
|
| ----- Original Message -----
| | From: "Sven Van Caekenberghe" <[hidden email]>
| | To: [hidden email]
| | Sent: Tuesday, April 24, 2012 10:06:22 AM
| | Subject: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
| |
| | Norbert, Dale, et al,
| |
| |
| | On 24 Apr 2012, at 18:43, Norbert Hartl wrote:
| |
| | > What is STON and STON-object syntax?
| |
| | Arhh, it is not yet finished, I did it for myself, not for trying
| | to
| | convince the world, since that it too hard anyway.
| |
| | This is the work-in-progress paper:
| |
| |
| |
| |
| | The code lives here: http://ss3.gemstone.com/ss/STON
| |
| | It is a slight variation of JSON, adding some Smalltalk concepts,
| | as
| | well as a solution to the class problem.
| |
| | But again, I am not trying to convince anyone to use this, I did it
| | as a proof of concept for myself, and I kind of like it.
| |
| | On the other hand, JSON is so ubiquitous that I wonder a bit, why
| | bother ?
| |
| | Sven
| |
| |
| | --
| | Sven Van Caekenberghe
| | http://stfx.eu
| | Smalltalk is the Red Pill
| |
| |
| |
| |
| |
|

Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Sven Van Caekenberghe
In reply to this post by Dale Henrichs

On 24 Apr 2012, at 19:22, Dale Henrichs wrote:

> Sven,
>
> I looked at your sample document in the pdf:
>
> TestDomainObject {
>  #created : DateAndTime [ '2012-02-14T16:40:15+01:00' ],
>  #modified : DateAndTime [ '2012-02-14T16:40:18+01:00' ],
>  #integer : 39581,
>  #float : 73.84789359463944,
>  #description : 'This is a test',
>  #color : #green,
>  #tags : [
>    #two,
>    #beta,
>    #medium
>  ],
>  #bytes : ByteArray [ 'afabfdf61d030f43eb67960c0ae9f39f' ],
>  #boolean : false
> }
>
> And this is very readable (to me)... not as many '[]' as I inferred from reading the tests:).
>
> I take back my YAML comparison ... this has the nice flavor of Smalltalk ...
>
> Dale

Thanks!

The main 'hack' is to extend JSON's list/array [] and map/dictionary {} with a very simple way to instanciate any class, by preceding them with a class name (or alias) and giving the control to instanciate to the class or instance. Furthermore, for Array and Dictionary these are optional. For the concrete representations choosen for say DateAndTime or ByteArray, human readability was the deciding factor. But any of these could be changed without hurting the concept.

Note that there are also references and implicit object counting and graph tracing with a defined visit order. In a discussion with Mariano I kind of decided to no longer treat Strings as shareable/referenceable objects. This is not yet reflected in the document but it is in the code.

Sven
Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

Dale Henrichs
In reply to this post by Sven Van Caekenberghe
Now I can take a crack at your questions:)

----- Original Message -----
| From: "Sven Van Caekenberghe" <[hidden email]>
| To: "Pharo Development" <[hidden email]>
| Sent: Tuesday, April 24, 2012 4:26:12 AM
| Subject: [Pharo-project] SimpleLiteralArray Spec
|
...snip
|
| Questions remain:
|
| - is the escaping OK ?
| - what about symbols without a leading # ?

If you can get rid of extra syntax, that would be good ...

| - what should be allowed in unquoted symbols ?
| - what should be allowed for classnames ?

For Cypress we decided that the name of the directory (className.class) would not necessarily be the real name of the class since the classname rules might vary across different platforms, so something along the same lines might make sense by encoding the real name of the class in a field of the object while preserving the external name ... that way you the external class name could be 'classname' as defined, while the real name of the class could be specified in an optional classname field when the real name can't be mapped to 'classnamed'
| - what about namespaces ?

For Cypress namespaces are handled by special class attributes , so perhaps there would be an optional class_properties field where additional meta information about the class name resolution can be stored

| - do we allow infinite precision integers ?
| - maybe the toplevel should always be an array ?

I like the idea that the top level can be any legal construct ...

|
|
| Now: shoot!
|
|
| Sven
|
|
| --
| Sven Van Caekenberghe
| http://stfx.eu
| Smalltalk is the Red Pill
|
|
|
|
|
|

Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Dale Henrichs
In reply to this post by Sven Van Caekenberghe
Sven,

Just an off-the-wall thought... If JSON were a subset of STON ... or possibly STON would recognize JSON and parse despite the different syntax...we'd have a nice multipurpose parser/writer that could speak JSON or STON interchangeably...

Dale

----- Original Message -----
| From: "Sven Van Caekenberghe" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 10:35:49 AM
| Subject: Re: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
|
|
| On 24 Apr 2012, at 19:22, Dale Henrichs wrote:
|
| > Sven,
| >
| > I looked at your sample document in the pdf:
| >
| > TestDomainObject {
| >  #created : DateAndTime [ '2012-02-14T16:40:15+01:00' ],
| >  #modified : DateAndTime [ '2012-02-14T16:40:18+01:00' ],
| >  #integer : 39581,
| >  #float : 73.84789359463944,
| >  #description : 'This is a test',
| >  #color : #green,
| >  #tags : [
| >    #two,
| >    #beta,
| >    #medium
| >  ],
| >  #bytes : ByteArray [ 'afabfdf61d030f43eb67960c0ae9f39f' ],
| >  #boolean : false
| > }
| >
| > And this is very readable (to me)... not as many '[]' as I inferred
| > from reading the tests:).
| >
| > I take back my YAML comparison ... this has the nice flavor of
| > Smalltalk ...
| >
| > Dale
|
| Thanks!
|
| The main 'hack' is to extend JSON's list/array [] and map/dictionary
| {} with a very simple way to instanciate any class, by preceding
| them with a class name (or alias) and giving the control to
| instanciate to the class or instance. Furthermore, for Array and
| Dictionary these are optional. For the concrete representations
| choosen for say DateAndTime or ByteArray, human readability was the
| deciding factor. But any of these could be changed without hurting
| the concept.
|
| Note that there are also references and implicit object counting and
| graph tracing with a defined visit order. In a discussion with
| Mariano I kind of decided to no longer treat Strings as
| shareable/referenceable objects. This is not yet reflected in the
| document but it is in the code.
|
| Sven
|

Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Sven Van Caekenberghe

On 24 Apr 2012, at 21:05, Dale Henrichs wrote:

> Sven,
>
> Just an off-the-wall thought... If JSON were a subset of STON ... or possibly STON would recognize JSON and parse despite the different syntax...we'd have a nice multipurpose parser/writer that could speak JSON or STON interchangeably...
>
> Dale

That is a great idea, Dale, I never thought of that !

Apart from the fact that I used single quotes for strings in STON, it already works (and I could enhance the parser to accept both I guess).

Here is an example using a JSON REST call to convert IP addresses to countries:

*nix Shell:

$ curl http://easy.t3-platform.net/rest/geo-ip?address=205.140.197.101         
     
{"latitude":38.0,"address":"205.140.197.101","country":"US","longitude":-97.0}

Smalltalk:

ZnClient new
        systemPolicy;
        beOneShot;
        url: 'http://easy.t3-platform.net/rest/geo-ip';
        queryAt: 'address' put: '205.140.197.101';
        accept: ZnMimeType applicationJson;
        contentReader: [ :entity |
                STON fromString: (entity contents replaceAll: $" with: $') ];
        get.  
       
a Dictionary('address'->'205.140.197.101' 'country'->'US' 'latitude'->38.0 'longitude'->-97.0 )


Thx,

Sven
Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Sven Van Caekenberghe

On 24 Apr 2012, at 21:40, Sven Van Caekenberghe wrote:

> Apart from the fact that I used single quotes for strings in STON, it already works (and I could enhance the parser to accept both I guess).

---
Name: Ston-Core-SvenVanCaekenberghe.27
Author: SvenVanCaekenberghe
Time: 24 April 2012, 10:13:07 pm
UUID: dd7c16b2-950c-4246-bf60-130de447cddf
Ancestors: Ston-Core-SvenVanCaekenberghe.26

extended the parser to allow either single or double quotes as String delimiters which gives us JSON backward compatibilty ! (thanks Dale Henrichs for this great idea).
---



Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Dale Henrichs
Very cool!

Let's see ... export the pharo implementation of STON in Cypress package format (using FileTree), then port STON to other dialects and voila:) the next generation of Cypress can use STON in place of JSON!

Dale

----- Original Message -----
| From: "Sven Van Caekenberghe" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 1:14:08 PM
| Subject: Re: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
|
|
| On 24 Apr 2012, at 21:40, Sven Van Caekenberghe wrote:
|
| > Apart from the fact that I used single quotes for strings in STON,
| > it already works (and I could enhance the parser to accept both I
| > guess).
|
| ---
| Name: Ston-Core-SvenVanCaekenberghe.27
| Author: SvenVanCaekenberghe
| Time: 24 April 2012, 10:13:07 pm
| UUID: dd7c16b2-950c-4246-bf60-130de447cddf
| Ancestors: Ston-Core-SvenVanCaekenberghe.26
|
| extended the parser to allow either single or double quotes as String
| delimiters which gives us JSON backward compatibilty ! (thanks Dale
| Henrichs for this great idea).
| ---
|
|
|
|

Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

Stéphane Ducasse
In reply to this post by Sven Van Caekenberghe

On Apr 24, 2012, at 1:26 PM, Sven Van Caekenberghe wrote:

> OK, just to get started, let try to write a spec for SimpleLiteralArray, this is based on JSON (and my own STON, not ready for public consumption). It will soon become clear that this is slightly harder than expected, IMO a writer is needed as well.
>
> S y n t a x
>
> value
>  primitive-value
>  list
>  nil
> primitive-value
>  number
>  true
>  false
>  symbol
>  string
>  classname

In a literal array a class name is just a symbol

#(Foo) first
        #Foo

> list
>  #()
>  #( elements )
> elements
>  value
>  value , elements
> string
>  ''
>  ' chars '
> chars
>  char
>  char chars
> char
>  any-printable-ASCII-character-
>    except-"-or-\
>  \"
>  \\
>  \/
>  \b
>  \f
>  \n
>  \r
>  \t
>  \u four-hex-digits

why do we need that?

> symbol
>  # chars-limited
>  # ' chars '
> chars-limited
>  char-limited
>  char-limited chars-limited
> char-limited
>  a-z A-Z 0-9 - _ . /
> classname
>  uppercase-alpha-char alphanumeric-char

there is not need for classname as element
classname is just a pair that the interpretation should handle if necessary


> number
>  int
>  int frac
>  int exp
>  int frac exp
> int
>  digit
>  digit1-9 digits
>  - digit
>  - digit1-9 digits
> frac
>  . digits
> exp
>  e digits
> digits
>  digit
>  digit digits
> e
>  e
>  e+
>  e-
>  E
>  E+
>  E-
>
>
> Questions remain:
>
> - is the escaping OK ?

I do not know but in literal array there is no escaping. Now if you want to put a cr into a string you probably need that

> - what about symbols without a leading # ?

#(foo) -> #(#foo)
#(true nil false 1 'n') -> #(true nil false 1 'n')

> - what should be allowed in unquoted symbols ?
I did not get the question

> - what should be allowed for classnames ?
Nothing

> - what about namespaces ?
Nothing no need I do not think that they are in literal arrays. (may be in VW but I did not check)

> - do we allow infinite precision integers ?
Yes

> - maybe the toplevel should always be an array ?

I would say so else this is not a literalArray but a literal.


>
>
> Now: shoot!
>
>
> Sven
>
>
> --
> Sven Van Caekenberghe
> http://stfx.eu
> Smalltalk is the Red Pill
>
>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

Dale Henrichs
Stef,

Which is more important:

  implementation or functionality?

You appear to be fixated on a particular solution and seem to be ignoring the actual problems.

Dale

----- Original Message -----
| From: "Stéphane Ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 3:13:20 PM
| Subject: Re: [Pharo-project] SimpleLiteralArray Spec
|
|
| On Apr 24, 2012, at 1:26 PM, Sven Van Caekenberghe wrote:
|
| > OK, just to get started, let try to write a spec for
| > SimpleLiteralArray, this is based on JSON (and my own STON, not
| > ready for public consumption). It will soon become clear that this
| > is slightly harder than expected, IMO a writer is needed as well.
| >
| > S y n t a x
| >
| > value
| >  primitive-value
| >  list
| >  nil
| > primitive-value
| >  number
| >  true
| >  false
| >  symbol
| >  string
| >  classname
|
| In a literal array a class name is just a symbol
|
| #(Foo) first
| #Foo
|
| > list
| >  #()
| >  #( elements )
| > elements
| >  value
| >  value , elements
| > string
| >  ''
| >  ' chars '
| > chars
| >  char
| >  char chars
| > char
| >  any-printable-ASCII-character-
| >    except-"-or-\
| >  \"
| >  \\
| >  \/
| >  \b
| >  \f
| >  \n
| >  \r
| >  \t
| >  \u four-hex-digits
|
| why do we need that?
|
| > symbol
| >  # chars-limited
| >  # ' chars '
| > chars-limited
| >  char-limited
| >  char-limited chars-limited
| > char-limited
| >  a-z A-Z 0-9 - _ . /
| > classname
| >  uppercase-alpha-char alphanumeric-char
|
| there is not need for classname as element
| classname is just a pair that the interpretation should handle if
| necessary
|
|
| > number
| >  int
| >  int frac
| >  int exp
| >  int frac exp
| > int
| >  digit
| >  digit1-9 digits
| >  - digit
| >  - digit1-9 digits
| > frac
| >  . digits
| > exp
| >  e digits
| > digits
| >  digit
| >  digit digits
| > e
| >  e
| >  e+
| >  e-
| >  E
| >  E+
| >  E-
| >
| >
| > Questions remain:
| >
| > - is the escaping OK ?
|
| I do not know but in literal array there is no escaping. Now if you
| want to put a cr into a string you probably need that
|
| > - what about symbols without a leading # ?
|
| #(foo) -> #(#foo)
| #(true nil false 1 'n') -> #(true nil false 1 'n')
|
| > - what should be allowed in unquoted symbols ?
| I did not get the question
|
| > - what should be allowed for classnames ?
| Nothing
|
| > - what about namespaces ?
| Nothing no need I do not think that they are in literal arrays. (may
| be in VW but I did not check)
|
| > - do we allow infinite precision integers ?
| Yes
|
| > - maybe the toplevel should always be an array ?
|
| I would say so else this is not a literalArray but a literal.
|
|
| >
| >
| > Now: shoot!
| >
| >
| > Sven
| >
| >
| > --
| > Sven Van Caekenberghe
| > http://stfx.eu
| > Smalltalk is the Red Pill
| >
| >
| >
| >
| >
|
|
|

Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Hannes Hirzel
In reply to this post by Dale Henrichs
On 4/24/12, Dale Henrichs <[hidden email]> wrote:
> Very cool!
>
> Let's see ... export the pharo implementation of STON in Cypress package
> format (using FileTree), then port STON to other dialects and voila:) the
> next generation of Cypress can use STON in place of JSON!
>
> Dale

Yes, this is promising.

First of all STON has a good name
'Smalltalk Object Notation'. People who know JSON will be instantly
familiar with the general idea. Sven writes this in the document
attached as a PDF at the beginning of this thread. [1]

Second
     STON offers more than JSON
     shared objects and cycles in the object graph are supported in STON.
     more data types

Third
   STON seems to be superset of JSON. Sven writes above that he
implemented this today in a new version.

Fourth
    It is literal Smalltalk if we accept a dictionary type {} as done
in Amber Smalltalk.

-- Hannes


[1] Sven Van Caekenberghe
If you have ever seen JSON (Javascript Object Notation) 1 , you will be
instantly familiar with Smaltalk Object Notation. It uses similar
primitive values, with the addition of a symbol type. Some details are
slightly
different though. For object values an optional class tag precedes a
representation that is either a list of values or a map of keys and
values. Exactly
how each class is represented is up to the implementation, although this
document suggests some conventions for well known classes.



> ----- Original Message -----
> | From: "Sven Van Caekenberghe" <[hidden email]>
> | To: [hidden email]
> | Sent: Tuesday, April 24, 2012 1:14:08 PM
> | Subject: Re: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
> |
> |
> | On 24 Apr 2012, at 21:40, Sven Van Caekenberghe wrote:
> |
> | > Apart from the fact that I used single quotes for strings in STON,
> | > it already works (and I could enhance the parser to accept both I
> | > guess).
> |
> | ---
> | Name: Ston-Core-SvenVanCaekenberghe.27
> | Author: SvenVanCaekenberghe
> | Time: 24 April 2012, 10:13:07 pm
> | UUID: dd7c16b2-950c-4246-bf60-130de447cddf
> | Ancestors: Ston-Core-SvenVanCaekenberghe.26
> |
> | extended the parser to allow either single or double quotes as String
> | delimiters which gives us JSON backward compatibilty ! (thanks Dale
> | Henrichs for this great idea).
> | ---
> |
> |
> |
> |
>
>

Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Hannes Hirzel
Additional context information for people who will come late to this
thread: We are discussing a human-readable-format, so neither
something like SIXX, nor a binary data exchange format.

On 4/25/12, H. Hirzel <[hidden email]> wrote:

> On 4/24/12, Dale Henrichs <[hidden email]> wrote:
>> Very cool!
>>
>> Let's see ... export the pharo implementation of STON in Cypress package
>> format (using FileTree), then port STON to other dialects and voila:) the
>> next generation of Cypress can use STON in place of JSON!
>>
>> Dale
>
> Yes, this is promising.
>
> First of all STON has a good name
> 'Smalltalk Object Notation'. People who know JSON will be instantly
> familiar with the general idea. Sven writes this in the document
> attached as a PDF at the beginning of this thread. [1]
>
> Second
>      STON offers more than JSON
>      shared objects and cycles in the object graph are supported in STON.
>      more data types
>
> Third
>    STON seems to be superset of JSON. Sven writes above that he
> implemented this today in a new version.
>
> Fourth
>     It is literal Smalltalk if we accept a dictionary type {} as done
> in Amber Smalltalk.
>
> -- Hannes
>
>
> [1] Sven Van Caekenberghe
> If you have ever seen JSON (Javascript Object Notation) 1 , you will be
> instantly familiar with Smaltalk Object Notation. It uses similar
> primitive values, with the addition of a symbol type. Some details are
> slightly
> different though. For object values an optional class tag precedes a
> representation that is either a list of values or a map of keys and
> values. Exactly
> how each class is represented is up to the implementation, although this
> document suggests some conventions for well known classes.
>
>
>
>> ----- Original Message -----
>> | From: "Sven Van Caekenberghe" <[hidden email]>
>> | To: [hidden email]
>> | Sent: Tuesday, April 24, 2012 1:14:08 PM
>> | Subject: Re: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
>> |
>> |
>> | On 24 Apr 2012, at 21:40, Sven Van Caekenberghe wrote:
>> |
>> | > Apart from the fact that I used single quotes for strings in STON,
>> | > it already works (and I could enhance the parser to accept both I
>> | > guess).
>> |
>> | ---
>> | Name: Ston-Core-SvenVanCaekenberghe.27
>> | Author: SvenVanCaekenberghe
>> | Time: 24 April 2012, 10:13:07 pm
>> | UUID: dd7c16b2-950c-4246-bf60-130de447cddf
>> | Ancestors: Ston-Core-SvenVanCaekenberghe.26
>> |
>> | extended the parser to allow either single or double quotes as String
>> | delimiters which gives us JSON backward compatibilty ! (thanks Dale
>> | Henrichs for this great idea).
>> | ---
>> |
>> |
>> |
>> |
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: SimpleLiteralArray Spec

Stéphane Ducasse
In reply to this post by Dale Henrichs

On Apr 25, 2012, at 12:17 AM, Dale Henrichs wrote:

> Stef,
>
> Which is more important:
>
>  implementation or functionality?

functionality: how to store meta data of class or method. So far I do not see why I need ston.

Now if you tell me that you need a textual serializer then this is another story and scenario.
But this is not the one we need to store meta data of class and method.

I'm not that approximate to mix the two.

Stef

>
> You appear to be fixated on a particular solution and seem to be ignoring the actual problems.
>
> Dale
>
> ----- Original Message -----
> | From: "Stéphane Ducasse" <[hidden email]>
> | To: [hidden email]
> | Sent: Tuesday, April 24, 2012 3:13:20 PM
> | Subject: Re: [Pharo-project] SimpleLiteralArray Spec
> |
> |
> | On Apr 24, 2012, at 1:26 PM, Sven Van Caekenberghe wrote:
> |
> | > OK, just to get started, let try to write a spec for
> | > SimpleLiteralArray, this is based on JSON (and my own STON, not
> | > ready for public consumption). It will soon become clear that this
> | > is slightly harder than expected, IMO a writer is needed as well.
> | >
> | > S y n t a x
> | >
> | > value
> | >  primitive-value
> | >  list
> | >  nil
> | > primitive-value
> | >  number
> | >  true
> | >  false
> | >  symbol
> | >  string
> | >  classname
> |
> | In a literal array a class name is just a symbol
> |
> | #(Foo) first
> | #Foo
> |
> | > list
> | >  #()
> | >  #( elements )
> | > elements
> | >  value
> | >  value , elements
> | > string
> | >  ''
> | >  ' chars '
> | > chars
> | >  char
> | >  char chars
> | > char
> | >  any-printable-ASCII-character-
> | >    except-"-or-\
> | >  \"
> | >  \\
> | >  \/
> | >  \b
> | >  \f
> | >  \n
> | >  \r
> | >  \t
> | >  \u four-hex-digits
> |
> | why do we need that?
> |
> | > symbol
> | >  # chars-limited
> | >  # ' chars '
> | > chars-limited
> | >  char-limited
> | >  char-limited chars-limited
> | > char-limited
> | >  a-z A-Z 0-9 - _ . /
> | > classname
> | >  uppercase-alpha-char alphanumeric-char
> |
> | there is not need for classname as element
> | classname is just a pair that the interpretation should handle if
> | necessary
> |
> |
> | > number
> | >  int
> | >  int frac
> | >  int exp
> | >  int frac exp
> | > int
> | >  digit
> | >  digit1-9 digits
> | >  - digit
> | >  - digit1-9 digits
> | > frac
> | >  . digits
> | > exp
> | >  e digits
> | > digits
> | >  digit
> | >  digit digits
> | > e
> | >  e
> | >  e+
> | >  e-
> | >  E
> | >  E+
> | >  E-
> | >
> | >
> | > Questions remain:
> | >
> | > - is the escaping OK ?
> |
> | I do not know but in literal array there is no escaping. Now if you
> | want to put a cr into a string you probably need that
> |
> | > - what about symbols without a leading # ?
> |
> | #(foo) -> #(#foo)
> | #(true nil false 1 'n') -> #(true nil false 1 'n')
> |
> | > - what should be allowed in unquoted symbols ?
> | I did not get the question
> |
> | > - what should be allowed for classnames ?
> | Nothing
> |
> | > - what about namespaces ?
> | Nothing no need I do not think that they are in literal arrays. (may
> | be in VW but I did not check)
> |
> | > - do we allow infinite precision integers ?
> | Yes
> |
> | > - maybe the toplevel should always be an array ?
> |
> | I would say so else this is not a literalArray but a literal.
> |
> |
> | >
> | >
> | > Now: shoot!
> | >
> | >
> | > Sven
> | >
> | >
> | > --
> | > Sven Van Caekenberghe
> | > http://stfx.eu
> | > Smalltalk is the Red Pill
> | >
> | >
> | >
> | >
> | >
> |
> |
> |
>


Reply | Threaded
Open this post in threaded view
|

Re: STON (was Re: SimpleLiteralArray Spec)

Göran Krampe
In reply to this post by Dale Henrichs
On 04/24/2012 07:28 PM, Dale Henrichs wrote:
> Sven,
>
> For tODE I am very interested having something with this kind of potential ... shipping "real" ST objects back and forth between Amber and Pharo or GemStone has some real appeal...
>
> So Goran, the STON format looks somewhat similar to Tirade ... what would Tirade look like for the same data set?

Hmmm, let's see.

> | TestDomainObject {
> |   #created : DateAndTime [ '2012-02-14T16:40:15+01:00' ],
> |   #modified : DateAndTime [ '2012-02-14T16:40:18+01:00' ],
> |   #integer : 39581,
> |   #float : 73.84789359463944,
> |   #description : 'This is a test',
> |   #color : #green,
> |   #tags : [
> |     #two,
> |     #beta,
> |     #medium
> |   ],
> |   #bytes : ByteArray [ 'afabfdf61d030f43eb67960c0ae9f39f' ],
> |   #boolean : false
> | }

It is an interesting question since Tirade is not meant to be parsed by
a generic parser like JSON or STON. Tirade is a series of Smalltalk
messages (unary or keyword) with arguments limited to literals (although
with some subtle additions to regular Smalltalk literals, like for
example Associations).

These messages give Tirade semantics and thus we do not normally add
"types" to the Tirade input.

So the above could be encoded as a single Tirade message like this:

created: '2012-02-14T16:40:15+01:00'
modified: '2012-02-14T16:40:18+01:00'
integer: 39581
float: 73.84789359463944
description: 'This is a test'
color: #green
tags: #(two beta medium)
bytes: 'afabfdf61d030f43eb67960c0ae9f39f'
boolean: false.

Note that the above syntactically is a regular Smalltalk keyword message
with no receiver to the left and ended with a "." (mandatory in Tirade).
Also, the tags could also be represented as {#two . #beta . #medium} but
although the syntax is like dynamic arrays in Pharo that does not mean
expressions are allowed in Tirade - only literals.

Normally there would be a subclass of TiradeReader that implements the
above message and then does the appropriate instantiations for #created,
#modified and #bytes given the string input for those arguments. This
method could look something like this (just pseudo code):

created: created
modified: modified
integer: int
float: float
description: description
color: color
tags: tags
bytes: bytes
boolean: boolean

result := TestDomainObject new
                created: (TimeStamp fromString: created);
                modified: (TimeStamp fromString: modified).
result integer: int; float: float; description: description;
        color: color; tags: tags;
        bytes: bytes asByteArray; boolean: boolean


If one instead uses a plain TiradeParser to parse the above it will of
course not have an implementation of the above method and will instead
by default produce an OrderedCollection of "messages" where each message
is an Association with the keyword selector as key and an array of
arguments as the value. So running TiradeParser>>parse: on the above
input gives:

an
OrderedCollection(#created:modified:integer:float:description:color:tags:bytes:boolean:->#('2012-02-14T16:40:15+01:00'
'2012-02-14T16:40:18+01:00' 39581 73.84789359463944 'This is a test'
#green #(true #beta #medium) 'afabfdf61d030f43eb67960c0ae9f39f' false))

Finally a totally different approach is to encode the above object as a
nested array with Associations, thus mimicking JSON/STON and it could
look like this:

testdomainobject: {
        #created -> '2012-02-14T16:40:15+01:00'.
        #modified -> '2012-02-14T16:40:18+01:00'.
        #integer -> 39581.
        #float -> 73.84789359463944.
        #description -> 'This is a test'.
        #color -> #green.
        #tags -> #(two beta medium).
        #bytes -> 'afabfdf61d030f43eb67960c0ae9f39f'.
        #boolean -> false
}.

NOTE: The -> syntax is added to Tirade as a "literal syntax" although it
is in fact a binary message in Pharo. This will create an Array with
Associations in it.

Again, if the reader implements the selector #testdomainobject: it will
know what to do with the argument. Again, using only TiradeParser it
would parse into:

an OrderedCollection(#testdomainobject:->{((Array new: 9) at: 1 put:
(#created->'2012-02-14T16:40:15+01:00'); at: 2 put:
(#modified->'2012-02-14T16:40:18+01:00'); at: 3 put: (#integer->39581);
at: 4 put: (#float->73.84789359463944); at: 5 put: (#description->'This
is a test'); at: 6 put: (#color->#green); at: 7 put: (#tags->#(true
#beta #medium )); at: 8 put:
(#bytes->'afabfdf61d030f43eb67960c0ae9f39f'); at: 9 put:
(#boolean->false); yourself)})

Oh, funky. Obviously that is how print-it behaves on an Array of
Assocations :)

Hopefully this gives a better idea of how Tirade works. Again, the three
articles on Tirade from my blog explains more details:

http://goran.krampe.se/blog/Squeak/Tirade.rdoc
http://goran.krampe.se/blog/Squeak/Tirade2.rdoc
http://goran.krampe.se/blog/Squeak/Tirade3.rdoc

NOTE: The part on "receiver juggling" was way crazy, forget that :)

regards, Göran

12