SimpleLiteralArray Spec

Stef,

I mentioned several weeks ago that the Cypress implementation takes advantage of the fact that the JSON parser produces dictionaries ... this is a bit of functionality that I called out as critical ... STON is the first Smalltalk-centric text format that I've seen that has a nice syntax for producing Dictionaries.

Of course, I think it is a bonus that you can use it to represent specific class instances as that can prove to be very useful.

Of course I am a Smalltalk programmer, so if you give me STON, instead of meticulously constructing a JSON structure that matches the fields of the classes (for each dialect) I think that we would be inclined to ask the class to create a STON representation and dump that to disk ...

So the immediate value of STON would be that we'd be sending messages to objects instead of translating formats .... the messages that we send will end up creating a nice human-readable format ... win-win.

Of course for multiple dialects we agreed that we would use the same set of field names for the class attributes so that a dialect could create the class using the fields that it recognized ... so that would imply some level of standardization of the STON field names to be used, but that shouldn't be too hard since there aren't that many fields to monkey with for classes ...

Dale

----- Original Message -----
| From: "Stéphane Ducasse" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 3:36:47 PM
| Subject: Re: [Pharo-project] SimpleLiteralArray Spec
|
|
| On Apr 25, 2012, at 12:17 AM, Dale Henrichs wrote:
|
| > Stef,
| >
| > Which is more important:
| >
| >  implementation or functionality?
|
| functionality: how to store meta data of class or method. So far I do
| not see why I need ston.
|
| Now if you tell me that you need a textual serializer then this is
| another story and scenario.
| But this is not the one we need to store meta data of class and
| method.
|
| I'm not that approximate to mix the two.
|
| Stef
| >
| > You appear to be fixated on a particular solution and seem to be
| > ignoring the actual problems.
| >
| > Dale
| >
| > ----- Original Message -----
| > | From: "Stéphane Ducasse" <[hidden email]>
| > | To: [hidden email]
| > | Sent: Tuesday, April 24, 2012 3:13:20 PM
| > | Subject: Re: [Pharo-project] SimpleLiteralArray Spec
| > |
| > |
| > | On Apr 24, 2012, at 1:26 PM, Sven Van Caekenberghe wrote:
| > |
| > | > OK, just to get started, let try to write a spec for
| > | > SimpleLiteralArray, this is based on JSON (and my own STON, not
| > | > ready for public consumption). It will soon become clear that
| > | > this
| > | > is slightly harder than expected, IMO a writer is needed as
| > | > well.
| > | >
| > | > S y n t a x
| > | >
| > | > value
| > | >  primitive-value
| > | >  list
| > | >  nil
| > | > primitive-value
| > | >  number
| > | >  true
| > | >  false
| > | >  symbol
| > | >  string
| > | >  classname
| > |
| > | In a literal array a class name is just a symbol
| > |
| > | #(Foo) first
| > | #Foo
| > |
| > | > list
| > | >  #()
| > | >  #( elements )
| > | > elements
| > | >  value
| > | >  value , elements
| > | > string
| > | >  ''
| > | >  ' chars '
| > | > chars
| > | >  char
| > | >  char chars
| > | > char
| > | >  any-printable-ASCII-character-
| > | >    except-"-or-\
| > | >  \"
| > | >  \\
| > | >  \/
| > | >  \b
| > | >  \f
| > | >  \n
| > | >  \r
| > | >  \t
| > | >  \u four-hex-digits
| > |
| > | why do we need that?
| > |
| > | > symbol
| > | >  # chars-limited
| > | >  # ' chars '
| > | > chars-limited
| > | >  char-limited
| > | >  char-limited chars-limited
| > | > char-limited
| > | >  a-z A-Z 0-9 - _ . /
| > | > classname
| > | >  uppercase-alpha-char alphanumeric-char
| > |
| > | there is not need for classname as element
| > | classname is just a pair that the interpretation should handle if
| > | necessary
| > |
| > |
| > | > number
| > | >  int
| > | >  int frac
| > | >  int exp
| > | >  int frac exp
| > | > int
| > | >  digit
| > | >  digit1-9 digits
| > | >  - digit
| > | >  - digit1-9 digits
| > | > frac
| > | >  . digits
| > | > exp
| > | >  e digits
| > | > digits
| > | >  digit
| > | >  digit digits
| > | > e
| > | >  e
| > | >  e+
| > | >  e-
| > | >  E
| > | >  E+
| > | >  E-
| > | >
| > | >
| > | > Questions remain:
| > | >
| > | > - is the escaping OK ?
| > |
| > | I do not know but in literal array there is no escaping. Now if
| > | you
| > | want to put a cr into a string you probably need that
| > |
| > | > - what about symbols without a leading # ?
| > |
| > | #(foo) -> #(#foo)
| > | #(true nil false 1 'n') -> #(true nil false 1 'n')
| > |
| > | > - what should be allowed in unquoted symbols ?
| > | I did not get the question
| > |
| > | > - what should be allowed for classnames ?
| > | Nothing
| > |
| > | > - what about namespaces ?
| > | Nothing no need I do not think that they are in literal arrays.
| > | (may
| > | be in VW but I did not check)
| > |
| > | > - do we allow infinite precision integers ?
| > | Yes
| > |
| > | > - maybe the toplevel should always be an array ?
| > |
| > | I would say so else this is not a literalArray but a literal.
| > |
| > |
| > | >
| > | >
| > | > Now: shoot!
| > | >
| > | >
| > | > Sven
| > | >
| > | >
| > | > --
| > | > Sven Van Caekenberghe
| > | > http://stfx.eu
| > | > Smalltalk is the Red Pill
| > | >
| > | >
| > | >
| > | >
| > | >
| > |
| > |
| > |
| >
|
|
|

Okay, so Tirade isn't a generic parser for a well-defined syntax like STON and it looks like it really shines when it comes to doing dynamic interpretation of streams of messages .... now I better understand how Tirade can be applied to deltas or creating executable logs that can be replayed ...

At this point in time for disk-based structures that humans and computers are expected to understand it seems that a well-defined syntax is useful.

Dale
----- Original Message -----
| From: "Göran Krampe" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 3:56:36 PM
| Subject: Re: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
|
| On 04/24/2012 07:28 PM, Dale Henrichs wrote:
| > Sven,
| >
| > For tODE I am very interested having something with this kind of
| > potential ... shipping "real" ST objects back and forth between
| > Amber and Pharo or GemStone has some real appeal...
| >
| > So Goran, the STON format looks somewhat similar to Tirade ... what
| > would Tirade look like for the same data set?
|
| Hmmm, let's see.
|
| > | TestDomainObject {
| > |   #created : DateAndTime [ '2012-02-14T16:40:15+01:00' ],
| > |   #modified : DateAndTime [ '2012-02-14T16:40:18+01:00' ],
| > |   #integer : 39581,
| > |   #float : 73.84789359463944,
| > |   #description : 'This is a test',
| > |   #color : #green,
| > |   #tags : [
| > |     #two,
| > |     #beta,
| > |     #medium
| > |   ],
| > |   #bytes : ByteArray [ 'afabfdf61d030f43eb67960c0ae9f39f' ],
| > |   #boolean : false
| > | }
|
| It is an interesting question since Tirade is not meant to be parsed
| by
| a generic parser like JSON or STON. Tirade is a series of Smalltalk
| messages (unary or keyword) with arguments limited to literals
| (although
| with some subtle additions to regular Smalltalk literals, like for
| example Associations).
|
| These messages give Tirade semantics and thus we do not normally add
| "types" to the Tirade input.
|
| So the above could be encoded as a single Tirade message like this:
|
| created: '2012-02-14T16:40:15+01:00'
| modified: '2012-02-14T16:40:18+01:00'
| integer: 39581
| float: 73.84789359463944
| description: 'This is a test'
| color: #green
| tags: #(two beta medium)
| bytes: 'afabfdf61d030f43eb67960c0ae9f39f'
| boolean: false.
|
| Note that the above syntactically is a regular Smalltalk keyword
| message
| with no receiver to the left and ended with a "." (mandatory in
| Tirade).
| Also, the tags could also be represented as {#two . #beta . #medium}
| but
| although the syntax is like dynamic arrays in Pharo that does not
| mean
| expressions are allowed in Tirade - only literals.
|
| Normally there would be a subclass of TiradeReader that implements
| the
| above message and then does the appropriate instantiations for
| #created,
| #modified and #bytes given the string input for those arguments. This
| method could look something like this (just pseudo code):
|
| created: created
| modified: modified
| integer: int
| float: float
| description: description
| color: color
| tags: tags
| bytes: bytes
| boolean: boolean
|
| result := TestDomainObject new
| created: (TimeStamp fromString: created);
| modified: (TimeStamp fromString: modified).
| result integer: int; float: float; description: description;
| color: color; tags: tags;
| bytes: bytes asByteArray; boolean: boolean
|
|
| If one instead uses a plain TiradeParser to parse the above it will
| of
| course not have an implementation of the above method and will
| instead
| by default produce an OrderedCollection of "messages" where each
| message
| is an Association with the keyword selector as key and an array of
| arguments as the value. So running TiradeParser>>parse: on the above
| input gives:
|
| an
| OrderedCollection(#created:modified:integer:float:description:color:tags:bytes:boolean:->#('2012-02-14T16:40:15+01:00'
| '2012-02-14T16:40:18+01:00' 39581 73.84789359463944 'This is a test'
| #green #(true #beta #medium) 'afabfdf61d030f43eb67960c0ae9f39f'
| false))
|
| Finally a totally different approach is to encode the above object as
| a
| nested array with Associations, thus mimicking JSON/STON and it could
| look like this:
|
| testdomainobject: {
| #created -> '2012-02-14T16:40:15+01:00'.
| #modified -> '2012-02-14T16:40:18+01:00'.
| #integer -> 39581.
| #float -> 73.84789359463944.
| #description -> 'This is a test'.
| #color -> #green.
| #tags -> #(two beta medium).
| #bytes -> 'afabfdf61d030f43eb67960c0ae9f39f'.
| #boolean -> false
| }.
|
| NOTE: The -> syntax is added to Tirade as a "literal syntax" although
| it
| is in fact a binary message in Pharo. This will create an Array with
| Associations in it.
|
| Again, if the reader implements the selector #testdomainobject: it
| will
| know what to do with the argument. Again, using only TiradeParser it
| would parse into:
|
| an OrderedCollection(#testdomainobject:->{((Array new: 9) at: 1 put:
| (#created->'2012-02-14T16:40:15+01:00'); at: 2 put:
| (#modified->'2012-02-14T16:40:18+01:00'); at: 3 put:
| (#integer->39581);
| at: 4 put: (#float->73.84789359463944); at: 5 put:
| (#description->'This
| is a test'); at: 6 put: (#color->#green); at: 7 put: (#tags->#(true
| #beta #medium )); at: 8 put:
| (#bytes->'afabfdf61d030f43eb67960c0ae9f39f'); at: 9 put:
| (#boolean->false); yourself)})
|
| Oh, funky. Obviously that is how print-it behaves on an Array of
| Assocations :)
|
| Hopefully this gives a better idea of how Tirade works. Again, the
| three
| articles on Tirade from my blog explains more details:
|
| http://goran.krampe.se/blog/Squeak/Tirade.rdoc
| http://goran.krampe.se/blog/Squeak/Tirade2.rdoc
| http://goran.krampe.se/blog/Squeak/Tirade3.rdoc
|
| NOTE: The part on "receiver juggling" was way crazy, forget that :)
|
| regards, Göran
|
|

Göran

Thank you for this illustration.

http://goran.krampe.se/blog/Squeak/Tirade.rdoc

I understand that

1) the Tirade parser parses a subset of regular Smalltalk messages.
2) the receiver is not given as it is assumed to be always the Tirade
parser object.
3) the Tirade parser acts like a builder for objects?
4) It solves problems JSON and STON do not.

I think this idea is very worth to pursue.

--Hannes

On 4/25/12, Göran Krampe <[hidden email]> wrote:

On 04/25/2012 01:25 AM, H. Hirzel wrote:
> Göran
>
> Thank you for this illustration.
>
> http://goran.krampe.se/blog/Squeak/Tirade.rdoc
>
> I understand that
>
> 1) the Tirade parser parses a subset of regular Smalltalk messages.

Exactly. It doesn't do binary messages. And of course, it doesn't do any
expressions.

> 2) the receiver is not given as it is assumed to be always the Tirade
> parser object.

Right, but if you use the TiradeReader (which is a bit more
experimental) it maintains a stack of objects and will send the next
Tirade message to the object on the top of the stack. Then one can
register a set of "control messages" and those are always sent to the
object first on the stack - which is the TiradeReader itself.

As I said, this is an experiment on a generic mechanism for being able
to construct complex objects from a series of Tirade messages.

> 3) the Tirade parser acts like a builder for objects?

The parser actually only parses the Tirade messages, it doesn't try to
build anything. But your own subclass of it should implement these
messages and typically build something.

> 4) It solves problems JSON and STON do not.
>
> I think this idea is very worth to pursue.

The elegancy of messaging is quite powerful and makes it easier to
construct a file format that is less fragile and has very little structure.

regards, Göran

On 04/25/2012 01:11 AM, Dale Henrichs wrote:
> Okay, so Tirade isn't a generic parser for a well-defined syntax like STON

Well... :) I am not sure if you read my whole post but it does do nested
arrays and associatons etc so it seems to match STON fairly well, if we
disregard the class mechanism of STON.

But yes, it is primarily constructed as a "serializer for Smalltalk
messages" and not as a "serializer for Smalltalk objects".

> and it looks like it really shines when it comes to doing dynamic interpretation of streams of messages ....

Right.

> now I better understand how Tirade can be applied to deltas or creating executable logs that can be replayed ...

Yes, although in DeltaStreams I do not use it like that - I use it to
serialize objects, but Deltas consists of lots of different Change
objects and each such Change object is serialized using a specific
Tirade message. Thus the reader simply implements all those messages and
creates the appropriate objects.

One *very* nice aspect of this is that we do not encode classes in the
actual files, the reader decides which class to instantiate. Thus the
files are much more robust against change.

In that sense I am not that fond of putting classes explicitly in the
STON file.

> At this point in time for disk-based structures that humans and computers are expected to understand it seems that a well-defined syntax is useful.
>
> Dale

Sure. And at least it was fun to explain a bit about Tirade. And I ended
up adding support for literal arrays :)

regards, Göran

Agreed,

I think there are valid use cases for both STON and Tirade ... there is some overlap and where there is overlap one or the other can be used ... then again each has some unique characteristics ... I'm glad that I've learned more about Tirade in the process, because it is much clearer to me where Tirade can be applied ...

Dale

----- Original Message -----
| From: "Göran Krampe" <[hidden email]>
| To: [hidden email]
| Sent: Tuesday, April 24, 2012 5:17:26 PM
| Subject: Re: [Pharo-project] STON (was Re:  SimpleLiteralArray Spec)
|
| On 04/25/2012 01:11 AM, Dale Henrichs wrote:
| > Okay, so Tirade isn't a generic parser for a well-defined syntax
| > like STON
|
| Well... :) I am not sure if you read my whole post but it does do
| nested
| arrays and associatons etc so it seems to match STON fairly well, if
| we
| disregard the class mechanism of STON.
|
| But yes, it is primarily constructed as a "serializer for Smalltalk
| messages" and not as a "serializer for Smalltalk objects".
|
| > and it looks like it really shines when it comes to doing dynamic
| > interpretation of streams of messages ....
|
| Right.
|
| > now I better understand how Tirade can be applied to deltas or
| > creating executable logs that can be replayed ...
|
| Yes, although in DeltaStreams I do not use it like that - I use it to
| serialize objects, but Deltas consists of lots of different Change
| objects and each such Change object is serialized using a specific
| Tirade message. Thus the reader simply implements all those messages
| and
| creates the appropriate objects.
|
| One *very* nice aspect of this is that we do not encode classes in
| the
| actual files, the reader decides which class to instantiate. Thus the
| files are much more robust against change.
|
| In that sense I am not that fond of putting classes explicitly in the
| STON file.
|
| > At this point in time for disk-based structures that humans and
| > computers are expected to understand it seems that a well-defined
| > syntax is useful.
| >
| > Dale
|
| Sure. And at least it was fun to explain a bit about Tirade. And I
| ended
| up adding support for literal arrays :)
|
| regards, Göran
|
|

Sven Van Caekenberghe wrote: The human readability is a strong point of JSON & STON, but is there any
requirement for "human edit-ability" by which I mean in a text editor?
In which case the referencing objects by counting "OrderedCollection [
Point [1, 2], @2, @2 ]" would be fragile, since insertion of an object
via text editor would offset all references.  Something like
"OrderedCollection@1 [ Point@2 [1, 2], @2, @2 ]" would be more robust.  
However if you expect to never need to edit the STON file in a text
editor, this point is mute.

The downside of this idea is a bit more cluttered look.  However I
notice that evaluating #123 in Workspace returns an integer rather than
a ByteSymbol.  So it would seem that #123 would never be written out in
STON format (from Pharo), which means "OrderedCollection#1 [ Point#2 [1,
2], #2, #2 ]" might be a less cluttered option, or perhaps a mix like
"OrderedCollection#1 [ Point#2 [1, 2], @2, @2 ]" doesn't detract too
much from readability.

Actually I think this enhances readability anyhow, since if there are a
lot of object references in the STON file, in a text viewer I would have
to need to manually count them to follow the references.

cheers -ben

On 25 Apr 2012, at 07:46, [hidden email] wrote:

> Is it just me or is there a problem?
>
> Can't get STON from SS3. Doesn't looks like down tough.

Apparently the package names themselves have different capitalization:

Gofer it
        url: 'http://ss3.gemstone.com/ss/STON';
        package: 'Ston-Core';
        package: 'Ston-Tests';
        load.

Yes, I am confused as well.

Sven

Ben,

On 25 Apr 2012, at 05:32, Ben Coman wrote:

> The human readability is a strong point of JSON & STON, but is there any requirement for "human edit-ability" by which I mean in a text editor?
> In which case the referencing objects by counting "OrderedCollection [ Point [1, 2], @2, @2 ]" would be fragile, since insertion of an object via text editor would offset all references.  Something like "OrderedCollection@1 [ Point@2 [1, 2], @2, @2 ]" would be more robust.  However if you expect to never need to edit the STON file in a text editor, this point is mute.
>
> The downside of this idea is a bit more cluttered look.  However I notice that evaluating #123 in Workspace returns an integer rather than a ByteSymbol.  So it would seem that #123 would never be written out in STON format (from Pharo), which means "OrderedCollection#1 [ Point#2 [1, 2], #2, #2 ]" might be a less cluttered option, or perhaps a mix like "OrderedCollection#1 [ Point#2 [1, 2], @2, @2 ]" doesn't detract too much from readability.
>
> Actually I think this enhances readability anyhow, since if there are a lot of object references in the STON file, in a text viewer I would have to need to manually count them to follow the references.

Thanks for the feedback, I will think about your idea of explicit numbering.

However, a format with numbered references will never be very human friendly I guess.
If the goal is to have like a config file, I would make sure there are no such references.

Sven

Ben Coman wrote:
>
>
> However I notice that evaluating #123 in Workspace returns an integer
> rather than a ByteSymbol.  


I meant also to poll how compatible the behaviour of #123 evaluating to
SmallInteger rather than ByteSymbol is across Smalltalk dialects...

Dne 25. 04. 2012 00:20, piše H. Hirzel:

> First of all STON has a good name
> 'Smalltalk Object Notation'. People who know JSON will be instantly
> familiar with the general idea.

Exactly!



--
Janko Mivšek
Aida/Web
Smalltalk Web Application Server
http://www.aidaweb.si

Ben Coman wrote:
> Sven Van Caekenberghe wrote:
> The human readability is a strong point of JSON & STON, but is there any
> requirement for "human edit-ability" by which I mean in a text editor?
> In which case the referencing objects by counting "OrderedCollection [
> Point [1, 2], @2, @2 ]" would be fragile, since insertion of an object
> via text editor would offset all references. Something like
> "OrderedCollection@1 [ Point@2 [1, 2], @2, @2 ]" would be more robust.
> However if you expect to never need to edit the STON file in a text
> editor, this point is mute.

Shouldn't relative numbering help here? Not a bulletproof solution, but
for simple changes, it suffices.

Herby