Pragma syntax

Hi Lukas,

----- Original Message -----
From: "Lukas Renggli" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Saturday, August 19, 2006 4:28 PM
Subject: Re: Re: Re: Re: Pragma syntax


I see. The scanner will naturally group <=  and <> into single tokens.
So, if binary selectors were allowed I would need to write < > 1> and < = #foo>.
Which isn't so bad.

No :) . But that doesn't mean that there aren't any.
I can't think of any good reason why I would want to create a method named
////\//\/\/\/\/\/\/\******&&&@@@@  , but the compiler doesn't stop me.
(Incidently, in Visual Smalltalk, and in VW, binary selectors are limited to 2
characters. I just don't get why).

>
> Note 1: I see the pragma selector as a message that can be sent to an
> object to configure it somehow, binary selectors usually have a
> slightly different meaning.

I don't get that.
Which object is #primitive: sent to, and how does it configure it?

>
> Note 2: It would be easy to allow pragmas like this <* 6>, <,
> 'world'>, basically one line of code that needs to be updated.

I would have thought that <* 6> would also need to be written < * 6> , in order
to parse correctly? Anyway. What you are saying is that it's easy to allow
binary selectors?

>
> > Excellent. I'll test it out once it's there.
>
> I published it to http://source.squeakfoundation.org/inbox, please
> have a look. The tests do all run, I only changed one method that does
> the parsing of the selector. The correction algorithm is the same as
> the one used to correct selectors of normal message sends in your
> code.

Thanks.
I've had a look, and it works great.

Cheers,
Andy

> > Note 1: I see the pragma selector as a message that can be sent to an
> > object to configure it somehow, binary selectors usually have a
> > slightly different meaning.
>
> I don't get that.
> Which object is #primitive: sent to, and how does it configure it?

No problem, you are not the only one :-(

After the compilation the message #primitive: with the given arguments
is performed on the parser. This is then changing the current method
to be a primitive. It works the same for #primitive:module: where
there is just slightly different code executed. Have a look at the
implementors of those two messages in a 3.9 image.

Now, how do I know what pragmas I should perform on the parser and
what pragmas not? It is hardcoded right now, but that could be solved
nicely as I suggested in my original mail that caused all this
discussion. So the parser-class could then declare some of its methods
(namely #primitive: and #primitive:module:, and also #apicall:module:
and #cdecl:module: if the FFI people would accept) as methods to be
called after the compilation. Very extensible. Very modular. Very much
oop. And no patching.

> > Note 2: It would be easy to allow pragmas like this <* 6>, <,
> > 'world'>, basically one line of code that needs to be updated.
>
> I would have thought that <* 6> would also need to be written < * 6> , in order
> to parse correctly? Anyway. What you are saying is that it's easy to allow
> binary selectors?

Yes you are right. #<* would probably be parsed as a binary selector
as well, I am not that much aware what exactly is considered a binary
selector in Squeak. I guess with spaces that would be fine. It is a
similar problem as they have in C++ with their nested template
declarations ;-)

If the community likes to have pragmas with a binary selector I can do
that. Users have to make sure that they put a space upfront.

> > I published it to http://source.squeakfoundation.org/inbox, please
> > have a look. The tests do all run, I only changed one method that does
> > the parsing of the selector. The correction algorithm is the same as
> > the one used to correct selectors of normal message sends in your
> > code.
>
> Thanks.
> I've had a look, and it works great.

Great, then we should maybe include it in the next version. It is only
a small change (to be clear: it adds a feature for the interactive
compiler mode, it doesn't change anything else). How to proceed?
Should I submit to Mantis? How does that work as 3.9 gamma is out now?

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

stéphane ducasse wrote:  > specific with it and query <ffi: module:> and do something else and not
 > query <myOwnPragmas:...>
>
> I would really like to know.

Sigh. That's what my first point was all about. Once more: Assuming an
extension like the FFI (or Sql) how would -without the FFI or Sql
support loaded- the compiler even know that <cdecl: 'void puts(char*)'>
has *meaning*? I.e., that it is not a pragma but that it needs to be
interpreted by some support code that isn't even present?

You, yourself were running into this problem. You didn't have the FFI
loaded. You tried to put in an FFI spec. (Fortunately) It didn't work
and raised your attention and let you to ask what is happening. Now
consider the situation that we would have used pragma syntax and that
the compiler would have silently accepted that "pragma". It wouldn't
have *looked* right but not *worked* right. But if you would've asked
even I would have no way of telling why "puts() doesn't work on your
machine".

Cheers,
   - Andreas

Lukas Renggli wrote:
Let's forget the syntax for a second since it ain't going to change. How
exactly would that work? Do you intend to make #primitive: work properly
or something else? My main reason for asking is that I tried to use
"self primitive: 120" in the FFI declaration but it didn't work.

Cheers,
   - Andreas

Tx I will have to digest that.

Stef

On 19 août 06, at 16:39, Andrew Tween wrote:

for 3.10 I guess :)
I did not had any time to fix the repository nor to look at the  
condenseSources bug.
I will give a try to look at it and hope adrian will be able to help  
there if I cannot find it.

Stef

On 19 août 06, at 20:28, Lukas Renggli wrote:

Ok I see.
This is why in VW Pragmas are declared first.
This way this would be solved. I will look in VW to see what they do.

> Let's forget the syntax for a second since it ain't going to change.

Amen, my god.

> How
> exactly would that work? Do you intend to make #primitive: work properly
> or something else? My main reason for asking is that I tried to use
> "self primitive: 120" in the FFI declaration but it didn't work.

Yes, that's because it is not implemented the way I proposed yet.
Right now everything is hardocded, patched, hacked, ... as it was
before introducing the pragmas.

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

(this is probably redundant with what was already said, I'm just
reformulating my way)

On 8/19/06, Andreas Raab <[hidden email]> wrote:
> 1. Primitives and FFI specs aren't pragmas

What about putting it the other way around: pragmas are syntactic
extensions (just like primitives and FFI). That would make pragmas a
particular kind of extension that the compiler (1) knows by default
and (2) treats in a relaxed way, since pragmas are more like comments
or optional metadata than instructions with an alien syntax.

I guess this is Andrew's <pragma: ... > proposition

BTW, what about
<Primitive at: 42>
<FFI call: ... module: ...>
<Pragma addMeToMenu: #thatOne withTitle: 'blah'>

each of these would tell the compiler to look for a plugin and send it a message
--
 Damien Pollet
 type less, do more

Lukas Renggli wrote:
>> Let's forget the syntax for a second since it ain't going to change.
>
> Amen, my god.

Getting personal now, are we? Anyway, since you haven't brought up any
new arguments I don't see any reason to change my position. In fact, I'm
still awaiting your response to these messages:

http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-August/107204.html
http://lists.squeakfoundation.org/pipermail/squeak-dev/2006-August/107243.html

Unfortunately, it does appear as if your only goal is indeed to change
the FFI syntax, but like I said this isn't going to happen without a
real benefit for the FFI users. Get used to it.

>> How
>> exactly would that work? Do you intend to make #primitive: work properly
>> or something else? My main reason for asking is that I tried to use
>> "self primitive: 120" in the FFI declaration but it didn't work.
>
> Yes, that's because it is not implemented the way I proposed yet.
> Right now everything is hardocded, patched, hacked, ... as it was
> before introducing the pragmas.

I don't understand this. Weren't you the guy who said how "clean and
extensible" this all is now? In any case, I was asking for how an
extension would register itself as a non-pragma primitive. If you were
planning on making Parser>>primitive: work properly this could be done
without affecting anything else and would certainly go a long ways to
simplify extension parsing (FFI or otherwise).

Cheers,
   - Andreas

> 1. Primitives, FFI-specs and some other extensions aren't pragmas.
> Treating them like pragmas is ignoring their entire raison d'etre.

I've never said primitives or FFI specs **are** pragmas, I only said
that they are ideally parsed and created using pragmas. It is solely a
compile-time thing, after that you have your compiled method.
Everything else doesn't make sense and you know that.

> 2. Unintelligible code should never be accepted by the compiler and we
> should make sure that syntactical extensions (FFI, Sql and other) are
> clearly marked as such.

I would also like to reserve a couple of selectors for special pragmas
as well as some message selectors in Object. The compiler should
prevent other people to use those selectors as long as my super secret
package is not loaded. These selectors belong to me and nobody should
accidently use or call them. I am the owner of those selector. Keep
away. ;-)

> 5. The <> escape syntax is one of the few extension points in Squeak.
> Loosing it is a shame.

You can still change the parser class for your own classes. This is a
much better extension point than to extend it by patching the parsers
(old parser, new parser, refactoring parser), decompilers,
syntax-highlighters, code competition engines, etc.

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

2006/8/19, Andreas Raab <[hidden email]>:
How is that different from the current situation? If I do:
puts: aString
    <cdecl: void 'puts' (char*)>
    ^ExternalFunction externalCallFailed "not self because it's not
implemented in Object"

I get: 'Unable to find function address'

How does that give me a way of telling why "puts() doesn't work on my
machine"?

Philippe

> >> Let's forget the syntax for a second since it ain't going to change.
> >
> > Amen, my god.
>
> Getting personal now, are we?

Indeed, I prefer to implement and/or change something instead of
talking endlessly.

If you write "it ain't going to change" and "this isn't going to
happen" there is no point in discussing with you. Sentences like these
are quite personal as well. At least it reads personal to me.

Yes in my imagination, it could be done "clean and extensible". It is
not done like that right now. That's why I proposed a change.

> In any case, I was asking for how an
> extension would register itself as a non-pragma primitive. If you were
> planning on making Parser>>primitive: work properly this could be done
> without affecting anything else and would certainly go a long ways to
> simplify extension parsing (FFI or otherwise).

Yes, could be done without affecting the current compiler behavior. It
would be a refactoring and it wouldn't change the way FFI is hacked
into the compiler. In the sense that everybody is doing his own dull
thing, change his own small world ...

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

Hi Lukas,
----- Original Message -----
From: "Lukas Renggli" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Saturday, August 19, 2006 7:28 PM
Subject: Re: Re: Re: Re: Re: Pragma syntax


> > > Note 1: I see the pragma selector as a message that can be sent to an
> > > object to configure it somehow, binary selectors usually have a
> > > slightly different meaning.
> >
> > I don't get that.
> > Which object is #primitive: sent to, and how does it configure it?
>
> No problem, you are not the only one :-(

:)

>
> After the compilation the message #primitive: with the given arguments
> is performed on the parser. This is then changing the current method
> to be a primitive. It works the same for #primitive:module: where
> there is just slightly different code executed. Have a look at the
> implementors of those two messages in a 3.9 image.

Ok. Yes, I see it now. thanks.

>
> Now, how do I know what pragmas I should perform on the parser and
> what pragmas not? It is hardcoded right now, but that could be solved
> nicely as I suggested in my original mail that caused all this
> discussion. So the parser-class could then declare some of its methods
> (namely #primitive: and #primitive:module:, and also #apicall:module:
> and #cdecl:module: if the FFI people would accept) as methods to be
> called after the compilation. Very extensible. Very modular. Very much
> oop. And no patching.

Yes. But, there are some potential drawbacks...
    1. I cannot write an extension which uses a name that is already in use by
the Parser.
        e.g. #notify: , #parse:class: etc.

    2. It might be a dangerous thing to use one of the method names defined in
Parser's superclasses. e.g. #error:

    3. No keyword methods can safely be added to the Parser in the future. The
new method's name might already be in use by an extension. A change to the
parser's interface (i.e. the selectors it implements/understands) could result
in an extension, and/or, the parser breaking.

Please see my post in the 'Re: An alternative FFI/Parser proposal' thread for a
possible alternative arrangement. What do you think?

Cheers,
Andy

..snip...

> Yes. But, there are some potential drawbacks...
>     1. I cannot write an extension which uses a name that is already in use by
> the Parser.
>         e.g. #notify: , #parse:class: etc.

Yes, but these methods need to be tagged, so the parser/compiler knows
that these should be called after compilation. All others pragma
selectors would remain unused and left for interpretation by your
application.

>     2. It might be a dangerous thing to use one of the method names defined in
> Parser's superclasses. e.g. #error:

I think it might make sense to call these methods on the parser, but
it could be any other class. It just needs to be a fixed place.

>     3. No keyword methods can safely be added to the Parser in the future. The
> new method's name might already be in use by an extension. A change to the
> parser's interface (i.e. the selectors it implements/understands) could result
> in an extension, and/or, the parser breaking.
>
> Please see my post in the 'Re: An alternative FFI/Parser proposal' thread for a
> possible alternative arrangement. What do you think?

I don't see a point in widening the allowed syntax within the angle
construct. Right now a pragma is using existing Smalltalk construct
that every newcomer can understand and use. Allowing different
syntaxes complicates things in different areas. The Smalltalk syntax
should remain minimal, and not get bloated with new things that nobody
will understand. There are already many examples where languages got
destroyed by adding XML, SQL, HTML, ... to the language itself.
Smalltalk provides different and better ways to change/extend its
language.

To summarize, these are basically two different views:

(1) Pragmas as a syntax extension, where on can put almost anything
between the angle brackets after an initial keyword selector defining
the syntax.

(2) Pragmas as a message send with literal arguments.

I am for the simple and powerful. And this is (2) in my opinion.

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch

Ha, I almost missed a syntax discussion while on vacation! Seems as  
religious as always ;-)

Am 20.08.2006 um 14:17 schrieb Lukas Renggli:

> To summarize, these are basically two different views:
>
> (1) Pragmas as a syntax extension, where on can put almost anything
> between the angle brackets after an initial keyword selector defining
> the syntax.

Not quite - primitives and FFI calls are not pragmas, but all three  
are syntactical extensions with possibly different rules.

Maybe this is just a difficulty in naming - you seem to call  
everything inside angle brackets a "pragma".

Anyway, for sake of discussion I'll stay with your meaning.

> (2) Pragmas as a message send with literal arguments.
>
> I am for the simple and powerful. And this is (2) in my opinion.

Slightly paraphrasing Andrew's suggestion, how about this:

(3) "Pragmas" are method annotations parsed as literal array.

Under this definition, current FFI syntax is entirely legal (just  
take an FFI declaration and replace <> with #()). Extension modules  
can hook into the compiler to process those "pragmas".

As simple and even more powerful than (2), I'd argue.

This is not a final concept, agreed, but somehow it feels odd to  
reduce everything to strings, as in your "unified" FFI pragma.

- Bert -

Lukas Renggli wrote:
>> 1. Primitives, FFI-specs and some other extensions aren't pragmas.
>> Treating them like pragmas is ignoring their entire raison d'etre.
>
> I've never said primitives or FFI specs **are** pragmas, I only said
> that they are ideally parsed and created using pragmas. It is solely a
> compile-time thing, after that you have your compiled method.
> Everything else doesn't make sense and you know that.

Ah, good. I was honestly not sure if this was what you meant. From my
POV it seemed as if you really think that primitives *are* pragmas and
that therefore FFI specs *should be* pragmas, too. I completely disagree
with that but as a position it's consistent that's why I spent that much
time on making clear that FFI specs are not pragmas.

We're making progress. Now that we've agreed on one key point let's go
to another. Under the assumption that FFI specs are pragmas it is pretty
clear that they should be in pragma format. Agreeing that they aren't
means that they *can* be but that they don't have to be (which is a much
weaker statement). I think you're trying to say the same by saying "they
are *ideally* parsed and created using pragmas". It seems to me we're
both agreeing that while it may have advantages to use pragma format
where possible (like for primitives) there is certainly no requirement
for that from a consistency perspective. Do we agree on that, too?

This isn't about reserving "super secret" selectors. This is about the
compiler being able to tell you "hey, you're missing something
big-time". And most of the time it won't be super-secret but rather
fairly well-known (like "dude - you need the Sql parser if you want to
use Sql statements") where when you see it you'll slap your forehead and
say "duh".

>> 5. The <> escape syntax is one of the few extension points in Squeak.
>> Loosing it is a shame.
>
> You can still change the parser class for your own classes. This is a
> much better extension point than to extend it by patching the parsers
> (old parser, new parser, refactoring parser), decompilers,
> syntax-highlighters, code competition engines, etc.

I disagree. The FFI is a great example for it - yes you could make it so
that the specs only work in subclasses of ExternalLibrary (IIRC, VW does
it that way) but it is really, really inconvenient from the user
perspective. The <> extension point is in particular useful if you have
a need to do extensions across the class hierarchy. Since I've used
both, I can testify that they both have their uses and those uses differ
significantly.

Cheers,
   - Andreas

Lukas Renggli wrote:
> Indeed, I prefer to implement and/or change something instead of
> talking endlessly.

That's of course up to you. I on the other hand, have to do these
discussions (and trust me, there are other things that I'd prefer to do)
simply because I'm the FFI maintainer.

>> Unfortunately, it does appear as if your only goal is indeed to change
>> the FFI syntax, but like I said this isn't going to happen without a
>> real benefit for the FFI users. Get used to it.
>
> If you write "it ain't going to change" and "this isn't going to
> happen" there is no point in discussing with you. Sentences like these
> are quite personal as well. At least it reads personal to me.

I apologize for that. I'm trying to make clear that my position hasn't
changed on these issues and that you would have a much, much easier time
to discuss these issues if, at least for the time being, you'd
acknowledge that there are reasons for preserving the existing syntax.

I'm really not sure what your goal is in this discussion but I've said
many times that I'm willing to discuss anything - except from a change
in the FFI syntax with no user benefit. If you can accept the latter for
the time being I think you'll find me much more agreeable.

> Yes in my imagination, it could be done "clean and extensible". It is
> not done like that right now. That's why I proposed a change.

Right. And I disagreed with the proposal and proposed an alternative
that would respect the FFI. To which I've only gotten some criticism but
no feasible alternative in return.

> Yes, could be done without affecting the current compiler behavior. It
> would be a refactoring and it wouldn't change the way FFI is hacked
> into the compiler. In the sense that everybody is doing his own dull
> thing, change his own small world ...

Like you proposed to me, you're completely free to fork. If you want to
change things mainstream you'll either have to learn to spend time
talking to convince other people or you'll have to learn how to get to
your goals in small steps. I don't see much of an alternative.

Cheers,
   - Andreas

Hi Lukas,
----- Original Message -----
From: "Lukas Renggli" <[hidden email]>
To: "The general-purpose Squeak developers list"
<[hidden email]>
Sent: Sunday, August 20, 2006 1:17 PM
Subject: Re: Re: Re: Re: Re: Re: Pragma syntax


> > Yes. But, there are some potential drawbacks...
> >     1. I cannot write an extension which uses a name that is already in use
by
> > the Parser.
> >         e.g. #notify: , #parse:class: etc.
>
> Yes, but these methods need to be tagged, so the parser/compiler knows
> that these should be called after compilation. All others pragma
> selectors would remain unused and left for interpretation by your
> application.
>
> >     2. It might be a dangerous thing to use one of the method names defined
in
> > Parser's superclasses. e.g. #error:
>
> I think it might make sense to call these methods on the parser, but
> it could be any other class. It just needs to be a fixed place.

Outside the Parser seems, to me, to be better. I am trying to implement some of
my suggestions at the moment. I think that if there is real code that can be
examined, then the discussion will become easier. I'll let you know when I have
something.

>
> >     3. No keyword methods can safely be added to the Parser in the future.
The
> > new method's name might already be in use by an extension. A change to the
> > parser's interface (i.e. the selectors it implements/understands) could
result
> > in an extension, and/or, the parser breaking.
> >
> > Please see my post in the 'Re: An alternative FFI/Parser proposal' thread
for a
> > possible alternative arrangement. What do you think?
>
> I don't see a point in widening the allowed syntax within the angle
> construct. Right now a pragma is using existing Smalltalk construct
> that every newcomer can understand and use.

Yes. But, right now FFI is using something else. And Andreas has said that he
won't change that.
So, currently, the allowed syntax is wider than you want it to be. And it is you
who is arguing for a change towards a narrower syntax.
Now if this discussion had occurred before FFI was created, then your viewpoint
would probably have been accepted. But, the fact remains that FFI exists, isn't
going to change, and so the question isn't what should have happened 5 years
ago, but what should happen now.


>Allowing different
> syntaxes complicates things in different areas. The Smalltalk syntax
> should remain minimal, and not get bloated with new things that nobody
> will understand. There are already many examples where languages got
> destroyed by adding XML, SQL, HTML, ... to the language itself.
> Smalltalk provides different and better ways to change/extend its
> language.

A cynic might suggest that Pragmas are a new thing that extends the syntax. And
given the discussion on "is a primitive a pragma?", there are already different
understandings of what they are.
My argument would be that if something new is going to be added to the language,
then make it as flexible, and powerful, and open as possible. If it is misused,
and people do stupid things with it, and create weird syntax, then so what? I
don't have to use their code if I don't want to. After all, we are talking about
optional extensions here.

>
> To summarize, these are basically two different views:
>
> (1) Pragmas as a syntax extension, where on can put almost anything
> between the angle brackets after an initial keyword selector defining
> the syntax.

Yes. But they aren't, necessarily, Pragmas. By definition. If the thing between
the brackets doesn't conform to the Pragma syntax, then it can't be a Pragma.

>
> (2) Pragmas as a message send with literal arguments.

Those things ARE Pragmas.

>
> I am for the simple and powerful. And this is (2) in my opinion.

I am arguing for the less restrictive, and the more powerful. Which is (1).
Actually, I am arguing for both (1) and (2). Or, rather, that (1) imples (2).
Give me (1) and I can still write Pragmas.
Give me (2) and I can no longer write FFI calls.

Cheers,
Andy

>
> Cheers,
> Lukas
>
> --
> Lukas Renggli
> http://www.lukas-renggli.ch
>
>

> We're making progress. Now that we've agreed on one key point let's go
> to another. Under the assumption that FFI specs are pragmas it is pretty
> clear that they should be in pragma format. Agreeing that they aren't
> means that they *can* be but that they don't have to be (which is a much
> weaker statement). I think you're trying to say the same by saying "they
> are *ideally* parsed and created using pragmas". It seems to me we're
> both agreeing that while it may have advantages to use pragma format
> where possible (like for primitives) there is certainly no requirement
> for that from a consistency perspective. Do we agree on that, too?

Yes, we agree: FFI declarations can be pragmas, but they don't have to
be. This is the status quo in 3.9.

Pragmas works well and they are well tested. There were no bug reports
or complaints as far as I know. I am using them daily. In the process
of this discussion I even improved the parsing of pragma selectors by
adding validation and auto-correction. So there is a valuable gain in
that area.

My initial goals were different though:

1. Simplify the parser, compiler, decompiler, syntax-highlighter,
auto-completor, editor, ... tool-chain by unifying the parsing of
pragmas, primitives and FFI calls.

2. Enhance the compiler, so that it can be easily extended without
having to patch its core methods. In other words: fully use the
existing pragma implementation in the compiler to turn the compiled
methods into a primitive or FFI call, if a defined meta-definition
(pragma) is given.

3. Give pragmas an easily understandable syntax that everybody can
trust on, that can be easily explained to newcomers and that give no
surprises if new packages are loaded. Personally I find it rather
strange to have a have a half-backed C declaration being parsed with a
Smalltalk parser.

> This isn't about reserving "super secret" selectors. This is about the
> compiler being able to tell you "hey, you're missing something
> big-time". And most of the time it won't be super-secret but rather
> fairly well-known (like "dude - you need the Sql parser if you want to
> use Sql statements") where when you see it you'll slap your forehead and
> say "duh".

If you want to be extensible you cannot know all the possible
extensions in advance. This is the whole point of extensiblilty.
Therefor it doesn't make sense to say "hey dude, you need extension XY
if you want to use this selector". Moreover the same selector might
mean something completely different in your context (image) than it
does in mine.

Cheers,
Lukas

--
Lukas Renggli
http://www.lukas-renggli.ch