An alternative FFI/Parser proposal

> Hi Folks -
>
> Regardless of Stephane's insults I'm actually interested in making  
> progress on the issue of how to integrate the FFI into the parser  
> in general. So here is an alternative proposal:
>
> Instead of insisting that the FFI syntax needs change, let's assume  
> that there may be differences in the pragma formats. That has been  
> true in the past and in my experience it is likely that it will be  
> true and valuable in the future, too (like for example, if somebody  
> wants to use blocks in a pragma).
>
> A good way of dealing with these differences would be if a client  
> could register specific pragmas which are parsed in a client  
> dependent manner. So that, for example, the FFI would register  
> <apicall:> and <cdecl:> for FFI specs and Tweak may register <on:>  
> for parsing method triggers[*]. In this case, Parser could simply  
> invoke the proper client for a registered pragma, pass it a stream  
> and let it decide what to do. Given a sufficient interface for  
> client and Parser, this would leave the entire responsibility with  
> the client instead of Parser, but Parser could still provide a  
> default implementation.
>
> [*] The main reason for Tweak to parse triggers separately is to  
> provide semantic checks. For example, the <on: event in: signaler>  
> annotation requires the signaler to be a field of the receiver.  
> Being able to hook into the parse in this way can be useful for  
> other kinds of semantic assertions.
>
> Unfortunately, there are also a couple of gotchas with the  
> proposal: Most importantly, it requires that any parser can hand  
> off the current input stream to a client and continue after it's  
> getting the stream back. Not sure if all parsers could easily do  
> that. In addition, the client would need to have sufficient access  
> to the parser to perform whatever action it requires, including  
> (potentially) correction or error handling. This may be tricky  
> since the existing parsers have no common protocols for that.  
> Lastly there is an issue with what exactly should happen if we're  
> trying to parse a pragma but lack the proper support (like parsing  
> an FFI spec without the FFI being present). I'd rather have it if  
> that the parser is aware of such problems and can raise an error  
> instead of trying to get the user to use something that won't work  
> anyway, but this may not be possible.
>
> In any case, this is a clear alternative that offers the same  
> benefits of the original proposal ("clean", "extensible") while  
> avoiding fundamentally breaking the FFI for no good reason. If  
> anyone were to implement that proposal it would certainly find my  
> support.
>
> Cheers,
>   - Andreas
>

> Hi Folks -
>
> Regardless of Stephane's insults I'm actually interested in making  
> progress on the issue of how to integrate the FFI into the parser in  
> general. So here is an alternative proposal:
>
> Instead of insisting that the FFI syntax needs change, let's assume that  
> there may be differences in the pragma formats. That has been true in  
> the past and in my experience it is likely that it will be true and  
> valuable in the future, too (like for example, if somebody wants to use  
> blocks in a pragma).
>
> A good way of dealing with these differences would be if a client could  
> register specific pragmas which are parsed in a client dependent manner.

> So that, for example, the FFI would register <apicall:> and <cdecl:> for  
> FFI specs and Tweak may register <on:> for parsing method triggers[*].  
> In this case, Parser could simply invoke the proper client for a  
> registered pragma, pass it a stream and let it decide what to do. Given  
> a sufficient interface for client and Parser, this would leave the  
> entire responsibility with the client instead of Parser, but Parser  
> could still provide a default implementation.
>
> [*] The main reason for Tweak to parse triggers separately is to provide  
> semantic checks. For example, the <on: event in: signaler> annotation  
> requires the signaler to be a field of the receiver. Being able to hook  
> into the parse in this way can be useful for other kinds of semantic  
> assertions.
>
> Unfortunately, there are also a couple of gotchas with the proposal:  
> Most importantly, it requires that any parser can hand off the current  
> input stream to a client and continue after it's getting the stream  
> back. Not sure if all parsers could easily do that.

>> A good way of dealing with these differences would be if a client could
>> register specific pragmas which are parsed in a client dependent manner.
>> So that, for example, the FFI would register <apicall:> and <cdecl:> for
>> FFI specs and Tweak may register <on:> for parsing method triggers[*].
>> In this case, Parser could simply invoke the proper client for a
>> registered pragma, pass it a stream and let it decide what to do. Given
>> a sufficient interface for client and Parser, this would leave the
>> entire responsibility with the client instead of Parser, but Parser
>> could still provide a default implementation.
>
> Nice idea, though ...
>
>> Unfortunately, there are also a couple of gotchas with the proposal:
>
> + support for different parser/compiler will be difficult
> + breaks encapsulation of the parser/compiler
> + de-compilation won't work out-of-the-box
> + inconsistent syntax: the current FFI syntax makes no sense, it is
> neither valid C nor valid Smalltalk code, I suspect it was done like
> this to simplify the parsing)
>
> Cheers,
> Lukas
>

>>> Unfortunately, there are also a couple of gotchas with the proposal:  
>>> Most importantly, it requires that any parser can hand off the current  
>>> input stream to a client and continue after it's getting the stream  
>>> back. Not sure if all parsers could easily do that.
>>  I can already see all the skip: -2's when the stream is passed ;-) How  
>> about passing an interface to parser (a specialized contract which  
>> delegates). That'd be a clean implementation.
>
> Yes, that would be the nicer solution if parsers can agree on a common  
> protocol. However, I'm not sure that's doable (and it seemed to  
> complicate the overall Gestalt of the proposal) so that in the mindset  
> of proposing the simplest thing that all parties could agree on I just  
> proposed a plain input stream.

> Brent Pinkney wrote:
>> This is a bit off topic, but is there some documentation or  
>> explanation
>> on why Tweak introduced the <on: > pragma to handle events.
>
> Because it binds the event more closely to the method and allows  
> better support in the tools. It is possible to use Tweak without  
> these annotations, simply by using something like here:
>
> Foo>>initialize
>     self startScript: #onMumble when:{self. #mumble}.
>
>> I am really interested in what problem it solves and why it is  
>> better than
>> other alternatives.
>
> Mostly you want to be able to put the method and its trigger in the  
> same place. Why separate the two if the sole purpose is that the  
> trigger fire that method. At which point you get added tool support  
> (the events are displayed in the browser) additional development  
> support (changing a trigger is represented live in all existing  
> instances) and it just simplifies code.
>
>> This is not a criticism at all - I really do want to know for my  
>> own didactic purposes.
>
> For didactic purposes it may be worthwhile to consider the  
> alternatives. There are two primary ones:
> a) Use method with fixed names, like in Morphic. When you implement  
> #mouseDown this method gets called. If you also implement  
> #handlesMouseDown: that is. As if the system couldn't be smart  
> enough to figure that out by itself ;-)
> b) Use some wiring mechanism, like #on:send:to:. This also requires  
> at least one additional method and it is not updated for instances  
> when you change it.
> In either case you have to go to at least two separate places to  
> combine the trigger with the method. With annotations you get all  
> the advantages at once:
> - you only need to touch a single place
> - the annotation defines the wiring to the event
> - the system updates the instances accordingly
>
> With the "extended" syntax <on: eventName in: signaler> the  
> advantages are even more obvious. When you use that form the system  
> automatically updates the event wiring when the value of the  
> signaler changes. For example, if you were to watch the "value"  
> field of an arbitrary object you might do something like here:
>
> onTargetValueChanged
>     <on: valueChanged in: target>
>
> And as you change the "target" the method gets automatically  
> pointed to the new signaler.
>
> Cheers,
>   - Andreas
>
>

>
> A good way of dealing with these differences would be if a client could
> register specific pragmas which are parsed in a client dependent manner.
> So that, for example, the FFI would register <apicall:> and <cdecl:> for
> FFI specs and Tweak may register <on:> for parsing method triggers[*].
> In this case, Parser could simply invoke the proper client for a
> registered pragma, pass it a stream and let it decide what to do. Given
> a sufficient interface for client and Parser, this would leave the
> entire responsibility with the client instead of Parser, but Parser
> could still provide a default implementation.

>
> [*] The main reason for Tweak to parse triggers separately is to provide
> semantic checks. For example, the <on: event in: signaler> annotation
> requires the signaler to be a field of the receiver. Being able to hook
> into the parse in this way can be useful for other kinds of semantic
> assertions.
>
> Unfortunately, there are also a couple of gotchas with the proposal:
> Most importantly, it requires that any parser can hand off the current
> input stream to a client and continue after it's getting the stream
> back. Not sure if all parsers could easily do that. In addition, the

>
> In any case, this is a clear alternative that offers the same benefits
> of the original proposal ("clean", "extensible") while avoiding
> fundamentally breaking the FFI for no good reason. If anyone were to
> implement that proposal it would certainly find my support.
>
> Cheers,
>    - Andreas
>
>

> Hi all,
> Having had a chance to ponder this, I think it could be evolved into a good
> solution.
>
> ..snip...
>> Instead of insisting that the FFI syntax needs change, let's assume that
>> there may be differences in the pragma formats. That has been true in
>> the past and in my experience it is likely that it will be true and
>> valuable in the future, too (like for example, if somebody wants to use
>> blocks in a pragma).
>
> Given that the FFI syntax isn't going to change; it is a *certainty* that there
> are differences in pragma formats. So any solution should allow for that. If FFI
> is allowed to have its own format, then my imaginary XYZ extension should also
> be allowed to.
>
>> A good way of dealing with these differences would be if a client could
>> register specific pragmas which are parsed in a client dependent manner.
>> So that, for example, the FFI would register <apicall:> and <cdecl:> for
>> FFI specs and Tweak may register <on:> for parsing method triggers[*].
>> In this case, Parser could simply invoke the proper client for a
>> registered pragma, pass it a stream and let it decide what to do. Given
>> a sufficient interface for client and Parser, this would leave the
>> entire responsibility with the client instead of Parser, but Parser
>> could still provide a default implementation.
>
> The implementation could be simplified by allowed only the tokens after the
> first keyword to vary, rather than everything in a <..>. This is enough for FFI,
> and retains at least part of the pragma syntax. e.g.
>     <a: i * j  b: k > is allowed.
>     <a: i > j b: k>  is not allowed (can't have embedded > )
>     <a b c d> is not allowed (can only be free form if first token is keyword)
>
> (If an extension wants to allow embedded > then it can specify that they are
> doubled i.e. >> )
>
>  The parser can now collect the data from the stream for each <...> construct.
> i.e. record the start point, skip all tokens until '>' is reached, and then
> store the source from start to end. These are then recorded as AngleConstructs
> (or whatever).
>
> For example,
>     <a: i * j b: k> produces this AngleConstruct
>         AngleConstruct(
>             selector: #a:
>             source: 'a: i * j b: k' )
>
> (Note that the b: keyword part does NOT form part of the selector)
>
> Having parsed all "pragmas" as free form angleConstructs, the parser then
> decides what to do with each one.
>
> angleConstructs do:[angle |
>     self compilerExtensions
>          detect:[extension |
>             (handled := extension canHandle: angle)
>                 ifTrue:[
>                     (extension compile: angle for: self)
>                     (realPragma := extension pragmaFor: angle)
>                         ifNotNil:[pragmas addLast: realPragma]].
>             handled]
>          ifNone:["error - no handler for this <...> "]].
>
> with some extra error handling etc.
> The key point is that there is a sequence of compilerExtensions, and so there is
> a precedence. Currently the order will be
>     PrimitiveExtensionHandler
>     FFIExtensionHandler
>     PragmaExtensionHandler
>
> Each handler performs its own parse of the AngleConstruct's source.
>
> Each handler can determine whether to record each of the angle constructs it
> handles as a real Pragma object (or specialized kind of Pragma etc).
> So, for example,  <a: i * j b: k> could be stored as...
>     MyPragma(selector: #a:b: , arguments: #( 'i * j' 'k' ))
>  or as
>     MyPragma(selector: #a: , arguments: #( 'i * j' #b: 'k'))
>
> If a handler chooses to add a Pragma (or specialized form of Pragma), then all
> the searching senders free stuff will be utilised.
> This could be extended to allow a handler to add any number of real Pragmas. For
> example,
>     <a: 1 ; b:2 ; c: 3> might result in Pragmas (#a #(1)) , (#b #(2)) and (#c
> #(3))
>
> FFI compilation (rightly) fails if FFI is not installed, because the call syntax
> is such that an FFI call can never be a valid Pragma. The distinct syntax can
> therefore be seen as an advantage, rather than a disadvantage.
>
> As Andreas has previously stated (in this, or another, thread) the specialized
> Pragmas can also deal with decompilation.
>
>
>> [*] The main reason for Tweak to parse triggers separately is to provide
>> semantic checks. For example, the <on: event in: signaler> annotation
>> requires the signaler to be a field of the receiver. Being able to hook
>> into the parse in this way can be useful for other kinds of semantic
>> assertions.
>>
>> Unfortunately, there are also a couple of gotchas with the proposal:
>> Most importantly, it requires that any parser can hand off the current
>> input stream to a client and continue after it's getting the stream
>> back. Not sure if all parsers could easily do that. In addition, the
>
> My suggestion avoids that problem. I am not sure what the cost is - efficiency
> etc.
>
>> client would need to have sufficient access to the parser to perform
>> whatever action it requires, including (potentially) correction or error
>> handling. This may be tricky since the existing parsers have no common
>> protocols for that. Lastly there is an issue with what exactly should
>
> I've only sketched out a protocol in one direction. I haven't considered how the
> extensions talk to the parser. But I don't think it would be too complex. What
> should the interface be?
>
>> happen if we're trying to parse a pragma but lack the proper support
>> (like parsing an FFI spec without the FFI being present). I'd rather
>> have it if that the parser is aware of such problems and can raise an
>> error instead of trying to get the user to use something that won't work
>> anyway, but this may not be possible.
>
> I think that the 'trick' is to ensure that a non-pragma syntax is used, so that
> compilation fails when the extension is missing (assuming, of course, that if
> the extension is missing, then so is its parser/compiler extensions)
>
>
> Cheers,
> Andy
>> In any case, this is a clear alternative that offers the same benefits
>> of the original proposal ("clean", "extensible") while avoiding
>> fundamentally breaking the FFI for no good reason. If anyone were to
>> implement that proposal it would certainly find my support.
>>
>> Cheers,
>>    - Andreas
>>
>>
>
>
>