Parser failure on FFI pragmas declaration in Pharo 5

Hi,

I stumbled upon what seems to me a strange issue in Pharo 5. The RBParser
fails to correctly parse the legacy FFI pragmas. This completely breaks down
the browser, the inspector and debugger (because as far as I understand all
use RBParser to correctly highlight syntax). I had the image crashed and
some red boxes at some point while insisting to inspect and debug. Overall
this is not a big issue but it raises some more general bells to me.

In order to reproduce this:
- load the official Pharo 5 (curl get.pharo.org/50+vm | bash)
- launch the image (./pharo-ui Pharo.image &)
- add the following repository
MCSmalltalkhubRepository
        owner: 'panuw'
        project: 'zeromq'
        user: ''
        password: ''
- load the last versions of ZeroMQ and ConfigurationOfZeroMQ (not sure if
the latter is needed)
- open a Nautilus Browser and look at the class method apiZmqBind:to: of the
ZmqApi class in the ZeroMQ package: you get a MessageNotUnderstood error
(receiver of keywords is nil). You can get past this by clicking on
"Abandon" but the source code is displayed in a corrupted way:
apiZmqBind: s
ocket to: endpoint <cdecl
- repeat a few times by looking at other methods until you get a red box:
then you cannot look at source code any more with this browser. If you are
obstinate and &quot;lucky&quot; you will succeed in crashing the image.
- you can pin the problem by running in a Playground
RBParser parseFaultyMethod: 'apiZmqBind: socket to: endpoint
        &lt;cdecl: long ''zmq_bind'' (ZmqApiSocket* char*) module:''zmq''>
        ^self externalCallFailed'.
and you'll see that the pragmas is not correctly parsed. (The root cause is
that the legacy adapter RBFFICallPragma does not follow the API defined by
its super class RBPragmaNode (selector, arguments, positions) and so is not
a properly defined node. I corrected the problem by computing and setting
the corresponding instance variables.)

1) As a beginner at Pharo, I find it difficult to deal with the various
versions of Pharo. ZeroMQ is the (only) Smalltalk-Pharo binding for zmq. It
dates back to Feb 2014 so I expected it to work in Pharo as of 3 years and a
half later (Pharo 6 dates back to June 2017).
I naively tried to load the package in a Pharo 6 image and it failed because
of a syntax error. As I had read a lot about the various FFI mechanisms, I
quickly understood that it must be because the FFI declarations in pragma
are not supported anymore.
I then loaded the package in a Pharo 5 image and I got the error that I
described. After finding the error and solving it, I guess that the FFI
declaration in pragma was barely supported in Pharo 5, which has already
switched to UFFI and that it is something dating back to Pharo 4. (I did not
try with Pharo 4 as I do not want to work with versions before 5).
Is there a way to know for a package what the compatible Pharo version is?
It seems that currently I have to look at dates, look at the features used
by the package and look for the history of development (fortunately the
mailing lists are easy to search) to understand which version is likely to
work.
Are not deprecations a bit too fast if a package written 3 years ago cannot
work in the latest Pharo version and trigger bugs in Pharo 5, which dates
back to May 2016 (so only a bit more than 2 years after)?
I find it a bit too fast as compared to mainstream languages. To my mind,
either deprecations should be slower or a version/dependencies system should
be there to help users.

2) Another question about versions: Pharo 6 is out since June, Pharo 7 is
under development. What is the status of Pharo 5? Already history or still
relevant?
I am asking because I corrected the problem of FFI declaration in pragma,
but it seems to me that it is not useful to publish this change as starting
from Pharo 6 this way to do FFI is not supported. So should I contribute? If
yes, how to "attach" the patch to Pharo 5?

3) As explained above, in Pharo 5, looking at the source trigger an error.
Even if this looks like a rare corner case, I think that the developer tools
should not trigger bugs when looking at source code, even less trigger a red
box in the source code viewer (in the browser, but the problem also occurs
--less strongly-- when looking at the object in an inspector: there should
not be "error printing" when it is only a syntax highlight problem). If the
code is malformed and the parser used to highlight syntax fails, there
should be a fallback such as the source code being displayed without any
highlight. It sends a very bad impression to have this kind of bugs when one
simply wants to look at code, not even running it.
I have not dug enough in this area of Pharo, but it seems to me that the
parser that is used to build the AST for code execution / method compilation
should not be the same as the parser used to highligh syntax. (Of course I
am not saying that there should be 2 distincts code base for the 2 parsers,
but they should at least run differently.) The first one must be strict with
errors as a malformed AST cannot be executed. The second one must be
lenient, as a malformed AST does not prevent to print the string of the
source code. Of course, at the end if the code is malformed there will be an
error at execution, but if the source code can be displayed even when it is
malformed, at least I have the opportunity to correct it so that it runs
correctly. (In this case, convert the old FFI pragma declaration into a
fficall:)
I may be missing something here but if this works the same in the most
up-to-date version of Pharo, the same kind of error might appear again.
What do you think?

4) A final remark: let us classify people as Beginner/Confirmed in
programming and B/C in Pharo (A BB is a beginner in programming and in
Pharo, a CC confirmed in both, a BC cannot exists and CB are those who
discover Pharo while knowing well other languages). Pharo seems to be great
for BB and CC. I went through the MOOC and the various books which are
great. My first steps in Pharo environment were great.
As a CC it seems to be great also as in the very small area of the system
where I took the time to drill into all the details, I could very easily
change things (and correct a bug), that would have been very difficult to
understand and change in a lot of other languages. Even hacking the VM seems
to be possible for a non-VM expert.
But I consider myself rather as a CB. As such I tend to try and do complex
things that I usually do in other languages and run into tricky problems.
These problems are rather dealt with and corrected by Pharo developers but
that as a user I would expect them to remain hidden to me or to be clearly
advertised in the docs. As compared to a BB, a CB is not going to stay in a
well delimited area where everything is smooth.
True, in a way it is a very strong incentive to become a Pharo expert! But I
am wondering if this aspect could be improved.

Thanks,
Bruno



--
View this message in context: http://forum.world.st/Parser-failure-on-FFI-pragmas-declaration-in-Pharo-5-tp4961737.html
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.

hi,

Old way to do FFI calls is no longer supported on Pharo, but this deprecation is very old (since Pharo 2). Now, in Pharo 4 we replaced the compiler (for OpalCompiler) and we no longer supported “pragma-like” calls, in part because they are “invalid” pragma calls (they do not agrees with pragma syntax) and in part because the way to go in pharo is using UFFI (before UFFI it was NB which was largely compatible).

I don’t know why ZeroMQ bindings are made using old format, but the way to advance them is to

> On 16 Aug 2017, at 23:31, bdurin <[hidden email]> wrote:
>
> Hi,
>
> I stumbled upon what seems to me a strange issue in Pharo 5. The RBParser
> fails to correctly parse the legacy FFI pragmas. This completely breaks down
> the browser, the inspector and debugger (because as far as I understand all
> use RBParser to correctly highlight syntax). I had the image crashed and
> some red boxes at some point while insisting to inspect and debug. Overall
> this is not a big issue but it raises some more general bells to me.
>
> In order to reproduce this:
> - load the official Pharo 5 (curl get.pharo.org/50+vm | bash)
> - launch the image (./pharo-ui Pharo.image &)
> - add the following repository
> MCSmalltalkhubRepository
>       owner: 'panuw'
>       project: 'zeromq'
>       user: ''
>       password: ''
> - load the last versions of ZeroMQ and ConfigurationOfZeroMQ (not sure if
> the latter is needed)
> - open a Nautilus Browser and look at the class method apiZmqBind:to: of the
> ZmqApi class in the ZeroMQ package: you get a MessageNotUnderstood error
> (receiver of keywords is nil). You can get past this by clicking on
> "Abandon" but the source code is displayed in a corrupted way:
> apiZmqBind: s
> ocket to: endpoint <cdecl
> - repeat a few times by looking at other methods until you get a red box:
> then you cannot look at source code any more with this browser. If you are
> obstinate and &quot;lucky&quot; you will succeed in crashing the image.
> - you can pin the problem by running in a Playground
> RBParser parseFaultyMethod: 'apiZmqBind: socket to: endpoint
>       &lt;cdecl: long ''zmq_bind'' (ZmqApiSocket* char*) module:''zmq''>
>       ^self externalCallFailed'.
> and you'll see that the pragmas is not correctly parsed. (The root cause is
> that the legacy adapter RBFFICallPragma does not follow the API defined by
> its super class RBPragmaNode (selector, arguments, positions) and so is not
> a properly defined node. I corrected the problem by computing and setting
> the corresponding instance variables.)
>
> 1) As a beginner at Pharo, I find it difficult to deal with the various
> versions of Pharo. ZeroMQ is the (only) Smalltalk-Pharo binding for zmq. It
> dates back to Feb 2014 so I expected it to work in Pharo as of 3 years and a
> half later (Pharo 6 dates back to June 2017).
> I naively tried to load the package in a Pharo 6 image and it failed because
> of a syntax error. As I had read a lot about the various FFI mechanisms, I
> quickly understood that it must be because the FFI declarations in pragma
> are not supported anymore.
> I then loaded the package in a Pharo 5 image and I got the error that I
> described. After finding the error and solving it, I guess that the FFI
> declaration in pragma was barely supported in Pharo 5, which has already
> switched to UFFI and that it is something dating back to Pharo 4. (I did not
> try with Pharo 4 as I do not want to work with versions before 5).
> Is there a way to know for a package what the compatible Pharo version is?
> It seems that currently I have to look at dates, look at the features used
> by the package and look for the history of development (fortunately the
> mailing lists are easy to search) to understand which version is likely to
> work.
> Are not deprecations a bit too fast if a package written 3 years ago cannot
> work in the latest Pharo version and trigger bugs in Pharo 5, which dates
> back to May 2016 (so only a bit more than 2 years after)?
> I find it a bit too fast as compared to mainstream languages. To my mind,
> either deprecations should be slower or a version/dependencies system should
> be there to help users.

Most of Pharo is largely compatible. Now, we cannot keep compatibility in some areas more than a couple of versions back because the effort of advance Pharo *and* keep compatibility is just too much.
- FFI changed a lot.
- Morphic changed something.
- Most of the rest is basically the same (just better).

> 2) Another question about versions: Pharo 6 is out since June, Pharo 7 is
> under development. What is the status of Pharo 5? Already history or still
> relevant?
> I am asking because I corrected the problem of FFI declaration in pragma,
> but it seems to me that it is not useful to publish this change as starting
> from Pharo 6 this way to do FFI is not supported. So should I contribute? If
> yes, how to "attach" the patch to Pharo 5?

Pharo5 is history.
We keep one version back (now Pharo6)
Again, a matter of effort and resources.

> 3) As explained above, in Pharo 5, looking at the source trigger an error.
> Even if this looks like a rare corner case, I think that the developer tools
> should not trigger bugs when looking at source code, even less trigger a red
> box in the source code viewer (in the browser, but the problem also occurs
> --less strongly-- when looking at the object in an inspector: there should
> not be "error printing" when it is only a syntax highlight problem). If the
> code is malformed and the parser used to highlight syntax fails, there
> should be a fallback such as the source code being displayed without any
> highlight. It sends a very bad impression to have this kind of bugs when one
> simply wants to look at code, not even running it.
> I have not dug enough in this area of Pharo, but it seems to me that the
> parser that is used to build the AST for code execution / method compilation
> should not be the same as the parser used to highligh syntax. (Of course I
> am not saying that there should be 2 distincts code base for the 2 parsers,
> but they should at least run differently.) The first one must be strict with
> errors as a malformed AST cannot be executed. The second one must be
> lenient, as a malformed AST does not prevent to print the string of the
> source code. Of course, at the end if the code is malformed there will be an
> error at execution, but if the source code can be displayed even when it is
> malformed, at least I have the opportunity to correct it so that it runs
> correctly. (In this case, convert the old FFI pragma declaration into a
> fficall:)
> I may be missing something here but if this works the same in the most
> up-to-date version of Pharo, the same kind of error might appear again.
> What do you think?

Sorry, but we will not accept old pragma format (as I said, is invalid… and ugly ;) ).
What I suggest is to rewrite the bindings of ZeroMQ to UFFI: it should be very straight forward and you will be contributing to the community in a way that will remain quite some years at least.
>
> 4) A final remark: let us classify people as Beginner/Confirmed in
> programming and B/C in Pharo (A BB is a beginner in programming and in
> Pharo, a CC confirmed in both, a BC cannot exists and CB are those who
> discover Pharo while knowing well other languages). Pharo seems to be great
> for BB and CC. I went through the MOOC and the various books which are
> great. My first steps in Pharo environment were great.
> As a CC it seems to be great also as in the very small area of the system
> where I took the time to drill into all the details, I could very easily
> change things (and correct a bug), that would have been very difficult to
> understand and change in a lot of other languages. Even hacking the VM seems
> to be possible for a non-VM expert.
> But I consider myself rather as a CB. As such I tend to try and do complex
> things that I usually do in other languages and run into tricky problems.
> These problems are rather dealt with and corrected by Pharo developers but
> that as a user I would expect them to remain hidden to me or to be clearly
> advertised in the docs. As compared to a BB, a CB is not going to stay in a
> well delimited area where everything is smooth.
> True, in a way it is a very strong incentive to become a Pharo expert! But I
> am wondering if this aspect could be improved.

thing is… non OO programmers will have problems to understand a pure OO language.
people working with Java, C#, C++ and others may think they do OO, but they don’t most of the time… then, switching paradigms is hard work.
even worst, smalltalk syntax is considered “alien” to people used to algol-based languages.

but we cannot do much more than we are trying to achieve in this area: make Pharo more compatible with “the rest of the world” when it make sense, but strongly stay in our “alieness” when it has sense (syntax, pureness, etc.).

Esteban

>
> Thanks,
> Bruno
>
>
>
> --
> View this message in context: http://forum.world.st/Parser-failure-on-FFI-pragmas-declaration-in-Pharo-5-tp4961737.html
> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>

On Thu, Aug 17, 2017 at 1:09 PM, Denis Kudriashov <[hidden email]> wrote:

Sorry, but we will not accept old pragma format (as I said, is invalid… and ugly ;) ).

But we will be able load old compiler (when it will be removed from standard image) to support such old code 

But then you should compile such code with the old compiler, not make opal compatible, no?

 

2017-08-17 11:48 GMT+02:00 Esteban Lorenzano <[hidden email]>:

hi,

Old way to do FFI calls is no longer supported on Pharo, but this deprecation is very old (since Pharo 2). Now, in Pharo 4 we replaced the compiler (for OpalCompiler) and we no longer supported “pragma-like” calls, in part because they are “invalid” pragma calls (they do not agrees with pragma syntax) and in part because the way to go in pharo is using UFFI (before UFFI it was NB which was largely compatible).

I don’t know why ZeroMQ bindings are made using old format, but the way to advance them is to

> On 16 Aug 2017, at 23:31, bdurin <[hidden email]> wrote:
>
> Hi,
>
> I stumbled upon what seems to me a strange issue in Pharo 5. The RBParser
> fails to correctly parse the legacy FFI pragmas. This completely breaks down
> the browser, the inspector and debugger (because as far as I understand all
> use RBParser to correctly highlight syntax). I had the image crashed and
> some red boxes at some point while insisting to inspect and debug. Overall
> this is not a big issue but it raises some more general bells to me.
>
> In order to reproduce this:
> - load the official Pharo 5 (curl get.pharo.org/50+vm | bash)
> - launch the image (./pharo-ui Pharo.image &)
> - add the following repository
> MCSmalltalkhubRepository
>       owner: 'panuw'
>       project: 'zeromq'
>       user: ''
>       password: ''
> - load the last versions of ZeroMQ and ConfigurationOfZeroMQ (not sure if
> the latter is needed)
> - open a Nautilus Browser and look at the class method apiZmqBind:to: of the
> ZmqApi class in the ZeroMQ package: you get a MessageNotUnderstood error
> (receiver of keywords is nil). You can get past this by clicking on
> "Abandon" but the source code is displayed in a corrupted way:
> apiZmqBind: s
> ocket to: endpoint <cdecl
> - repeat a few times by looking at other methods until you get a red box:
> then you cannot look at source code any more with this browser. If you are
> obstinate and &quot;lucky&quot; you will succeed in crashing the image.
> - you can pin the problem by running in a Playground
> RBParser parseFaultyMethod: 'apiZmqBind: socket to: endpoint
>       &lt;cdecl: long ''zmq_bind'' (ZmqApiSocket* char*) module:''zmq''>
>       ^self externalCallFailed'.
> and you'll see that the pragmas is not correctly parsed. (The root cause is
> that the legacy adapter RBFFICallPragma does not follow the API defined by
> its super class RBPragmaNode (selector, arguments, positions) and so is not
> a properly defined node. I corrected the problem by computing and setting
> the corresponding instance variables.)
>
> 1) As a beginner at Pharo, I find it difficult to deal with the various
> versions of Pharo. ZeroMQ is the (only) Smalltalk-Pharo binding for zmq. It
> dates back to Feb 2014 so I expected it to work in Pharo as of 3 years and a
> half later (Pharo 6 dates back to June 2017).
> I naively tried to load the package in a Pharo 6 image and it failed because
> of a syntax error. As I had read a lot about the various FFI mechanisms, I
> quickly understood that it must be because the FFI declarations in pragma
> are not supported anymore.
> I then loaded the package in a Pharo 5 image and I got the error that I
> described. After finding the error and solving it, I guess that the FFI
> declaration in pragma was barely supported in Pharo 5, which has already
> switched to UFFI and that it is something dating back to Pharo 4. (I did not
> try with Pharo 4 as I do not want to work with versions before 5).
> Is there a way to know for a package what the compatible Pharo version is?
> It seems that currently I have to look at dates, look at the features used
> by the package and look for the history of development (fortunately the
> mailing lists are easy to search) to understand which version is likely to
> work.
> Are not deprecations a bit too fast if a package written 3 years ago cannot
> work in the latest Pharo version and trigger bugs in Pharo 5, which dates
> back to May 2016 (so only a bit more than 2 years after)?
> I find it a bit too fast as compared to mainstream languages. To my mind,
> either deprecations should be slower or a version/dependencies system should
> be there to help users.

Most of Pharo is largely compatible. Now, we cannot keep compatibility in some areas more than a couple of versions back because the effort of advance Pharo *and* keep compatibility is just too much.
- FFI changed a lot.
- Morphic changed something.
- Most of the rest is basically the same (just better).

> 2) Another question about versions: Pharo 6 is out since June, Pharo 7 is
> under development. What is the status of Pharo 5? Already history or still
> relevant?
> I am asking because I corrected the problem of FFI declaration in pragma,
> but it seems to me that it is not useful to publish this change as starting
> from Pharo 6 this way to do FFI is not supported. So should I contribute? If
> yes, how to "attach" the patch to Pharo 5?

Pharo5 is history.
We keep one version back (now Pharo6)
Again, a matter of effort and resources.

> 3) As explained above, in Pharo 5, looking at the source trigger an error.
> Even if this looks like a rare corner case, I think that the developer tools
> should not trigger bugs when looking at source code, even less trigger a red
> box in the source code viewer (in the browser, but the problem also occurs
> --less strongly-- when looking at the object in an inspector: there should
> not be "error printing" when it is only a syntax highlight problem). If the
> code is malformed and the parser used to highlight syntax fails, there
> should be a fallback such as the source code being displayed without any
> highlight. It sends a very bad impression to have this kind of bugs when one
> simply wants to look at code, not even running it.
> I have not dug enough in this area of Pharo, but it seems to me that the
> parser that is used to build the AST for code execution / method compilation
> should not be the same as the parser used to highligh syntax. (Of course I
> am not saying that there should be 2 distincts code base for the 2 parsers,
> but they should at least run differently.) The first one must be strict with
> errors as a malformed AST cannot be executed. The second one must be
> lenient, as a malformed AST does not prevent to print the string of the
> source code. Of course, at the end if the code is malformed there will be an
> error at execution, but if the source code can be displayed even when it is
> malformed, at least I have the opportunity to correct it so that it runs
> correctly. (In this case, convert the old FFI pragma declaration into a
> fficall:)
> I may be missing something here but if this works the same in the most
> up-to-date version of Pharo, the same kind of error might appear again.
> What do you think?

Sorry, but we will not accept old pragma format (as I said, is invalid… and ugly ;) ).
What I suggest is to rewrite the bindings of ZeroMQ to UFFI: it should be very straight forward and you will be contributing to the community in a way that will remain quite some years at least.
>
> 4) A final remark: let us classify people as Beginner/Confirmed in
> programming and B/C in Pharo (A BB is a beginner in programming and in
> Pharo, a CC confirmed in both, a BC cannot exists and CB are those who
> discover Pharo while knowing well other languages). Pharo seems to be great
> for BB and CC. I went through the MOOC and the various books which are
> great. My first steps in Pharo environment were great.
> As a CC it seems to be great also as in the very small area of the system
> where I took the time to drill into all the details, I could very easily
> change things (and correct a bug), that would have been very difficult to
> understand and change in a lot of other languages. Even hacking the VM seems
> to be possible for a non-VM expert.
> But I consider myself rather as a CB. As such I tend to try and do complex
> things that I usually do in other languages and run into tricky problems.
> These problems are rather dealt with and corrected by Pharo developers but
> that as a user I would expect them to remain hidden to me or to be clearly
> advertised in the docs. As compared to a BB, a CB is not going to stay in a
> well delimited area where everything is smooth.
> True, in a way it is a very strong incentive to become a Pharo expert! But I
> am wondering if this aspect could be improved.

thing is… non OO programmers will have problems to understand a pure OO language.
people working with Java, C#, C++ and others may think they do OO, but they don’t most of the time… then, switching paradigms is hard work.
even worst, smalltalk syntax is considered “alien” to people used to algol-based languages.

but we cannot do much more than we are trying to achieve in this area: make Pharo more compatible with “the rest of the world” when it make sense, but strongly stay in our “alieness” when it has sense (syntax, pureness, etc.).

Esteban

>
> Thanks,
> Bruno
>
>
>
> --
> View this message in context: http://forum.world.st/Parser-failure-on-FFI-pragmas-declaration-in-Pharo-5-tp4961737.html
> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>

--

Guille Polito Research Engineer French National Center for Scientific Research - http://www.cnrs.fr Web: http://guillep.github.io Phone: <a href="tel:+33%206%2052%2070%2066%2013" value="+33652706613" target="_blank">+33 06 52 70 66 13

> On 17. Aug 2017, at 19:37, Denis Kudriashov <[hidden email]> wrote:


Hey!

> Yes.
>
> Also simple solution can be to override compiler of problem classes to return old compiler.
>
> I know it is better to rewrite code but it can be not simple task when there are a lot of ffi-methods.


I ran into this problem with the wonderful (as it had a SHA256 implementation) NaCl bindings.


Pharo5:
 * RBParser still has the currentScope variable and can import it
 * Syntax highlighting ends in an exception (which I disabled)

Pharo6:
 * RBParser doesn't have currentScope anymore so I patched it out
 * Syntax highlighting seems to work fine


@Esteban: Would you accept a change to the FFI-Pharo5Compat to not use the currentScope variable/reduce error checking? Or would you accept it in a FFI-Pharo6Compat package?

I think it would help to be able to load the Nacl code in Pharo6 and then fix it?


what do you think?

        holger