FFI issue in Squeak 4.2 ?

I'm struggling to load Xtreams-Xtras into the new release of Squeak. I sprinkled all the classes with allowUnderscoreSelectors (true) and allowUnderscoreAssignments (false), made sure I don't have any underscores on class side, made sure none of the classes have underscores in their names, but it still seems that unless i explicitly turn off allowUnderscoreAssignments in global preferences it won't compile the instance side methods with underscore selectors.

Then even when I get past it with the help of the global preference, I still get the following syntax error which doesn't make any sense to me:

HMAC_CTX_cleanup: ctx
        <cdecl: void 'HMAC_CTX_cleanup' (HMACCTX*Matching number of arguments expected ->) module: 'libcrypto'>
        ^self externalCallFailed

HMACCTX is a subclass of ExternalStructure. This all seems to be fine on Pharo side and I would swear I could load it into an earlier Squeak version, but I don't remember the details anymore. I even made sure I'm using the exact same versions of FFI packages as I do in Pharo (the configuration from MetacelloRepository that I normally load in Pharo doesn't seem to be quite up to date), but that didn't help either. Any suggestions ? How should one go about writing portable FFI code (I mean portable between Pharo and Squeak)? Any chance that FFI support would be in the base image one day so that one didn't have to worry about where to get it and which one ?

Thanks,

Martin

On Sun, 20 Feb 2011, [hidden email] wrote:

> I'm struggling to load Xtreams-Xtras into the new release of Squeak. I sprinkled all the classes with allowUnderscoreSelectors (true) and allowUnderscoreAssignments (false), made sure I don't have any underscores on class side, made sure none of the classes have underscores in their names, but it still seems that unless i explicitly turn off allowUnderscoreAssignments in global preferences it won't compile the instance side methods with underscore selectors.

There's no need to turn off #allowUnderscoreAsAssignment if the code is
not ambiguous (it's not). The need for the global
#allowUnderscoreSelectors is because of MC. It tries to load EVPMD >>
#'block_size' before EVPMD class >> #allowUnderscoreSelectors which won't
work.

IMHO it's better to just use the global preference, since the per class
methods are not good enough.

>
> Then even when I get past it with the help of the global preference, I still get the following syntax error which doesn't make any sense to me:
>
> HMAC_CTX_cleanup: ctx
> <cdecl: void 'HMAC_CTX_cleanup' (HMACCTX*Matching number of arguments expected ->) module: 'libcrypto'>
> ^self externalCallFailed
>
> HMACCTX is a subclass of ExternalStructure. This all seems to be fine on Pharo side and I would swear I could load it into an earlier Squeak version, but I don't remember the details anymore. I even made sure I'm using the exact same versions of FFI packages as I do in Pharo (the configuration from MetacelloRepository that I normally load in Pharo doesn't seem to be quite up to date), but that didn't help either. Any suggestions ? How should one go about writing portable FFI code (I mean portable between Pharo and Squeak)? Any chance that FFI support would be in the base image one day so that one didn't have to worry about where to get it and which one ?

FFI is fine. The cause of the problem is String >> #numArgs. $_ is not
accepted by it as a valid selector character, so it returns -1 instead of
1 for #'HMAC_CTX_cleanup:'. It will be fixed in the Trunk soon, along
with the other underscore related problems.

I doubt FFI will be preloaded in images, because it has some security
risk. However there's a simple solution: Metacello. A Metacello
configuration can ensure that FFI is loaded before the FFI specific parts
are loaded, take care about platform specific stuff, like changing the
preferences, etc.

I managed to load XTreams to Squeak Trunk (possibly works with 4.2 too)
using the following script:

"Make sure that underscores are allowed in selectors."
Character
  compile: 'tokenish
  "Answer whether the receiver is a valid token-character--letter, digit, or
  colon."

  ^self == $_ or: [self == $: or: [self isLetter or: [self isDigit]]]'
  classified: 'testing'.
String initialize.
Scanner prefAllowUnderscoreSelectors: true.
"Load FFI"
Installer squeak
  project: 'FFI';
  install: 'FFI-Pools';
  install: 'FFI-Kernel';
  install: 'FFI-Tests';
  install: 'FFI-Win32';
  install: 'FFI-MacOS';
  install: 'FFI-Unix'.
"Load XTreams"
Installer ss
  project: 'Xtreams';
  install: 'Xtreams-Support';
  install: 'Xtreams-Core-';
  install: 'Xtreams-Terminals-';
  install: 'Xtreams-Transforms-';
  install: 'Xtreams-Substreams-';
  install: 'Xtreams-Parsing-';
  " --- tests follow --- "
  install: 'Xtreams-CoreTests';
  install: 'Xtreams-TerminalsTests';
  install: 'Xtreams-TransformsTests';
  install: 'Xtreams-SubstreamsTests';
  install: 'Xtreams-ParsingTests';
  " --- following require FFI --- "
  install: 'Xtreams-Xtras-';
  install: 'Xtreams-XtrasTests'.

I also uploaded a new version of Xtreams-Xtras to the repository which
fixes XTHMACTest and XTHashTest for Squeak.

With these changes, all tests are green on Windows.


Levente


>
> Thanks,
>
> Martin
>
>
>

"Levente Uzonyi"<[hidden email]> wrote:
> There's no need to turn off #allowUnderscoreAsAssignment if the code is
> not ambiguous (it's not).

I was getting desperate, so I was trying whatever could possibly make a difference :-). I'll clean it up again.

> The need for the global
> #allowUnderscoreSelectors is because of MC. It tries to load EVPMD >>
> #'block_size' before EVPMD class >> #allowUnderscoreSelectors which won't
> work.
>
> IMHO it's better to just use the global preference, since the per class
> methods are not good enough.

> FFI is fine. The cause of the problem is String >> #numArgs. $_ is not
> accepted by it as a valid selector character, so it returns -1 instead of
> 1 for #'HMAC_CTX_cleanup:'. It will be fixed in the Trunk soon, along
> with the other underscore related problems.

Excellent!

> I doubt FFI will be preloaded in images, because it has some security risk.

Security risks that outweigh benefits of having the ability to call the outside world without much hassle? I'm curious, but it sounds like the community already had this argument and decided it's not worth it. No need to stir up old arguments, I'll see if I can find something in the archives.

> However there's a simple solution: Metacello. A Metacello
> configuration can ensure that FFI is loaded before the FFI specific parts
> are loaded, take care about platform specific stuff, like changing the
> preferences, etc.

Hm, so if I create a metacello configuration and assuming there are some pre/post load hooks, can I turn the global flag on just for the duration of the load ? I prefer not to mess with global state of people's images if it can be avoided (I mean beyond adding a pile of new code to it :-).

> I managed to load XTreams to Squeak Trunk (possibly works with 4.2 too)
> using the following script:
> ...
> I also uploaded a new version of Xtreams-Xtras to the repository which
> fixes XTHMACTest and XTHashTest for Squeak.
>
> With these changes, all tests are green on Windows.

Excellent! So that means the new Bcrypt based hashing worked for you too. Thank you very much! I really appreciate your help.

Martin

In Sq 4.2 Help/Extending the system Workspace, there is a script to load FFI and friends:

"FFI: http://source.squeak.org/FFI.html"
(Installer repository: 'http://source.squeak.org/FFI')
        install: 'FFI-Pools';
        install: 'FFI-Kernel';
        install: 'FFI-Tests';
        install: 'FFI-Win32';
        install: 'FFI-MacOS';
        install: 'FFI-Unix'.

Ken G. Brown



At 9:26 PM -0500 2/20/11, [hidden email] apparently wrote:

On Sun, 20 Feb 2011, [hidden email] wrote:

> "Levente Uzonyi"<[hidden email]> wrote:
>> There's no need to turn off #allowUnderscoreAsAssignment if the code is
>> not ambiguous (it's not).
>
> I was getting desperate, so I was trying whatever could possibly make a difference :-). I'll clean it up again.

Oh, so you added the selector to every class. In that case there may be
ambiguity, but the chance is very small.

IIRC the last discussion about it was years ago, so maybe it's worth
reviewing the idea.

>
>> However there's a simple solution: Metacello. A Metacello
>> configuration can ensure that FFI is loaded before the FFI specific parts
>> are loaded, take care about platform specific stuff, like changing the
>> preferences, etc.
>
> Hm, so if I create a metacello configuration and assuming there are some pre/post load hooks, can I turn the global flag on just for the duration of the load ? I prefer not to mess with global state of people's images if it can be avoided (I mean beyond adding a pile of new code to it :-).

Yes. I hacked together a basic Metacello configuration for Xtreams, which
automatically loads FFI if the Xtras package is requested (and FFI is
not loaded) and fixes the underscore problems for Squeak. I uploaded it to
the MetacelloRepository (http://squeaksource.com/MetacelloRepository ).
You can load Xtreams into Squeak 4.2 with it using the following script:

Installer ss
  project: 'MetacelloRepository';
  install: 'ConfigurationOfXtreams'.
(Smalltalk at: #ConfigurationOfXtreams) perform: #loadBleedingEdge

The Pharo version of this script should use Gofer instead of Installer to
load the configuration itself, but the last line should remain the same.

There are no versions yet, just a baseline, so one can only load the
#bleedingEdge versions of the packages. This is equivalent with the
current loader scripts.

I also defined some groups which only load some, but not all packages
(no tests, no Xtras, etc). See the commit message or the specification
itself for details.

Yes. Before XP users start wondering why this doesn't work for them,
bcrypt is only supported on Vista and later versions (according to msdn).


Levente

>
> Martin
>

Hi, can you clarify whether this loads a fixed-configuration or a
variable configuration?

I will document this in SqueakMap, but I think it's important to know
whether this "configuration" is a true configuration or if it's a
load-script that fetches the latest-and-greatest?  #loadBleedingEdge
suggests the latter, and, if so, would you also specify a load-script
that will ensure longevity in Squeak 4.2?

I think we should be explicit about load-scripts kept in Metacello,
like we are now doing in SM, and say, "you can load the 'latest
bleeding-edge' Xtreams code with the following script" rather than
saying, "into Squeak 4.2" because there is no guarantee that will
remain the case.

Thanks and Regards,
  Chris

On 20.02.2011, at 18:26, [hidden email] wrote:

> "Levente Uzonyi"<[hidden email]> wrote:
>
>> I doubt FFI will be preloaded in images, because it has some security risk.
>
> Security risks that outweigh benefits of having the ability to call the outside world without much hassle? I'm curious, but it sounds like the community already had this argument and decided it's not worth it. No need to stir up old arguments, I'll see if I can find something in the archives.

It's mostly to ensure no part of the base system depends on FFI. If it was included by default, people might get the idea to "enhance" Squeak by relying on it. We don't want that.

The proper Squeak way to expose more of the operating system would be a primitive - what is it that XStreams needs that we don't have yet?

- Bert -

On Mon, 21 Feb 2011, Chris Muller wrote:
"baseline" and "version" are Metacello terms. A "baseline" specifies a
set of packages and their dependencies. A "version" defines which version
of those packages should be loaded. So this loads the latest version of
all packages. So this script is the "head" on SqueakMap.

>
> I will document this in SqueakMap, but I think it's important to know
> whether this "configuration" is a true configuration or if it's a
> load-script that fetches the latest-and-greatest?  #loadBleedingEdge
> suggests the latter, and, if so, would you also specify a load-script
> that will ensure longevity in Squeak 4.2?

The latest version of the packages can be considered as stable, all tests
pass. I can create a "version" with these and mark it as stable for
Squeak 4.2. When this is done, a different script can be added for 4.2 on
SqueakMap.


Levente

On Tue, 22 Feb 2011, Levente Uzonyi wrote:
I just did this, so the following script should work from Squeak 4.2:

Installer ss
  project: 'MetacelloRepository';
  install: 'ConfigurationOfXtreams'.
(Smalltalk at: #ConfigurationOfXtreams) perform: #load


Levente

"Bert Freudenberg"<[hidden email]> wrote:
> It's mostly to ensure no part of the base system depends on FFI. If it was included by default, people might get the idea to "enhance" Squeak by relying on it. We don't want that.
>
> The proper Squeak way to expose more of the operating system would be a primitive - what is it that XStreams needs that we don't have yet?

That is interesting. Our inclination on the VW side is to lift things out of the VM into the VI as much as is feasible. Granted, the VM situation is different there. Presumably in Squeak/Pharo it is easier for people to write primitives, which could make a difference. But I wonder how you overcome deployment obstacles, e.g. to deploy a new plugin you need to compile it on all the relevant platforms, get it included in the relevant distribution channels, etc. Does that really scale ?

In my specific case with Xtreams I want to take advantage of OS level cryptographic primitives. These days they are pretty much always present on any reasonably recent platform. They are faster (usually orders of magnitude faster), someone else is responsible for keeping the algorithm portfolio up to date, etc. The other half of my motivation is making sure that an external implementation can be plugged in and demonstrate how to do it. Often you simply have to use specific implementations, either because it has to have particular certification, or be on particular "approved" list or it is backed by hardware support that has to be leveraged (accelerators, hardware security tokens, etc). So in this case I'm trying to use the native BCrypt APIs on Windows and OpenSSL's libcrypto everywhere else (for now). With decent FFI, all I really need to maintain is my set of packages on squeaksource. If done right, it should just work right after loading. I think that's about as easy as it can get. I'm not sure I could take on maintenance of a plugin as easily.

Cheers,

Martin

"Levente Uzonyi"<[hidden email]> wrote:
> I just did this, so the following script should work from Squeak 4.2:
>
> Installer ss
>   project: 'MetacelloRepository';
>   install: 'ConfigurationOfXtreams'.
> (Smalltalk at: #ConfigurationOfXtreams) perform: #load

Thank you! This is very helpful. I didn't have much time to play with this yet (holidays interfered), but it's looking good. Loading into Squeak works fine as you said. I'll have to wrap my head around metacello some more to figure out what needs to be done on Pharo side, but this is certainly much better than starting from scratch.

Thanks,

Martin

On Tue, 22 Feb 2011, [hidden email] wrote:

> "Levente Uzonyi"<[hidden email]> wrote:
>> I just did this, so the following script should work from Squeak 4.2:
>>
>> Installer ss
>>   project: 'MetacelloRepository';
>>   install: 'ConfigurationOfXtreams'.
>> (Smalltalk at: #ConfigurationOfXtreams) perform: #load
>
> Thank you! This is very helpful. I didn't have much time to play with this yet (holidays interfered), but it's looking good. Loading into Squeak works fine as you said. I'll have to wrap my head around metacello some more to figure out what needs to be done on Pharo side, but this is certainly much better than starting from scratch.

In case of Pharo 1.2 the following should work:

Gofer new
  squeaksource: 'MetacelloRepository';
  package: 'ConfigurationOfXtreams';
  load.
(Smalltalk at: #ConfigurationOfXtreams) load


Levente

>
> Thanks,
>
> Martin
>