Xtreams-Xtras: hashing and HMAC

I started playing with FFI and got the hashing streams working. They require OpenSSL's libcrypto, which should be available on any reasonably recent Unix and OSX. The VW version also has interface to native Windows bcrypt.dll (available on Vista and later) but I haven't done that part yet. Anyway the hashing streams seem to work and performance isn't bad either. Hashing 100MB of ones

[ (([ 1 ] reading limiting: 100000000) hashing: 'md5')
                -= 0; close; digest
] timeToRun

takes little over 5 seconds on not particularly fast machine. The above expression runs about 2.5 times faster on VW on the same machine, but that's probably due to the time spent in the block based terminal. Profile on VW says that only about 15% of the time is spent in the hashing layer. Hashing the 21M pharo image is pretty much instant:

((FileDirectory default / 'pharo.image') reading hashing: 'md5') -= 0; close; digest

The code is in Xtreams-Xtras and Xtreams-XtrasTests (note that you'll have to load FFI!). I tested it on Pharo 1.2 beta with both cog vm and standard vm. There are few test failures with file terminals that don't seem to like to be closed twice. Should be an easy fix.

Enjoy,

Martin

P.S.: If any resident FFI experts find the time and interest to look over my C bindings and find something wrong, please let me know. In particular I have't quite figured out how to work with struct fields that contain arrays (e.g. 'unsigned char [128]'). I figured out that I can declare the actual size of the field as a third parameter in the #fields declaration, but the accessors generated for the field are not right. I suspect they need to be some form of ExternalData but I don't quite grok how that should be created (see e.g. HMAC_CTX>>key).

Also it seems that there isn't a type definition for 'int' so I'm replacing all references to that with 'long'. Is that how it's done?

Martin,

On 17 Jan 2011, at 15:01, [hidden email] wrote:

> The code is in Xtreams-Xtras and Xtreams-XtrasTests (note that you'll have to load FFI!). I tested it on Pharo 1.2 beta with both cog vm and standard vm. There are few test failures with file terminals that don't seem to like to be closed twice. Should be an easy fix.

Could it be that you forgot to check in #hashing: for read streams ? Or did I load the code while you were updating ?

Sven

"Sven Van Caekenberghe"<[hidden email]> wrote:
> > The code is in Xtreams-Xtras and Xtreams-XtrasTests (note that you'll have to load FFI!). I tested it on Pharo 1.2 beta with both cog vm and standard vm. There are few test failures with file terminals that don't seem to like to be closed twice. Should be an easy fix.
>
> Could it be that you forgot to check in #hashing: for read streams ? Or did I load the code while you were updating ?

Ah, my bad, I misspelled the category on XTReadStream and so it was omitted. I didn't find a way to browse a "package" in Pharo 1.2 so I'm struggling a bit to keep the categories in line. Thanks for the note, a new version of Xtras is on squeaksource.

On 18 Jan 2011, at 19:57, [hidden email] wrote:

> "Sven Van Caekenberghe"<[hidden email]> wrote:
>>> The code is in Xtreams-Xtras and Xtreams-XtrasTests (note that you'll have to load FFI!). I tested it on Pharo 1.2 beta with both cog vm and standard vm. There are few test failures with file terminals that don't seem to like to be closed twice. Should be an easy fix.
>>
>> Could it be that you forgot to check in #hashing: for read streams ? Or did I load the code while you were updating ?
>
> Ah, my bad, I misspelled the category on XTReadStream and so it was omitted.

No problem.

> I didn't find a way to browse a "package" in Pharo 1.2 so I'm struggling a bit to keep the categories in line.

Hmm, click the 'Browse' button in either the Monticello Browser or the Repository Browser with the correct package selected ? But I guess you know that as well...

> Thanks for the note, a new version of Xtras is on squeaksource.

Works much better, down to 7 failures out of 785!

The problem is the one I reported earlier: XTFileReadingWritingTest expecting a read stream to be aware of data written to a file after it was opened. This does not seem to work on Mac OS X.

Sven

"Sven Van Caekenberghe"<[hidden email]> wrote:
> > I didn't find a way to browse a "package" in Pharo 1.2 so I'm struggling a bit to keep the categories in line.
>
> Hmm, click the 'Browse' button in either the Monticello Browser or the Repository Browser with the correct package selected ? But I guess you know that as well...

Well, those seem to be "read-only", i.e. any edits I try to make in those seem to disappear, so I've been ignoring those browsers. Interestingly, the one-click image (Pharo 1.1.1) seems to have 'browse package' in the SystemBrowser menus which seems to be more functional, but that doesn't seem present in the PharoCore 1.2, so I'm just getting by with the basic SystemBrowser atm.

> The problem is the one I reported earlier: XTFileReadingWritingTest expecting a read stream to be aware of data written to a file after it was opened. This does not seem to work on Mac OS X.

Yeah I have your previous report still in my inbox, but I'm kinda stuck there. As I said I don't have OSX to work with so I can't dig in to figure out what's going on. Given that it works on OSX in VW, I'm inclined to say that it should work on OSX in Squeak too. I wonder if there's some mixup in terms of file descriptors, maybe we're not talking to the same one or something such. I think Nicolas has OSX available so I'm hoping he might be able to find the time to look into it at some point.

On 18 Jan 2011, at 21:24, [hidden email] wrote:

> "Sven Van Caekenberghe"<[hidden email]> wrote:
>>> I didn't find a way to browse a "package" in Pharo 1.2 so I'm struggling a bit to keep the categories in line.
>>
>> Hmm, click the 'Browse' button in either the Monticello Browser or the Repository Browser with the correct package selected ? But I guess you know that as well...
>
> Well, those seem to be "read-only", i.e. any edits I try to make in those seem to disappear, so I've been ignoring those browsers. Interestingly, the one-click image (Pharo 1.1.1) seems to have 'browse package' in the SystemBrowser menus which seems to be more functional, but that doesn't seem present in the PharoCore 1.2, so I'm just getting by with the basic SystemBrowser atm.

Indeed, they are a read only view on what Monticello thinks the package is (the same browser is used to look at MC packages not loaded). And you are right, going between that browser and the regular one is a bit hard, if possible at all.

>> The problem is the one I reported earlier: XTFileReadingWritingTest expecting a read stream to be aware of data written to a file after it was opened. This does not seem to work on Mac OS X.
>
> Yeah I have your previous report still in my inbox, but I'm kinda stuck there. As I said I don't have OSX to work with so I can't dig in to figure out what's going on. Given that it works on OSX in VW, I'm inclined to say that it should work on OSX in Squeak too. I wonder if there's some mixup in terms of file descriptors, maybe we're not talking to the same one or something such. I think Nicolas has OSX available so I'm hoping he might be able to find the time to look into it at some point.

OK.

Thx for the responses,

Sven