CurlPlugin (call for testing)

Hi all.

        I?ve updated the CurlPlugin swiki page:  
http://minnow.cc.gatech.edu/squeak/5865
Current state of the plugin is completely different to that of the  
original demo, that is: it supports lot of options, it is non-blocking  
(with the exception of 'file://' urls transfers) and it is defensively  
written (this one is subjective :).

        It took much more time than I expected (was far more busy all this time  
than usually). I?ve noticed that native SSL http support emerged during  
this month (what is excellent), so the main need for  the plugin seems to  
be gone. It still may be useful (it is probably is/can be faster, smarter  
in corners and more powerful). With the respect to the community and Avi,  
I don?t want to abuse his generosity, so we can skip that part about $$.  
This doesn?t mean that I?m refusing from a beer on occasion, however :)

        I?m going to leave for a vacation soon for two weeks, during this time I  
will not have an access to Internet other than via astral. Tomorrow I will  
try to put the Linux version (I need to compile it against the same lib  
versions as a windows pack) and will wait for opinions.
Areas where I need a feedback:
1) What people do with cookies
2) What people might want to do with http-headers
3) What the heck is multiform post and
4) What people think about this in general
Based on the opinions or their absence I?ll decide to continue with it as  
with a pet project (it is not polished yet) or to do something else.

Regards,
Danil

2006/10/11, danil osipchuk <[hidden email]>:
I'm thinking about using it for SqueakSource because HTTP POST with
proxies is borken in HttpSocket right now.

In general:
Cool stuff!

> Based on the opinions or their absence I?ll decide to continue with it as
> with a pet project (it is not polished yet) or to do something else.
>
> Regards,
> Danil
>
>

Danil,

Great job, thanks for working on this!  The fact that we did a native SSL in
no way decreases the value of LibCurl!  

There are a lot of reasons why someone would use LibCurl instead of our
native SSL including the fact that our code is not tested and LibCurl uses
OpenSSL which is going through FIPS Certifcation.

I hope that you will consider this project as extremely important to the
community.  In general the more options that we make available in Squeak the
easier it will be for people to get their project completed.

Thank you again for your work!  Nice job!

Ron Teitelbaum
Squeak Cryptography Team Leader

On Oct 11, 2006, at 10:07 AM, danil osipchuk wrote:
>
> It took much more time than I expected (was far more busy all this  
> time than usually). I?ve noticed that native SSL http support  
> emerged during this month (what is excellent), so the main need  
> for  the plugin seems to be gone. It still may be useful (it is  
> probably is/can be faster, smarter in corners and more powerful).  
> With the respect to the community and Avi, I don?t want to abuse  
> his generosity, so we can skip that part about $$. This doesn?t  
> mean that I?m refusing from a beer on occasion, however :)

If we skip the $$, this whole bounty experiment is pointless - please  
send me an email offline so we can co-ordinate payment.  Though IIRC  
a Mac OS X build of the plugin was one of the required deliverables :).

> I?m going to leave for a vacation soon for two weeks, during this  
> time I will not have an access to Internet other than via astral.  
> Tomorrow I will try to put the Linux version (I need to compile it  
> against the same lib versions as a windows pack) and will wait for  
> opinions.

Excellent, they'll be forthcoming.

Cheers,
Avi

On Thu, 12 Oct 2006 08:37:30 +0300, Avi Bryant <[hidden email]>  
wrote:

>
> If we skip the $$, this whole bounty experiment is pointless - please  
> send me an email offline so we can co-ordinate payment.  Though IIRC a  
> Mac OS X build of the plugin was one of the required deliverables :).

Well, it is not yet completed anyway (lets call it alpha), so I suggest to  
postpone this aspect till we have something nearly finished.  Certainly  
MacOS build still is a requirement and I didn't forget about it :)

John, could you please take a look at this humble creation for review and  
for a MacOS build? I think it has reached the point after which the main  
infrastructure will not change dramatically.

I'm compiling recent versions of the plugin against a daily snapshot  
http://cool.haxx.se/curl-daily/, particulary this one: September 30, 2006  
(because there are bugs being fixed). To get most of it libcurl should be  
compiled with c-ares (async DNS lookups) and OpenSSL.
Typical unix-like system will probably already have a stable libcurl  
installed, so here is a deployment complication - one needs to put newer  
libs (at least the version curl 7.15.5) somewhere in LD_LIBRARY_PATH  
before the libs present when starting Squeak.

As far a I know, the main difference between gnu-C and MacOS plugin  
versions will be in memory management functions they use (for example  
NewPtr instead of malloc). I've tried to abstract this difference into the  
CurlPlugin class>>macSpecificDefines using the idea found in the RePlugin.

Cheers,
        Danil

On Wed, 11 Oct 2006 21:53:41 +0300, Philippe Marschall  
<[hidden email]> wrote:


>> Areas where I need a feedback:
>> 1)      What people do with cookies
>> 2)      What people might want to do with http-headers
>> 3)      What the heck is multiform post and
>> 4)      What people think about this in general
>
> I'm thinking about using it for SqueakSource because HTTP POST with
> proxies is borken in HttpSocket right now.
>

Great, because this is exactly the part where I would appreciate an advice.
Philippe, could you please give me an idea how can I test HTTP-POSTs (is  
not tested currently although it is present)? I guess a simple seaside app  
will be fine, yes? Should it be a plain post of the multipart-post (and  
which of them is mainstream)?

Regards

        Danil

On 11-Oct-06, at 10:37 PM, Avi Bryant wrote:

It's late here on the west coast, but the plugin code was extruded  
ok, so let me compile up an os-x plugin on thursday at some point.
>

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On 11-Oct-06, at 10:37 PM, Avi Bryant wrote:

Ok, I compiled it and ran the

Curl new getContentsUrl: 'http://www.squeak.org/'
which then gives me
'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http:/....

Even the
  Transcript open.
     (Curl new
            idleBlock: [:c | Transcript nextPutAll: (c dataSize  
printString , ' bytes read.\ ' withCRs); endEntry];
            getContentsUrl: 'http://squeak.org')
                openInWorkspaceWithTitle: 'squeak page'

works

In the morning I need to try macintel.

I'll note one problem was the usage of

CURLINFO_FTP_ENTRY_PATH

This does not appear to be in the curl.h headers that ship with 10.4
For now I prim fail primInfoFtpEntryPath.


--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On 12-Oct-06, at 1:07 AM, danil osipchuk wrote:

>
> John, could you please take a look at this humble creation for  
> review and for a MacOS build? I think it has reached the point  
> after which the main infrastructure will not change dramatically.

ftpLocalWorkDir

        ^FileDirectory on: 'c:\Danil\Deploy\Upload\tmp'

that needs to change so that it's platform agnostic.

If you are going to run newer version of curl on os-x then we'll need  
to statically link in the newer library.

>
> As far a I know, the main difference between gnu-C and MacOS plugin  
> versions will be in memory management functions they use (for  
> example NewPtr instead of malloc). I've tried to abstract this  
> difference into the CurlPlugin class>>macSpecificDefines using the  
> idea found in the RePlugin.

Nah, os-x is unix, calloc/malloc is fine, NewPtr is 1984 technology...


I'll note I think 11 tests ran, the only one that failed was to do  
with that CURLINFO_FTP_ENTRY_PATH issue, and of course the ftp tests  
since I'm too tire to alter ftpLocalWorkDir tonight.

If you've only 11 tests, then I'd suggest others could add a few more  
tests, that is the easy lifting of the project.


--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

John, thank you

>>
>> John, could you please take a look at this humble creation for review  
>> and for a MacOS build? I think it has reached the point after which the  
>> main infrastructure will not change dramatically.
>
> ftpLocalWorkDir
>
> ^FileDirectory on: 'c:\Danil\Deploy\Upload\tmp'
>

Ftp tests are very dependent on the local setup of the test evironment
(local running ftp server, etc). They never will work out of box. So
probably we can not get a lot of improvement in this area.

Is it possible to have a statically linked shared library? That would be
an ideal solution, but as far as I know it is not possible with gnu tools,
yes (don't know how about MacOS)?

>>
>> As far a I know, the main difference between gnu-C and MacOS plugin  
>> versions will be in memory management functions they use (for example  
>> NewPtr instead of malloc). I've tried to abstract this difference into  
>> the CurlPlugin class>>macSpecificDefines using the idea found in the  
>> RePlugin.
>
> Nah, os-x is unix, calloc/malloc is fine, NewPtr is 1984 technology...

Heh, glad to hear that (not having MacOS at hand I had to rely on the
RePlugin comment :) )

> I'll note I think 11 tests ran, the only one that failed was to do with  
> that CURLINFO_FTP_ENTRY_PATH issue, and of course the ftp tests since  
> I'm too tire to alter ftpLocalWorkDir tonight.
>
> If you've only 11 tests, then I'd suggest others could add a few more  
> tests, that is the easy lifting of the project.

These tests are covering only an api implementation, that is they ensure
if the feature is implemented and can be used with good/bad input. Each of
the test covers an api-protocol family.
However, they doesn't check if behaviour is correct like what ftp test
does. Setting up proper test suit if what I think is the most daunting
task now

Cheers,
        Danil

On Oct 12, 2006, at 1:01 AM, John M McIntosh wrote:
> Ok, I compiled it and ran the
>
> Curl new getContentsUrl: 'http://www.squeak.org/'
> which then gives me
> '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http:/....
>

That's nifty.  Can you get the http headers too?

Apparently it is possible, at least I managed to link statically  
CurlPlugin against it dependencies (it is 3.4M therefore):

http://minnow.cc.gatech.edu/squeak/uploads/5865/CurlPlugin.tar.gz

kubuntu@kubuntu-laptop:~/Squeak/CurlPlugin$ file CurlPlugin
CurlPlugin: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV),  
not stripped

kubuntu@kubuntu-laptop:~/Squeak/CurlPlugin$ ls -l
total 3944
-rw-r--r-- 1 kubuntu kubuntu    6319 2006-10-12 17:42 CurlFtpTests.st
-rwxr-xr-x 1 kubuntu kubuntu 3386131 2006-10-12 17:47 CurlPlugin
-rw-r--r-- 1 kubuntu kubuntu  426317 2006-10-12 17:49 CurlPluginCaCert.pem
-rw-r--r-- 1 kubuntu kubuntu  147811 2006-10-12 17:45 CurlPlugin.st
-rw-r--r-- 1 kubuntu kubuntu   40468 2006-10-12 17:42 Curl.st
-rw-r--r-- 1 kubuntu kubuntu    7378 2006-10-12 17:46 CurlTests.st

kubuntu@kubuntu-laptop:~/Squeak/CurlPlugin$ ldd CurlPlugin
         linux-gate.so.1 =>  (0xffffe000)
         libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7c8d000)
         /lib/ld-linux.so.2 (0x80000000)


It works on my box, however i had to do a trick in order to complie it[*]  
so I wonder will it work anywhere other than my laptop?

[*]
This is a fragment from CurlPlugin/Makefile:

LINK            = $(LIBTOOL) --mode=link \
                   $(CC) $(CFLAGS) $(XCFLAGS) \
                   $(LDFLAGS) $(XLDFLAGS)  -shared -avoid-version -module  
-rpath $(plgdir)\
                   -Wl,-Bstatic -Wl,-lcurl -Wl,-lcares -Wl,-lidn -Wl,-lssl \
                   -Wl,-lcrypto -Wl,-lz -o


On my systems it initially failed with the error 'ldd: could not find  
libgcc_s'
There was no such lib as libgcc_s.a untill I added symlink to the  
libgcc_eh.a

I'm curios how legitimate is that :)

On Thu, 12 Oct 2006 17:08:38 +0300, Todd Blanchard <[hidden email]>  
wrote:

Something like this?

(Curl new
        onGetHeader;
        getContentsUrl: 'http://www.squeak.org/') ->

'HTTP/1.1 200 OK
Date: Thu, 12 Oct 2006 14:17:28 GMT
Server: Comanche/6.2 (unix)
Cache-Control: no-cache
X-Wiki-Copyright: Software Composition Group, University of Berne, 2003
X-Wiki-Engine: SmallWiki 0.9.23
Content-type: text/html
Content-length: 12554

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"  
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html  
xmlns="http://www.w3.org/1999/xhtml"

mmm, sure, I'll take no credit tho, lots of work by Danil.


Curl new onGetHeader getContentsUrl: 'http://www.squeak.org/'

'HTTP/1.1 200 OK
Date: Thu, 12 Oct 2006 14:18:55 GMT
Server: Comanche/6.2 (unix)
Cache-Control: no-cache
X-Wiki-Copyright: Software Composition Group, University of Berne, 2003
X-Wiki-Engine: SmallWiki 0.9.23
Content-type: text/html
Content-length: 12554

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0....



On 12-Oct-06, at 7:08 AM, Todd Blanchard wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Ok, I've posted the plugin to the experimental folder on my ftp  
server and idisk, found via

http://www.smalltalkconsulting.com/squeak.html

look for the

CurlPlugin.1.0.0.bundle.zip

This is a universal powerpc/macintel bundle and will go into the  
Resources folder of your Squeak.app with the other plugins or a  
separate Plugins in the same directory as your Squeak.app.

Testing implies it works with both platforms

To make this plugin I had to use darwinports to install the latest  
curl library on my machine, and a macIntel machine.
Then the xcode is setup to pull the static library for libcurl.a from  
the powerpc and the macIntel machine plus
the dynamic library reference for ssl, zlib, and crypto.

Note I did have to comment out the //FORCURL #define socklen_t int
in the config.h found in the mac tree that is normally used to build  
the VM because of a conflict in the headers.


Lastly I've updated the squeak SVN repository with the xcode project  
for the CURLPlugin in the mac tree.

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

2006/10/12, danil osipchuk <[hidden email]>:
What we do is more or less the same as in
MCSMReleaseRepository>>#releaseVersion:url:
Which doesn't work because HTTPSocket doesn't respect proxy settings for POST.

As I understand it, it's not multipart but url-encoded. I don't know
wheter this is common or not, it's just what we do.

Sorry but I have to real advice how to test it (without setting up a
proxy server).

If you don't read the proxy settings for Squeak that's fine. We can do that to.

Philippe