HTTPSocket issue

Hello,

I'm trying to make a call using HTTPSocket to a server. In Pharo (1.2)
my response time is sub second but in GemStone (2.4.4.4, with
Squeak-DaleHenrichs.252), I get a response after about 1 minute. The
contents of the response also differs. Here's the example I used:

HTTPSocket
        httpPostDocument: 'http://api-verify.recaptcha.net/verify'
        args: (Dictionary new
                at: 'privatekey' put: #('6LdUKcESAAAAAO8hlc8PyQCRgkS5gXLXTSK4B7b2');
                at: 'remoteip' put: #('');
                at: 'response' put: #('whatever');
                at: 'challenge' put:
#('03AHJ_Vus2ELMgMly7AajxZfgnKUkdOTJWHT2E-OA6pKs16V82abS8oXANUxYn8ziFLs7gRg_Iy3HMVhH8gMzeI8aO3Y7IZJuQDYh7zV8EeIgIjvIclPUCAOtIVblw1mKnf5Mp93tQSABUdeyzwUpgAtgURAiFpFZOAA');
                yourself)
        accept: 'application/octet-stream'
        request: ''

In Pharo, my result is:

'false
incorrect-captcha-sol'

and in GemStone:

'73
false
''Input error: challenge: Error parsing captcha challenge
value\nprivatekey: Format of site key was invalid\n''
0

'

Can you please help?

What I can see is that in HTTPSocket | getRestOfBuffer: we are in
waitForReadDataUpToMs: 5000. But at this point, the buf (fer) already
contains the result above. So, it does not seem to pick up that we've
already read the response.

I would like to mention that I'm also trying to port SoapOpera and got
the SoapClient going in GemStone. I get a quick (and correct) response
in Pharo, but on the GS side it hangs up in the same way. Here's the
Soap test:

((SoapCallEntry tcpHost: 'swikis.ddo.jp' port: 8823) newCall)
                methodName: 'reverseString';
                addParameterNamed: #aString;
                invokeAndReturnWithValues: #(#abc)

Thanks a lot
Otto

Hi Otto,

I don't know if this affects your problem but what is sent out the wire
when using HTTPSocket class side methods in Gemstone is different than
in Pharo.  At least that is the case with the
httpGetDocument:args:accept:request: method.  The Gemstone
implementation adds the port number after the host name. e.g.

If you were using httpGetDocument: your call to
'http://api-verify.recaptcha.net/verify' is unchanged in Pharo but
becomes 'http://api-verify.recaptcha.net:80/verify' in Gemstone.  I
don't know if the httpPostDocument: methods do the same or not but if so
that might be causing a problem on the recaptcha.net side that creates
your delays.

I override the Gemstone version of httpGetDocument:args:accept:request:
with this method:

httpGetDocument: url args: args accept: mimeType request: requestString
        "Return the exact contents of a web object. Asks for the given MIME
type. If mimeType is nil, use 'text/html'. An extra requestString may be
submitted and must end with crlf.  The parsed header is saved. Use a
proxy server if one has been registered.  tk 7/23/97 17:12"
        "Note: To fetch raw data, you can use the MIME type
'application/octet-stream'."

        | serverName serverAddr port sock header length bare page list firstData
aStream index connectToHost connectToPort type newUrl portSuffix |
        bare _ (url asLowercase beginsWith: 'http://')
                ifTrue: [url copyFrom: 8 to: url size]
                ifFalse: [url].
        bare _ bare copyUpTo: $#.  "remove fragment, if specified"
        serverName _ bare copyUpTo: $/.
        page _ bare copyFrom: serverName size + 1 to: bare size.
        (serverName includes: $:)
                ifTrue: [ index _ serverName indexOf: $:.
                        port _ (serverName copyFrom: index+1 to: serverName size) asNumber.
                        serverName _ serverName copyFrom: 1 to: index-1.
                        portSuffix:=':', port printString.]
                ifFalse: [ port _ self defaultPort.
                        portSuffix:=''. ].
        page size = 0 ifTrue: [page _ '/'].
        "add arguments"
        args ifNotNil: [page _ page, (self argString: args) ].
        "check for proxy"
        (self shouldUseProxy: serverName)
                ifFalse: [
                        connectToHost := serverName.
                        connectToPort := port]
                ifTrue: [
                        connectToHost := self httpProxyServer.
                        connectToHost isNil ifTrue: [connectToHost := serverName.].
                        connectToPort := self httpProxyPort.
                        page := 'http://', serverName, portSuffix, page. "put back
together"].

        serverAddr _ GsSocket getHostAddressByName: connectToHost.
        serverAddr ifNil: [
                ^ 'Could not resolve the server named: ', connectToHost].

3 timesRepeat: [
        [ sock _ HTTPSocket connectToServerOnHost: serverAddr port:
connectToPort ] on: Error do: [:ex | ^ 'Server ',connectToHost,' is not
responding: ', ex description ].
        sock sendCommand: 'GET ', page, ' HTTP/1.0', CrLf,
                (mimeType ifNotNil: ['ACCEPT: ', mimeType, CrLf] ifNil: ['']),
                'ACCEPT: text/html', CrLf, "Always accept plain text"
                HTTPBlabEmail, "may be empty"
                requestString, "extra user request. Authorization"
                self userAgentString, CrLf,
                'Host: ', serverName,  portSuffix, CrLf. "blank line
automatically added"

        list _ sock getResponseUpTo: CrLf, CrLf ignoring: (String with: CR).
"list = header, CrLf, CrLf,
beginningOfData"
        header _ list at: 1.
        firstData _ list at: 3.
        header isEmpty
                ifTrue: [aStream _ 'server aborted early']
                ifFalse: [
                        "dig out some headers"
                        sock header: header.
                        length _ sock getHeader: 'content-length'.
                        length ifNotNil: [ length _ length asNumber ].
                        type _ sock getHeader: 'content-type'.
                        sock responseCode first = $3 ifTrue: [
                                newUrl _ sock getHeader: 'location'.
                                newUrl ifNotNil: [
                                        sock close.
                                        newUrl _ self expandUrl: newUrl ip: serverAddr port: connectToPort.
                                        ^self httpGetDocument: newUrl args: args  accept: mimeType request:
requestString] ].
                        aStream _ sock getRestOfBuffer: firstData totalLength: length.
                        "a 400-series error"
                        sock responseCode first = $4 ifTrue: [^ header, aStream contents].
                        ].
        sock close. "Always OK to destroy!"
        aStream class ~~ String ifTrue: [
  ^ self mimeDocumentClass contentType: type content: aStream contents
url: url].
        aStream = 'server aborted early' ifTrue: [ ^aStream ].
        ].

(Array with: 'HTTPSocket class>>httpGetDocument:args:accept:request:'
with: aStream with: url) perform: #inspect.

        ^'some other bad thing happened!'


Hope this helps.

Paul.








On 02/21/2011 11:03 AM, Otto Behrens wrote:

Thanks, Paul.

This did not make a difference though. Can you do me a huge favour by
trying the example in your environment? If it works there, I know that
it has something to do with my environment.

Cheers
Otto

Thanks! Just got it going. Saw that the argString: method on GS was
inlined and behaved differently. Decided to take the Pharo code and
get that to work in GS and with little modification, it did.

Here's the code (on HTTPSocket):

initHTTPSocket: httpUrl timeoutSecs: timeout ifError: aBlock
        ^self initHTTPSocket: httpUrl wait: timeout ifError: aBlock

httpPostDocument: url  args: argsDict accept: mimeType request: requestString
        "like httpGET, except it does a POST instead of a GET.  POST allows
data to be uploaded"

        | s header length page list firstData aStream type newUrl httpUrl argString |
        httpUrl := Url absoluteFromText: url.
        page := httpUrl fullPath.
        "add arguments"
        argString := argsDict
                ifNotNil: [
                        argString := self argString: argsDict.
                        argString first = $? ifTrue: [ argString := argString copyFrom: 2
to: argString size]]
                ifNil: [''].

        s := HTTPSocket new.
        s := self initHTTPSocket: httpUrl timeoutSecs: 30 ifError:
[:errorString | ^errorString].
        s sendCommand: 'POST ', page, ' HTTP/1.0', String crlf,
                (mimeType ifNotNil: ['ACCEPT: ', mimeType, String crlf] ifNil: ['']),
                'ACCEPT: text/html', String crlf, "Always accept plain text"
                HTTPProxyCredentials,
                HTTPBlabEmail, "may be empty"
                requestString, "extra user request. Authorization"
                self userAgentString, String crlf,
                'Content-type: application/x-www-form-urlencoded', String crlf,
                'Content-length: ', argString size printString, String crlf,
                'Host: ', httpUrl authority, String crlf.  "blank line automatically added"

        "umur - IE sends argString without a $? and swiki expects so"
        s sendCommand: argString.

        "get the header of the reply"
        list := s getResponseUpTo: String crlf, String crlf ignoring: String
cr. "list = header, CrLf, CrLf, beginningOfData"
        header := list at: 1.
        "Transcript show: page; cr; show: argsStream contents; cr; show: header; cr."
        firstData := list at: 3.

        "dig out some headers"
        s header: header.
        length := s getHeader: 'content-length'.
        length ifNotNil: [ length := length asNumber ].
        type := s getHeader: 'content-type'.
        s responseCode first = $3 ifTrue: [
                newUrl := s getHeader: 'location'.
                newUrl ifNotNil: [
                        "umur 6/25/2003 12:58 - If newUrl is relative then we need to make
it absolute."
                        newUrl := (httpUrl newFromRelativeText: newUrl) asString.
                        self flag: #refactor. "get, post, postmultipart are almost doing
the same stuff"
                        s destroy.
                        "^self httpPostDocument: newUrl  args: argsDict  accept: mimeType"
                        ^self httpGetDocument: newUrl accept: mimeType ] ].

        aStream := s getRestOfBuffer: firstData totalLength: length.
        s responseCode = '401' ifTrue: [^ header, aStream contents].
        s destroy. "Always OK to destroy!"

        ^ MIMEDocument contentType: type  content: aStream contents url: url


On Mon, Feb 21, 2011 at 7:54 PM, Paul DeBruicker <[hidden email]> wrote:

Hi,
There is a problem with #httpPostDocument: in gemstone. (Yes, I made the port, and I made a mistake with the API, I'm sorry for that).

In pharo, this api:

TTPSocket
        httpPostDocument: 'http://api-verify.recaptcha.net/verify'
        args: (Dictionary new
                at: 'privatekey' put: #('6LdUKcESAAAAAO8hlc8PyQCRgkS5gXLXTSK4B7b2');
                at: 'remoteip' put: #('');
                at: 'response' put: #('whatever');
                ...

works completely fine. But in Gemstone, the arguments dictionary receives a plain String, not an Array. So,

TTPSocket
        httpPostDocument: 'http://api-verify.recaptcha.net/verify'
        args: (Dictionary new
                at: 'privatekey' put: '6LdUKcESAAAAAO8hlc8PyQCRgkS5gXLXTSK4B7b2';
                at: 'remoteip' put: '';
                at: 'response' put: 'whatever';
                ...

should work better :P
of course, the real fix is to change the Gemstone implementation to match Pharo one, but I didn't find the time yet to do it :(

Cheers,
Esteban

El 21/02/2011, a las 3:46p.m., Otto Behrens escribió: