The Inbox: WebClient-HTTP-cmm.5.mcz

> Chris Muller uploaded a new version of WebClient-HTTP to project The Inbox:
> http://source.squeak.org/inbox/WebClient-HTTP-cmm.5.mcz
>
> ==================== Summary ====================
>
> Name: WebClient-HTTP-cmm.5
> Author: cmm
> Time: 11 September 2016, 4:59:26.542985 pm
> UUID: e203b772-bc1c-4516-8bf4-4ea1bc1edb7d
> Ancestors: WebClient-HTTP-cmm.4
>
> Don't force Accept header of 'text/html', let clients specify that if its appropriate.  This fixes accessing servers that demand it not be present.
>
> =============== Diff against WebClient-HTTP-cmm.4 ===============
>
> Item was changed:
>   ----- Method: HTTPSocket class>>httpGetDocument:args:accept:request: (in category '*webclient-http') -----
>   httpGetDocument: url args: args accept: mimeType request: requestString
>         "Return the exact contents of a web object. Asks for the given MIME type. If mimeType is nil, use 'text/html'. An extra requestString may be submitted and must end with crlf.  The parsed header is saved. Use a proxy server if one has been registered.  tk 7/23/97 17:12"
>
>         "Note: To fetch raw data, you can use the MIME type 'application/octet-stream'."
>
>         | client xhdrs resp urlString progress |
>         "Normalize the url"
>         urlString := (Url absoluteFromText: url) asString.
>
>         args ifNotNil: [
>                 urlString := urlString, (self argString: args)
>         ].
>
>         "Some raw extra headers which historically have been added"
>         xhdrs := HTTPProxyCredentials,
>                 HTTPBlabEmail,  "may be empty"
>                 requestString.  "extra user request. Authorization"
>
>         client := WebClient new.
>         ^[resp := client httpGet: urlString do:[:req|
>                 "Add ACCEPT header"
>                 mimeType ifNotNil:[req headerAt: 'Accept' put: mimeType].
>
> -               "Always accept plain text"
> -               req addHeader: 'Accept' value: 'text/html'.
> -
>                 "Add the additional headers"
>                 (WebUtils readHeadersFrom: xhdrs readStream)
>                         do:[:assoc| req addHeader: assoc key value: assoc value]].
>
>         progress := [:total :amount|
>                 (HTTPProgress new) total: total; amount: amount; signal: 'Downloading...'
>         ].
>
>         "Simulate old HTTPSocket return behavior"
>         (resp code between: 200 and: 299)
>                 ifTrue:[MIMEDocument contentType: resp contentType
>                                 content: (resp contentWithProgress: progress) url: url]
>                 ifFalse:[resp asString, resp content].
>         ] ensure:[client destroy].
>   !
>
>

>
> On Sun, Sep 11, 2016 at 4:59 PM,  <[hidden email]> wrote:
>> Chris Muller uploaded a new version of WebClient-HTTP to project The Inbox:
>> http://source.squeak.org/inbox/WebClient-HTTP-cmm.5.mcz
>>
>> ==================== Summary ====================
>>
>> Name: WebClient-HTTP-cmm.5
>> Author: cmm
>> Time: 11 September 2016, 4:59:26.542985 pm
>> UUID: e203b772-bc1c-4516-8bf4-4ea1bc1edb7d
>> Ancestors: WebClient-HTTP-cmm.4
>>
>> Don't force Accept header of 'text/html', let clients specify that if its appropriate.  This fixes accessing servers that demand it not be present.
>>
>> =============== Diff against WebClient-HTTP-cmm.4 ===============
>>
>> Item was changed:
>>  ----- Method: HTTPSocket class>>httpGetDocument:args:accept:request: (in category '*webclient-http') -----
>>  httpGetDocument: url args: args accept: mimeType request: requestString
>>        "Return the exact contents of a web object. Asks for the given MIME type. If mimeType is nil, use 'text/html'. An extra requestString may be submitted and must end with crlf.  The parsed header is saved. Use a proxy server if one has been registered.  tk 7/23/97 17:12"
>>
>>        "Note: To fetch raw data, you can use the MIME type 'application/octet-stream'."
>>
>>        | client xhdrs resp urlString progress |
>>        "Normalize the url"
>>        urlString := (Url absoluteFromText: url) asString.
>>
>>        args ifNotNil: [
>>                urlString := urlString, (self argString: args)
>>        ].
>>
>>        "Some raw extra headers which historically have been added"
>>        xhdrs := HTTPProxyCredentials,
>>                HTTPBlabEmail,  "may be empty"
>>                requestString.  "extra user request. Authorization"
>>
>>        client := WebClient new.
>>        ^[resp := client httpGet: urlString do:[:req|
>>                "Add ACCEPT header"
>>                mimeType ifNotNil:[req headerAt: 'Accept' put: mimeType].
>>
>> -               "Always accept plain text"
>> -               req addHeader: 'Accept' value: 'text/html'.
>> -
>>                "Add the additional headers"
>>                (WebUtils readHeadersFrom: xhdrs readStream)
>>                        do:[:assoc| req addHeader: assoc key value: assoc value]].
>>
>>        progress := [:total :amount|
>>                (HTTPProgress new) total: total; amount: amount; signal: 'Downloading...'
>>        ].
>>
>>        "Simulate old HTTPSocket return behavior"
>>        (resp code between: 200 and: 299)
>>                ifTrue:[MIMEDocument contentType: resp contentType
>>                                content: (resp contentWithProgress: progress) url: url]
>>                ifFalse:[resp asString, resp content].
>>        ] ensure:[client destroy].
>>  !
>>
>>
>

>
> On 12.09.2016, at 00:02, Chris Muller <[hidden email]> wrote:
>
>> With this fix, I can do:
>>
>>   'https://www.quandl.com/api/v3/databases.csv' asUrl retrieveContents
>>
>> without, I can't.
>
>
> Well, shouldn't it be something like this then?:
>
> Accept: text/html; q=1.0, */*; q=0.1
>
> Best regards
>         -Tobias
>
>>
>> On Sun, Sep 11, 2016 at 4:59 PM,  <[hidden email]> wrote:
>>> Chris Muller uploaded a new version of WebClient-HTTP to project The Inbox:
>>> http://source.squeak.org/inbox/WebClient-HTTP-cmm.5.mcz
>>>
>>> ==================== Summary ====================
>>>
>>> Name: WebClient-HTTP-cmm.5
>>> Author: cmm
>>> Time: 11 September 2016, 4:59:26.542985 pm
>>> UUID: e203b772-bc1c-4516-8bf4-4ea1bc1edb7d
>>> Ancestors: WebClient-HTTP-cmm.4
>>>
>>> Don't force Accept header of 'text/html', let clients specify that if its appropriate.  This fixes accessing servers that demand it not be present.
>>>
>>> =============== Diff against WebClient-HTTP-cmm.4 ===============
>>>
>>> Item was changed:
>>>  ----- Method: HTTPSocket class>>httpGetDocument:args:accept:request: (in category '*webclient-http') -----
>>>  httpGetDocument: url args: args accept: mimeType request: requestString
>>>        "Return the exact contents of a web object. Asks for the given MIME type. If mimeType is nil, use 'text/html'. An extra requestString may be submitted and must end with crlf.  The parsed header is saved. Use a proxy server if one has been registered.  tk 7/23/97 17:12"
>>>
>>>        "Note: To fetch raw data, you can use the MIME type 'application/octet-stream'."
>>>
>>>        | client xhdrs resp urlString progress |
>>>        "Normalize the url"
>>>        urlString := (Url absoluteFromText: url) asString.
>>>
>>>        args ifNotNil: [
>>>                urlString := urlString, (self argString: args)
>>>        ].
>>>
>>>        "Some raw extra headers which historically have been added"
>>>        xhdrs := HTTPProxyCredentials,
>>>                HTTPBlabEmail,  "may be empty"
>>>                requestString.  "extra user request. Authorization"
>>>
>>>        client := WebClient new.
>>>        ^[resp := client httpGet: urlString do:[:req|
>>>                "Add ACCEPT header"
>>>                mimeType ifNotNil:[req headerAt: 'Accept' put: mimeType].
>>>
>>> -               "Always accept plain text"
>>> -               req addHeader: 'Accept' value: 'text/html'.
>>> -
>>>                "Add the additional headers"
>>>                (WebUtils readHeadersFrom: xhdrs readStream)
>>>                        do:[:assoc| req addHeader: assoc key value: assoc value]].
>>>
>>>        progress := [:total :amount|
>>>                (HTTPProgress new) total: total; amount: amount; signal: 'Downloading...'
>>>        ].
>>>
>>>        "Simulate old HTTPSocket return behavior"
>>>        (resp code between: 200 and: 299)
>>>                ifTrue:[MIMEDocument contentType: resp contentType
>>>                                content: (resp contentWithProgress: progress) url: url]
>>>                ifFalse:[resp asString, resp content].
>>>        ] ensure:[client destroy].
>>>  !
>>>
>>>
>>
>
>

>
> On Sun, Sep 11, 2016 at 5:05 PM, Tobias Pape <[hidden email]> wrote:
>>
>> On 12.09.2016, at 00:02, Chris Muller <[hidden email]> wrote:
>>
>>> With this fix, I can do:
>>>
>>>  'https://www.quandl.com/api/v3/databases.csv' asUrl retrieveContents
>>>
>>> without, I can't.
>>
>>
>> Well, shouldn't it be something like this then?:
>>
>> Accept: text/html; q=1.0, */*; q=0.1
>>
>> Best regards
>>        -Tobias
>>
>>>
>>> On Sun, Sep 11, 2016 at 4:59 PM,  <[hidden email]> wrote:
>>>> Chris Muller uploaded a new version of WebClient-HTTP to project The Inbox:
>>>> http://source.squeak.org/inbox/WebClient-HTTP-cmm.5.mcz
>>>>
>>>> ==================== Summary ====================
>>>>
>>>> Name: WebClient-HTTP-cmm.5
>>>> Author: cmm
>>>> Time: 11 September 2016, 4:59:26.542985 pm
>>>> UUID: e203b772-bc1c-4516-8bf4-4ea1bc1edb7d
>>>> Ancestors: WebClient-HTTP-cmm.4
>>>>
>>>> Don't force Accept header of 'text/html', let clients specify that if its appropriate.  This fixes accessing servers that demand it not be present.
>>>>
>>>> =============== Diff against WebClient-HTTP-cmm.4 ===============
>>>>
>>>> Item was changed:
>>>> ----- Method: HTTPSocket class>>httpGetDocument:args:accept:request: (in category '*webclient-http') -----
>>>> httpGetDocument: url args: args accept: mimeType request: requestString
>>>>       "Return the exact contents of a web object. Asks for the given MIME type. If mimeType is nil, use 'text/html'. An extra requestString may be submitted and must end with crlf.  The parsed header is saved. Use a proxy server if one has been registered.  tk 7/23/97 17:12"
>>>>
>>>>       "Note: To fetch raw data, you can use the MIME type 'application/octet-stream'."
>>>>
>>>>       | client xhdrs resp urlString progress |
>>>>       "Normalize the url"
>>>>       urlString := (Url absoluteFromText: url) asString.
>>>>
>>>>       args ifNotNil: [
>>>>               urlString := urlString, (self argString: args)
>>>>       ].
>>>>
>>>>       "Some raw extra headers which historically have been added"
>>>>       xhdrs := HTTPProxyCredentials,
>>>>               HTTPBlabEmail,  "may be empty"
>>>>               requestString.  "extra user request. Authorization"
>>>>
>>>>       client := WebClient new.
>>>>       ^[resp := client httpGet: urlString do:[:req|
>>>>               "Add ACCEPT header"
>>>>               mimeType ifNotNil:[req headerAt: 'Accept' put: mimeType].
>>>>
>>>> -               "Always accept plain text"
>>>> -               req addHeader: 'Accept' value: 'text/html'.
>>>> -
>>>>               "Add the additional headers"
>>>>               (WebUtils readHeadersFrom: xhdrs readStream)
>>>>                       do:[:assoc| req addHeader: assoc key value: assoc value]].
>>>>
>>>>       progress := [:total :amount|
>>>>               (HTTPProgress new) total: total; amount: amount; signal: 'Downloading...'
>>>>       ].
>>>>
>>>>       "Simulate old HTTPSocket return behavior"
>>>>       (resp code between: 200 and: 299)
>>>>               ifTrue:[MIMEDocument contentType: resp contentType
>>>>                               content: (resp contentWithProgress: progress) url: url]
>>>>               ifFalse:[resp asString, resp content].
>>>>       ] ensure:[client destroy].
>>>> !
>>>>
>>>>
>>>
>>
>>
>