[ANN] WebClient and WebServer 1.0 for Squeak

> I have a similar problem, I want to post a document to a couchDB
> instance. Without German umlauts it works fine but if I add them I get
> an error message.
>
> host :='<a href="http://localhost:5984'">http://localhost:5984'.
> r := WriteStream on: (String new: 1000).
> d := Dictionary new. d at: 'content' put: 'äs' squeakToUtf8.
> (JsonObject newFrom: d) jsonWriteOn: r.
> WebClient httpPut: host, '/notes/test5' content:  r contents  type:
> 'text/plain;charset=utf-8'.
>
> The error message of the WebResponse is
> '{"error":"bad_request","reason":"invalid UTF-8 JSON"}
> '
>
> The same thing happens if I omit #squeakToUtf8
>
> If I inspect
>     r contents
> I get
>    '{"content":"\uC3\uA4s"}'

> The test case made simpler
>
> WebClient httpPut: host, '/notes/test7' content:
> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>
> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8 JSON"}
> '
>
> whereas
>
> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
> type: 'text/plain'.
>
> gives back
> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
> '
>
> --Hannes

> On Mon, 10 May 2010, Hannes Hirzel wrote:
>
>> The test case made simpler
>>
>> WebClient httpPut: host, '/notes/test7' content:
>> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>>
>> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
>> JSON"}
>> '
>>
>> whereas
>>
>> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
>> type: 'text/plain'.
>>
>> gives back
>> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
>> '
>>
>> --Hannes
>
>

> Hi, Andreas,
>
> just one feature request: support streaming in WebClient,
> i.e. apart from being able to fetch a response content in a single blob,
> also provide an API, which would enable WebClient user to read content
> step by step,
> by using stream protocol, i.e. #next, #next: etc.
>
> I see you having a #streamFrom:to:size:progress:
> in WebMessage, but its a little bit too high level (requires an output
> stream and includes a progress block)
> and its end consumer is a #content message.
>
> It would be nice to have something like:
>
> response := WebClient httpGet: 'http://foo.bar'.
>
> content := response content. "read all at once".
>

> contentStream := response contentStream.  "read content using a stream"
> [contentStream atEnd] whileFalse: [
>  c := contentStream next.  ....
> ].
>
> There are some special uses of HTTP protocol which establishing a
> permanent socket connection
> and then sending a content in a small portions piece by piece over a time.
> Obviously, with such kind of connection, if you try to get all content
> at once (by using #content) you'll never get it to the end, because
> there's always more to read, if you wait long enough.
> But if you allow a streaming, then user could read content by portions
> and handle portions step by step and don't have to wait till all
> content arrives.
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>

> On 5/10/10, radoslav hodnicak <[hidden email]> wrote:
>>
>> Which JSON package/version are you using? I fixed a bug in the one
>> distributed with SCouchDB few weeks ago, where it didn't encode utf8
>> characters properly - the correct escaped form is \uNNNN - always padded
>> to 4 Ns. that's why you get that warning, yours is only 2-3
>>
>> rado
>
> I have been using
> http://www.squeaksource.com/JSON (over 7000 downloads)
> in combination with WebClient.
>
> Thank you Rado, I found
> http://www.squeaksource.com/SCouchDB/SCouchDB-Core-rh.8.mcz
> and will have a look at it.
> (Your comment: added handling of utf8 encoded input data - this is
> necessary for couchdb-lucene which sends results directly in utf8 and
> not \uNNNN encoded)
>

> --Hannes
>
>
>> On Mon, 10 May 2010, Hannes Hirzel wrote:
>>
>>> The test case made simpler
>>>
>>> WebClient httpPut: host, '/notes/test7' content:
>>> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>>>
>>> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
>>> JSON"}
>>> '
>>>
>>> whereas
>>>
>>> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
>>> type: 'text/plain'.
>>>
>>> gives back
>>> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>> '
>>>
>>> --Hannes
>>
>>
>
>

> just one feature request: support streaming in WebClient,
> i.e. apart from being able to fetch a response content in a single blob,
> also provide an API, which would enable WebClient user to read content
> step by step, by using stream protocol, i.e. #next, #next: etc.
>
> I see you having a #streamFrom:to:size:progress:
> in WebMessage, but its a little bit too high level (requires an output
> stream and includes a progress block)
> and its end consumer is a #content message.
>
> It would be nice to have something like:
>
> response := WebClient httpGet: 'http://foo.bar'.
>
> content := response content. "read all at once".

> On 5/11/2010 7:52 AM, Igor Stasenko wrote:
>>
>> just one feature request: support streaming in WebClient,
>> i.e. apart from being able to fetch a response content in a single blob,
>> also provide an API, which would enable WebClient user to read content
>> step by step, by using stream protocol, i.e. #next, #next: etc.
>>
>> I see you having a #streamFrom:to:size:progress:
>> in WebMessage, but its a little bit too high level (requires an output
>> stream and includes a progress block)
>> and its end consumer is a #content message.
>>
>> It would be nice to have something like:
>>
>> response := WebClient httpGet: 'http://foo.bar'.
>>
>> content := response content. "read all at once".
>
> That works out of the box.
>
>> contentStream := response contentStream.  "read content using a stream"
>> [contentStream atEnd] whileFalse: [
>>   c := contentStream next.  ....
>> ].
>
> And that works out of the box, too :-) With a small modification. The
> convenience APIs on the class side close the client and prefetch the
> response. In other words, in order to stream you need to write, e.g,
>
>  client := WebClient new.
>  [resp := client httpGet:'http://www.squeak.org'.
>  length := resp contentLength.
>  stream := resp contentStream.
>  [length > 0] whileTrue:[
>    stream next: (length min: 100).
>  ]] ensure:[stream close].
>

> On 11 May 2010 19:21, Andreas Raab<[hidden email]>  wrote:
>> And that works out of the box, too :-) With a small modification. The
>> convenience APIs on the class side close the client and prefetch the
>> response. In other words, in order to stream you need to write, e.g,
>>
>>   client := WebClient new.
>>   [resp := client httpGet:'http://www.squeak.org'.
>>   length := resp contentLength.
>>   stream := resp contentStream.
>>   [length>  0] whileTrue:[
>>     stream next: (length min: 100).
>>   ]] ensure:[stream close].
>>
> good. Its quite similar to what i did in SCouchDB.
>
> But i'd like to note, that content length is not mandatory field, and
> your example won't work for a content with undetermined length.

>> A couple of things to keep in mind though: The above doesn't deal with the
>> server returning an HTTP/1.0 response (no content-length) or other
>> specialties like chunked-encoding content transfer. That's why the
>> high-level functions are advantageous because they deal with all of that.
>>
>
> In SCouchDB I dealing with chunked-encoding content by wrapping the
> original stream with chunked stream
> which deals with this transparently, and so the end user don't have to
> deal with it by itself.
> Then, similarily, for utf-8 content, i also just wrapping a source
> stream with own utf8-decoder stream,
> and so, the content consumer don't have to care about it by itself,
> all it needs to know to use stream protocol.
>
> So, stream-wrappers is a powerful abstraction, alas, the current
> Stream package lack of good support for 'wrapper stream' such kind of
> abstraction. :(
>
>> Cheers,
>>   - Andreas
>>
>>
>

> On 11 May 2010 17:44, Hannes Hirzel <[hidden email]> wrote:
>> On 5/10/10, radoslav hodnicak <[hidden email]> wrote:
>>>
>>> Which JSON package/version are you using? I fixed a bug in the one
>>> distributed with SCouchDB few weeks ago, where it didn't encode utf8
>>> characters properly - the correct escaped form is \uNNNN - always padded
>>> to 4 Ns. that's why you get that warning, yours is only 2-3
>>>
>>> rado
>>
>> I have been using
>> http://www.squeaksource.com/JSON (over 7000 downloads)
>> in combination with WebClient.
>>
>> Thank you Rado, I found
>> http://www.squeaksource.com/SCouchDB/SCouchDB-Core-rh.8.mcz
>> and will have a look at it.
>> (Your comment: added handling of utf8 encoded input data - this is
>> necessary for couchdb-lucene which sends results directly in utf8 and
>> not \uNNNN encoded)
>>
> SCouchDB using a forked version of JSON package, which you can find in
> SCouchDB repository
> http://www.squeaksource.com/SCouchDB/JSON-Igor.Stasenko.34.mcz
>
> If you looking for that method, it can be found in Json>>unescapeUnicode
>
>
>> --Hannes
>>
>>
>>> On Mon, 10 May 2010, Hannes Hirzel wrote:
>>>
>>>> The test case made simpler
>>>>
>>>> WebClient httpPut: host, '/notes/test7' content:
>>>> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>>>>
>>>> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
>>>> JSON"}
>>>> '
>>>>
>>>> whereas
>>>>
>>>> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
>>>> type: 'text/plain'.
>>>>
>>>> gives back
>>>> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>>> '
>>>>
>>>> --Hannes
>>>
>>>
>>
>>
>
>
>
> --
> Best regards,
> Igor Stasenko AKA sig.
>
>

> 1) UFT8 conversion
> 2) Change to JSON package of Tony Garnock-Jones
> 3) My updated Test case
> 4) Conclusion
>
>
> 1) UFT8 conversion
>
> My question was:
>     How do I convert a WideString to UTF8?
>
>
> Levente answered:
>
> There are various possibilities:
> 'äbc' squeakToUtf8.
> 'äbc' convertToEncoding: 'utf-8'.
> 'äbc' convertToWithConverter: UTF8TextConverter new.
> UTF8TextConverter new encodeString: 'äbc'.
>
>
>
> 2) Change to JSON package of Tony Garnock-Jones
>
> As CouchDB stores UTF8 values I did not want to escape them with
> \uNNNN as the forked JSON package in SCouchDB does. But instead I
> wanted to keep UTF8 in the db. As Rado pointed out the UFT8 conversion
> is not correct in the original JSON package.
>
> So I did the following correction.
>
> In the class
>   String  - category *JSON-writing
>   (from package http://www.squeaksource.com/JSON)
> I replaced
>
>   jsonWriteOn: aStream
> | replacement |
> aStream nextPut: $".
> self do: [ :ch |
> (replacement := Json escapeForCharacter: ch)    "***"
> ifNil: [ aStream nextPut: ch ]
> ifNotNil: [ aStream nextPutAll: replacement ] ].
> aStream nextPut: $".
>
>
> WITH
>
>   jsonWriteOn: aStream
> aStream nextPut: $".
> aStream nextPutAll:  (UTF8TextConverter new encodeString: self).
> aStream nextPut: $".
>
>
> "*** NOTE: escapeForCharacter is incorrectly implemented in
> http://www.squeaksource.com/JSON
> and is corrected by Rado in the SCouchDB fork of the package JSON
> http://www.squeaksource.com/SCouchDB/SCouchDB-Core-rh.8.mcz"
>
>
>
> 3) My updated Test case
>
> myWideString := ('ä', 8220 asCharacter asString, Character cr, 'b').
> d := Dictionary new. d at: 'title' put:   'aTitle'. d at: 'body' put:
> myWideString.
> r := WriteStream on: String new.
> (JsonObject newFrom: d) jsonWriteOn: r.
> WebClient httpPut: host, '/notes/test24' content: r contents type:
> 'text/plain'.
>
> RESULT: OK.
>
>
>
> 4) Conclusion
>
> With the change to the JSON package I am now fine in using WebClient
> for storing objects in a couchdB.
>
> However I did not commit my change to
>   http://www.squeaksource.com/JSON
> as I do not (yet) understand the full impact of it.
>
>
> Thank you Andreas Raab, Levente Uzony and Rado Hodnicak for your help
>
> --Hannes
>
> On 5/11/10, Igor Stasenko <[hidden email]> wrote:
>> On 11 May 2010 17:44, Hannes Hirzel <[hidden email]> wrote:
>>> On 5/10/10, radoslav hodnicak <[hidden email]> wrote:
>>>>
>>>> Which JSON package/version are you using? I fixed a bug in the one
>>>> distributed with SCouchDB few weeks ago, where it didn't encode utf8
>>>> characters properly - the correct escaped form is \uNNNN - always
>>>> padded
>>>> to 4 Ns. that's why you get that warning, yours is only 2-3
>>>>
>>>> rado
>>>
>>> I have been using
>>> http://www.squeaksource.com/JSON (over 7000 downloads)
>>> in combination with WebClient.
>>>
>>> Thank you Rado, I found
>>> http://www.squeaksource.com/SCouchDB/SCouchDB-Core-rh.8.mcz
>>> and will have a look at it.
>>> (Your comment: added handling of utf8 encoded input data - this is
>>> necessary for couchdb-lucene which sends results directly in utf8 and
>>> not \uNNNN encoded)
>>>
>> SCouchDB using a forked version of JSON package, which you can find in
>> SCouchDB repository
>> http://www.squeaksource.com/SCouchDB/JSON-Igor.Stasenko.34.mcz
>>
>> If you looking for that method, it can be found in Json>>unescapeUnicode
>>
>>
>>> --Hannes
>>>
>>>
>>>> On Mon, 10 May 2010, Hannes Hirzel wrote:
>>>>
>>>>> The test case made simpler
>>>>>
>>>>> WebClient httpPut: host, '/notes/test7' content:
>>>>> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>>>>>
>>>>> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
>>>>> JSON"}
>>>>> '
>>>>>
>>>>> whereas
>>>>>
>>>>> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
>>>>> type: 'text/plain'.
>>>>>
>>>>> gives back
>>>>> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>>>> '
>>>>>
>>>>> --Hannes
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>>
>

>> So the question is: How do I convert a WideString to UTF8
>
> There are various possibilities:
> 'äbc' squeakToUtf8.
> 'äbc' convertToEncoding: 'utf-8'.
> 'äbc' convertToWithConverter: UTF8TextConverter new.
> UTF8TextConverter new encodeString: 'äbc'.
>
>
> Levente
>

> 1) UFT8 conversion
> 2) Change to JSON package of Tony Garnock-Jones
> 3) My updated Test case
> 4) Conclusion
>
>
> 1) UFT8 conversion
>
> My question was:
>    How do I convert a WideString to UTF8?
>
>
> Levente answered:
>
> There are various possibilities:
> 'äbc' squeakToUtf8.
> 'äbc' convertToEncoding: 'utf-8'.
> 'äbc' convertToWithConverter: UTF8TextConverter new.
> UTF8TextConverter new encodeString: 'äbc'.
>
>
>
> 2) Change to JSON package of Tony Garnock-Jones
>
> As CouchDB stores UTF8 values I did not want to escape them with
> \uNNNN as the forked JSON package in SCouchDB does.

> So I did the following correction.
>
> In the class
>  String  - category *JSON-writing
>  (from package http://www.squeaksource.com/JSON)
> I replaced
>
>  jsonWriteOn: aStream
>        | replacement |
>        aStream nextPut: $".
>        self do: [ :ch |
>                (replacement := Json escapeForCharacter: ch)    "***"
>                        ifNil: [ aStream nextPut: ch ]
>                        ifNotNil: [ aStream nextPutAll: replacement ] ].
>        aStream nextPut: $".
>
>
> WITH
>
>  jsonWriteOn: aStream
>        aStream nextPut: $".
>        aStream nextPutAll:  (UTF8TextConverter new encodeString: self).
>        aStream nextPut: $".
>

>
>
> 3) My updated Test case
>
> myWideString := ('ä', 8220 asCharacter asString, Character cr, 'b').
> d := Dictionary new. d at: 'title' put:   'aTitle'. d at: 'body' put:
> myWideString.
> r := WriteStream on: String new.
> (JsonObject newFrom: d) jsonWriteOn: r.
> WebClient httpPut: host, '/notes/test24' content: r contents type: 'text/plain'.
>
> RESULT: OK.
>
>
>
> 4) Conclusion
>
> With the change to the JSON package I am now fine in using WebClient
> for storing objects in a couchdB.
>
> However I did not commit my change to
>  http://www.squeaksource.com/JSON
> as I do not (yet) understand the full impact of it.
>
>
> Thank you Andreas Raab, Levente Uzony and Rado Hodnicak for your help
>
> --Hannes
>
> On 5/11/10, Igor Stasenko <[hidden email]> wrote:
>> On 11 May 2010 17:44, Hannes Hirzel <[hidden email]> wrote:
>>> On 5/10/10, radoslav hodnicak <[hidden email]> wrote:
>>>>
>>>> Which JSON package/version are you using? I fixed a bug in the one
>>>> distributed with SCouchDB few weeks ago, where it didn't encode utf8
>>>> characters properly - the correct escaped form is \uNNNN - always padded
>>>> to 4 Ns. that's why you get that warning, yours is only 2-3
>>>>
>>>> rado
>>>
>>> I have been using
>>> http://www.squeaksource.com/JSON (over 7000 downloads)
>>> in combination with WebClient.
>>>
>>> Thank you Rado, I found
>>> http://www.squeaksource.com/SCouchDB/SCouchDB-Core-rh.8.mcz
>>> and will have a look at it.
>>> (Your comment: added handling of utf8 encoded input data - this is
>>> necessary for couchdb-lucene which sends results directly in utf8 and
>>> not \uNNNN encoded)
>>>
>> SCouchDB using a forked version of JSON package, which you can find in
>> SCouchDB repository
>> http://www.squeaksource.com/SCouchDB/JSON-Igor.Stasenko.34.mcz
>>
>> If you looking for that method, it can be found in Json>>unescapeUnicode
>>
>>
>>> --Hannes
>>>
>>>
>>>> On Mon, 10 May 2010, Hannes Hirzel wrote:
>>>>
>>>>> The test case made simpler
>>>>>
>>>>> WebClient httpPut: host, '/notes/test7' content:
>>>>> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>>>>>
>>>>> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
>>>>> JSON"}
>>>>> '
>>>>>
>>>>> whereas
>>>>>
>>>>> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
>>>>> type: 'text/plain'.
>>>>>
>>>>> gives back
>>>>> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>>>> '
>>>>>
>>>>> --Hannes
>>>>
>>>>
>>>
>>>
>>
>>
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>>
>>
>
>

> On 5/11/2010 8:09 AM, Igor Stasenko wrote:
>>
>> It would be also nice to have support of other methods, as specified
>> in HTTP 1.1 protocol:
>> http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
>>
>> Then i would be able to use WebClient as a backend in SCouchDB project,
>> which currently using own subset of HTTP.
>
> What other methods do you need? There should be no problem adding any I just
> had no need for them initially.
>