[ANN] WebClient and WebServer 1.0 for Squeak

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
50 messages Options
123
Reply | Threaded
Open this post in threaded view
|

[ANN] WebClient and WebServer 1.0 for Squeak

Andreas.Raab
Folks -

I'm happy to announce the release of WebClient and WebServer 1.0 for
Squeak 4.1. I started this little side project about half a year ago
when I realized that I had just written the third almost-but-not-quite
complete HTTP server implementation internally. The thing is, HTTP
servers and clients are tremendously useful. Having a small and compact
implementation offers all sorts of interesting uses. For example, we use
HTTP as command and control interface for various backend server
aspects; be that generating status XML for processing by a front-end php
app, or profiling of the live application, or RTSP handling, or
server-to-server communication. Given all these interesting simple
applications I figured it was about time to write a *simple* general
purpose HTTP server (yes, I'm looking at you Kom).

While writing it I quickly noticed that I need an accompanying client
since HTTPSocket was just too awful for testing. It's incomplete,
inconsistent and broken in lots of interesting ways. The good thing is
that since both client and server share much of the representation, the
overhead is very small.

As a result, we now have what I think is a pretty decent HTTP server and
client implementation for Squeak 4.1 and hopefully we can nuke
HTTPSocket sometime soon. WebClient is superior in *every* respect.

Oh, one word about the name: HTTPxxx is taken, HttpXxxx is taken, so I
was left with the choice of either going for HtTpClient / HtTpServer, or
WebClient / WebServer. The name's a little broad, granted, but better
than mixed caps :-)

 From the project page (http://www.squeaksource.com/WebClient):

-------------------------------------------------------------------

Project Description
WebClient and WebServer are simple, compact, and easy to use HTTP client
and server implementations.

Installation
To install WebClient and WebServer, execute the following from a
Workspace in Squeak:

(Installer ss project: 'WebClient')
        install: 'WebClient-Core';
        install: 'WebClient-Tests';
        install: 'WebClient-Help'.

In addition to the core functionality, a set of patches is provided by
the 'WebClient-HTTP' package replacing HTTPSocket by WebClient (which
means WebClient will be used for Monticello, updates etc):

(Installer ss project: 'WebClient')
        install: 'WebClient-HTTP';

An experimental Seaside 3.0 adapter for WebServer is available in the
'WebClient-Seaside' package:

(Installer ss project: 'WebClient')
        install: 'WebClient-Seaside';

Help is provided via HelpSystem but make sure you also look at the
example in class WebClient and WebServer itself.

-------------------------------------------------------------------

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

RE: [ANN] WebClient and WebServer 1.0 for Squeak

Travis Kay
Awesome, thanks Andreas!

Travis




Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Casey Ransberger-2
In reply to this post by Andreas.Raab
Are you intent on replacing HTTPSocket in the long run? I have an RPC-ish thing that uses it, so I'd like to be clear on whether I can expect it to be around for the next release.

On Tue, May 4, 2010 at 10:14 PM, Andreas Raab <[hidden email]> wrote:
Folks -

I'm happy to announce the release of WebClient and WebServer 1.0 for Squeak 4.1. I started this little side project about half a year ago when I realized that I had just written the third almost-but-not-quite complete HTTP server implementation internally. The thing is, HTTP servers and clients are tremendously useful. Having a small and compact implementation offers all sorts of interesting uses. For example, we use HTTP as command and control interface for various backend server aspects; be that generating status XML for processing by a front-end php app, or profiling of the live application, or RTSP handling, or server-to-server communication. Given all these interesting simple applications I figured it was about time to write a *simple* general purpose HTTP server (yes, I'm looking at you Kom).

While writing it I quickly noticed that I need an accompanying client since HTTPSocket was just too awful for testing. It's incomplete, inconsistent and broken in lots of interesting ways. The good thing is that since both client and server share much of the representation, the overhead is very small.

As a result, we now have what I think is a pretty decent HTTP server and client implementation for Squeak 4.1 and hopefully we can nuke HTTPSocket sometime soon. WebClient is superior in *every* respect.

Oh, one word about the name: HTTPxxx is taken, HttpXxxx is taken, so I was left with the choice of either going for HtTpClient / HtTpServer, or WebClient / WebServer. The name's a little broad, granted, but better than mixed caps :-)

>From the project page (http://www.squeaksource.com/WebClient):

-------------------------------------------------------------------

Project Description
WebClient and WebServer are simple, compact, and easy to use HTTP client and server implementations.

Installation
To install WebClient and WebServer, execute the following from a Workspace in Squeak:

(Installer ss project: 'WebClient')
       install: 'WebClient-Core';
       install: 'WebClient-Tests';
       install: 'WebClient-Help'.

In addition to the core functionality, a set of patches is provided by the 'WebClient-HTTP' package replacing HTTPSocket by WebClient (which means WebClient will be used for Monticello, updates etc):

(Installer ss project: 'WebClient')
       install: 'WebClient-HTTP';

An experimental Seaside 3.0 adapter for WebServer is available in the 'WebClient-Seaside' package:

(Installer ss project: 'WebClient')
       install: 'WebClient-Seaside';

Help is provided via HelpSystem but make sure you also look at the example in class WebClient and WebServer itself.

-------------------------------------------------------------------

Cheers,
 - Andreas




--
Casey Ransberger


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Andreas.Raab
On 5/4/2010 11:11 PM, Casey Ransberger wrote:
> Are you intent on replacing HTTPSocket in the long run? I have an
> RPC-ish thing that uses it, so I'd like to be clear on whether I can
> expect it to be around for the next release.

In the long run: Yes. In the next release: No. And "replacing" can mean
that the public interface in HTTPSocket remains. That's what the
WebClient-HTTP package does. It patches HTTPSocket to use WebClient
while preserving the identical external interface.

At this point, I'd just like people to try it out. Since you're doing
RPCish stuff, this sounds like just the thing WebClient and WebServer
were specifically designed for, so you might give it a shot. I think
you'll quickly learn to see the advantage of WebClient over HTTPSocket :-)

Cheers,
   - Andreas

> On Tue, May 4, 2010 at 10:14 PM, Andreas Raab <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Folks -
>
>     I'm happy to announce the release of WebClient and WebServer 1.0 for
>     Squeak 4.1. I started this little side project about half a year ago
>     when I realized that I had just written the third
>     almost-but-not-quite complete HTTP server implementation internally.
>     The thing is, HTTP servers and clients are tremendously useful.
>     Having a small and compact implementation offers all sorts of
>     interesting uses. For example, we use HTTP as command and control
>     interface for various backend server aspects; be that generating
>     status XML for processing by a front-end php app, or profiling of
>     the live application, or RTSP handling, or server-to-server
>     communication. Given all these interesting simple applications I
>     figured it was about time to write a *simple* general purpose HTTP
>     server (yes, I'm looking at you Kom).
>
>     While writing it I quickly noticed that I need an accompanying
>     client since HTTPSocket was just too awful for testing. It's
>     incomplete, inconsistent and broken in lots of interesting ways. The
>     good thing is that since both client and server share much of the
>     representation, the overhead is very small.
>
>     As a result, we now have what I think is a pretty decent HTTP server
>     and client implementation for Squeak 4.1 and hopefully we can nuke
>     HTTPSocket sometime soon. WebClient is superior in *every* respect.
>
>     Oh, one word about the name: HTTPxxx is taken, HttpXxxx is taken, so
>     I was left with the choice of either going for HtTpClient /
>     HtTpServer, or WebClient / WebServer. The name's a little broad,
>     granted, but better than mixed caps :-)
>
>      >From the project page (http://www.squeaksource.com/WebClient):
>
>     -------------------------------------------------------------------
>
>     Project Description
>     WebClient and WebServer are simple, compact, and easy to use HTTP
>     client and server implementations.
>
>     Installation
>     To install WebClient and WebServer, execute the following from a
>     Workspace in Squeak:
>
>     (Installer ss project: 'WebClient')
>             install: 'WebClient-Core';
>             install: 'WebClient-Tests';
>             install: 'WebClient-Help'.
>
>     In addition to the core functionality, a set of patches is provided
>     by the 'WebClient-HTTP' package replacing HTTPSocket by WebClient
>     (which means WebClient will be used for Monticello, updates etc):
>
>     (Installer ss project: 'WebClient')
>             install: 'WebClient-HTTP';
>
>     An experimental Seaside 3.0 adapter for WebServer is available in
>     the 'WebClient-Seaside' package:
>
>     (Installer ss project: 'WebClient')
>             install: 'WebClient-Seaside';
>
>     Help is provided via HelpSystem but make sure you also look at the
>     example in class WebClient and WebServer itself.
>
>     -------------------------------------------------------------------
>
>     Cheers,
>       - Andreas
>
>
>
>
> --
> Casey Ransberger
>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Casey Ransberger-2
Funny comment inline below...

On Tue, May 4, 2010 at 11:22 PM, Andreas Raab <[hidden email]> wrote:
On 5/4/2010 11:11 PM, Casey Ransberger wrote:
Are you intent on replacing HTTPSocket in the long run? I have an
RPC-ish thing that uses it, so I'd like to be clear on whether I can
expect it to be around for the next release.

In the long run: Yes. In the next release: No. And "replacing" can mean that the public interface in HTTPSocket remains. That's what the WebClient-HTTP package does. It patches HTTPSocket to use WebClient while preserving the identical external interface.

I'm beginning to recognize a pattern in your behavior:)
 
<snip>
Cheers,
 - Andreas

--
Casey Ransberger


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

christophe.jalady
In reply to this post by Andreas.Raab
Just a simple test under Squeak 4.1: Http GET with Basic Auth is working.

But I have trouble under Pharo:
I have added to my Montecillo
MCHttpRepository
    location: 'http://www.squeaksource.com/WebClient'
and I have tried to load the WebClient core package (version 15).

A simple "Http GET" is not working. it fail trying to read the content
(ByteString>>new:streamsContent: is not found)

Do you plan to make your package "pharo compliant" ? (or maybe it should be
Pharo wich become "WebClient compliant" ?)

By the way, thank you for your great work on this project !!

Christophe.


> Folks -
>
> I'm happy to announce the release of WebClient and WebServer 1.0 for
> Squeak 4.1. I started this little side project about half a year ago
> when I realized that I had just written the third almost-but-not-quite
> complete HTTP server implementation internally. The thing is, HTTP
> servers and clients are tremendously useful. Having a small and compact
> implementation offers all sorts of interesting uses. For example, we use
> HTTP as command and control interface for various backend server
> aspects; be that generating status XML for processing by a front-end php
> app, or profiling of the live application, or RTSP handling, or
> server-to-server communication. Given all these interesting simple
> applications I figured it was about time to write a *simple* general
> purpose HTTP server (yes, I'm looking at you Kom).
>
> While writing it I quickly noticed that I need an accompanying client
> since HTTPSocket was just too awful for testing. It's incomplete,
> inconsistent and broken in lots of interesting ways. The good thing is
> that since both client and server share much of the representation, the
> overhead is very small.
>
> As a result, we now have what I think is a pretty decent HTTP server and
> client implementation for Squeak 4.1 and hopefully we can nuke
> HTTPSocket sometime soon. WebClient is superior in *every* respect.
>
> Oh, one word about the name: HTTPxxx is taken, HttpXxxx is taken, so I
> was left with the choice of either going for HtTpClient / HtTpServer, or
> WebClient / WebServer. The name's a little broad, granted, but better
> than mixed caps :-)
>
>  From the project page (http://www.squeaksource.com/WebClient):
>
> -------------------------------------------------------------------
>
> Project Description
> WebClient and WebServer are simple, compact, and easy to use HTTP client
> and server implementations.
>
> Installation
> To install WebClient and WebServer, execute the following from a
> Workspace in Squeak:
>
> (Installer ss project: 'WebClient')
> install: 'WebClient-Core';
> install: 'WebClient-Tests';
> install: 'WebClient-Help'.
>
> In addition to the core functionality, a set of patches is provided by
> the 'WebClient-HTTP' package replacing HTTPSocket by WebClient (which
> means WebClient will be used for Monticello, updates etc):
>
> (Installer ss project: 'WebClient')
> install: 'WebClient-HTTP';
>
> An experimental Seaside 3.0 adapter for WebServer is available in the
> 'WebClient-Seaside' package:
>
> (Installer ss project: 'WebClient')
> install: 'WebClient-Seaside';
>
> Help is provided via HelpSystem but make sure you also look at the
> example in class WebClient and WebServer itself.
>
> -------------------------------------------------------------------
>
> Cheers,
>    - Andreas
>
>



Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Andreas.Raab
On 5/5/2010 3:43 PM, [hidden email] wrote:
> Just a simple test under Squeak 4.1: Http GET with Basic Auth is working.

Glad to see it works for you.

> But I have trouble under Pharo:
> I have added to my Montecillo
> MCHttpRepository
>      location: 'http://www.squeaksource.com/WebClient'
> and I have tried to load the WebClient core package (version 15).
>
> A simple "Http GET" is not working. it fail trying to read the content
> (ByteString>>new:streamsContent: is not found)
>
> Do you plan to make your package "pharo compliant" ? (or maybe it should be
> Pharo wich become "WebClient compliant" ?)

Well, either way :-) I've added a 'WebClient-Pharo' package with a bunch
of patches that will make WebClient work under Pharo 1.0 but I noticed
that one bug affecting SocketStream may indicate that some other bugs in
SocketStream are not fixed either. Unfortunately WebClient and WebServer
rely heavily on correct functioning of SocketStream and my test coverage
is not 100% so there may be undetected issues. If you find any, let me
know and I'll issue another patch (and increase test coverage along the
way :-)

> By the way, thank you for your great work on this project !!

You're very welcome.

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

christophe.jalady
Looks working under pharo now.

But it looks like there is a bug with the encoding !

I try to GET a document encoded as utf-8. The header of the response contain
"content-type'->'text/html;charset=utf-8". But the characters are wrong.

I have tried a quick test by replacing "ByteString" with "WideString" in
WebResponse>>content without success.

I will try to investigate more this week-end.

Christophe




> On 5/5/2010 3:43 PM, [hidden email] wrote:
> > Just a simple test under Squeak 4.1: Http GET with Basic Auth is working.
>
> Glad to see it works for you.
>
> > But I have trouble under Pharo:
> > I have added to my Montecillo
> > MCHttpRepository
> >      location: 'http://www.squeaksource.com/WebClient'
> > and I have tried to load the WebClient core package (version 15).
> >
> > A simple "Http GET" is not working. it fail trying to read the content
> > (ByteString>>new:streamsContent: is not found)
> >
> > Do you plan to make your package "pharo compliant" ? (or maybe it should be
> > Pharo wich become "WebClient compliant" ?)
>
> Well, either way :-) I've added a 'WebClient-Pharo' package with a bunch
> of patches that will make WebClient work under Pharo 1.0 but I noticed
> that one bug affecting SocketStream may indicate that some other bugs in
> SocketStream are not fixed either. Unfortunately WebClient and WebServer
> rely heavily on correct functioning of SocketStream and my test coverage
> is not 100% so there may be undetected issues. If you find any, let me
> know and I'll issue another patch (and increase test coverage along the
> way :-)
>
> > By the way, thank you for your great work on this project !!
>
> You're very welcome.
>
> Cheers,
>    - Andreas
>
>



Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Andreas.Raab
On 5/6/2010 3:32 PM, [hidden email] wrote:
> But it looks like there is a bug with the encoding !

The behavior you are seeing is intentional. WebClient doesn't interpret
the content of a response. It simply gives you the bytes that you can
deal with in any way you see fit. That includes dealing with encodings.

> I try to GET a document encoded as utf-8. The header of the response contain
> "content-type'->'text/html;charset=utf-8". But the characters are wrong.

They are UTF-8 encoded just as the content type implies.

> I have tried a quick test by replacing "ByteString" with "WideString" in
> WebResponse>>content without success.

Yes, you need to decode the input. You can for example use the method
#utf8ToSqueak to decode the string into UTF-8.

> I will try to investigate more this week-end.

Good luck!

Cheers,
   - Andreas

>> On 5/5/2010 3:43 PM, [hidden email] wrote:
>>> Just a simple test under Squeak 4.1: Http GET with Basic Auth is working.
>>
>> Glad to see it works for you.
>>
>>> But I have trouble under Pharo:
>>> I have added to my Montecillo
>>> MCHttpRepository
>>>       location: 'http://www.squeaksource.com/WebClient'
>>> and I have tried to load the WebClient core package (version 15).
>>>
>>> A simple "Http GET" is not working. it fail trying to read the content
>>> (ByteString>>new:streamsContent: is not found)
>>>
>>> Do you plan to make your package "pharo compliant" ? (or maybe it should be
>>> Pharo wich become "WebClient compliant" ?)
>>
>> Well, either way :-) I've added a 'WebClient-Pharo' package with a bunch
>> of patches that will make WebClient work under Pharo 1.0 but I noticed
>> that one bug affecting SocketStream may indicate that some other bugs in
>> SocketStream are not fixed either. Unfortunately WebClient and WebServer
>> rely heavily on correct functioning of SocketStream and my test coverage
>> is not 100% so there may be undetected issues. If you find any, let me
>> know and I'll issue another patch (and increase test coverage along the
>> way :-)
>>
>>> By the way, thank you for your great work on this project !!
>>
>> You're very welcome.
>>
>> Cheers,
>>     - Andreas
>>
>>
>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

laza
In reply to this post by Andreas.Raab
I seem to have some issues while using a proxy with WebClient and
WebClient-HTTP.

I had a proxy configuration set in the Preferences Browser (localhost
8888). To not use any proxyserver it was sufficient before to just
erase the server name from the preferences. You just can't set an
empty string for the proxy port preference anyway.
Now WebClient only seems to check if the proxyserver name is nil but
not if it is empty and obviously fails to connect to some proxy with
blank hostname.

Further if there is some valid proxy setting in place, it returns a
400 bad request. I guess it's the missing protocol/host in the url.

HTTP/1.0 400 Bad Request
server: squid/2.6.STABLE18
date: Mon, 10 May 2010 09:26:18 GMT
content-type: text/html
content-length: 1485
expires: Mon, 10 May 2010 09:26:18 GMT
x-squid-error: ERR_INVALID_REQ 0
x-cache: MISS from xyz.host.de
x-cache-lookup: NONE from xyz.host.de:80
proxy-connection: close

[...]

GET /trunk/?C=M;O=D HTTP/1.1
User-Agent: Squeak4.1alpha-10129
Authorization: Basic bGF6YTpwb3BwZW4=
Accept: */*
Host: source.squeak.org

[...]

Alex

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Levente Uzonyi-2
On Mon, 10 May 2010, Alexander Lazarević wrote:

> I seem to have some issues while using a proxy with WebClient and
> WebClient-HTTP.
>
> I had a proxy configuration set in the Preferences Browser (localhost
> 8888). To not use any proxyserver it was sufficient before to just
> erase the server name from the preferences. You just can't set an
> empty string for the proxy port preference anyway.
> Now WebClient only seems to check if the proxyserver name is nil but
> not if it is empty and obviously fails to connect to some proxy with
> blank hostname.
>
> Further if there is some valid proxy setting in place, it returns a
> 400 bad request. I guess it's the missing protocol/host in the url.
>
> HTTP/1.0 400 Bad Request
> server: squid/2.6.STABLE18
> date: Mon, 10 May 2010 09:26:18 GMT
> content-type: text/html
> content-length: 1485
> expires: Mon, 10 May 2010 09:26:18 GMT
> x-squid-error: ERR_INVALID_REQ 0
> x-cache: MISS from xyz.host.de
> x-cache-lookup: NONE from xyz.host.de:80
> proxy-connection: close
>
> [...]
>
> GET /trunk/?C=M;O=D HTTP/1.1
> User-Agent: Squeak4.1alpha-10129
> Authorization: Basic bGF6YTpwb3BwZW4=
If that's your real password, you better change it. ;)


Levente

> Accept: */*
> Host: source.squeak.org
>
> [...]
>
> Alex
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

laza
Nahh!! :D

2010/5/10 Levente Uzonyi <[hidden email]>:

> On Mon, 10 May 2010, Alexander Lazarević wrote:
>
>> I seem to have some issues while using a proxy with WebClient and
>> WebClient-HTTP.
>>
>> I had a proxy configuration set in the Preferences Browser (localhost
>> 8888). To not use any proxyserver it was sufficient before to just
>> erase the server name from the preferences. You just can't set an
>> empty string for the proxy port preference anyway.
>> Now WebClient only seems to check if the proxyserver name is nil but
>> not if it is empty and obviously fails to connect to some proxy with
>> blank hostname.
>>
>> Further if there is some valid proxy setting in place, it returns a
>> 400 bad request. I guess it's the missing protocol/host in the url.
>>
>> HTTP/1.0 400 Bad Request
>> server: squid/2.6.STABLE18
>> date: Mon, 10 May 2010 09:26:18 GMT
>> content-type: text/html
>> content-length: 1485
>> expires: Mon, 10 May 2010 09:26:18 GMT
>> x-squid-error: ERR_INVALID_REQ 0
>> x-cache: MISS from xyz.host.de
>> x-cache-lookup: NONE from xyz.host.de:80
>> proxy-connection: close
>>
>> [...]
>>
>> GET /trunk/?C=M;O=D HTTP/1.1
>> User-Agent: Squeak4.1alpha-10129
>> Authorization: Basic bGF6YTpwb3BwZW4=
>
> If that's your real password, you better change it. ;)
>
>
> Levente
>
>> Accept: */*
>> Host: source.squeak.org
>>
>> [...]
>>
>> Alex
>>
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Hannes Hirzel
In reply to this post by Andreas.Raab
Hello

On 5/7/10, Andreas Raab <[hidden email]> wrote:
> On 5/6/2010 3:32 PM, [hidden email] wrote:
>> But it looks like there is a bug with the encoding !
>
> The behavior you are seeing is intentional. WebClient doesn't interpret
> the content of a response. It simply gives you the bytes that you can
> deal with in any way you see fit. That includes dealing with encodings.

>> I have tried a quick test by replacing "ByteString" with "WideString" in
>> WebResponse>>content without success.
>
> Yes, you need to decode the input. You can for example use the method
> #utf8ToSqueak to decode the string into UTF-8.


I have a similar problem, I want to post a document to a couchDB
instance. Without German umlauts it works fine but if I add them I get
an error message.

host :='<a href="http://localhost:5984'">http://localhost:5984'.
r := WriteStream on: (String new: 1000).
d := Dictionary new. d at: 'content' put: 'äs' squeakToUtf8.
(JsonObject newFrom: d) jsonWriteOn: r.
WebClient httpPut: host, '/notes/test5' content:  r contents  type:
'text/plain;charset=utf-8'.

The error message of the WebResponse is
'{"error":"bad_request","reason":"invalid UTF-8 JSON"}
'

The same thing happens if I omit #squeakToUtf8

If I inspect
   r contents
I get
  '{"content":"\uC3\uA4s"}'


Contrariwise if I post

    host :='<a href="http://localhost:5984'">http://localhost:5984'.
    r := WriteStream on: (String new: 1000).
    d := Dictionary new. d at: 'content' put: 'abc' squeakToUtf8.
    (JsonObject newFrom: d) jsonWriteOn: r.
    WebClient httpPut: host, '/notes/test5' content:  r contents
type: 'text/plain;charset=utf-8'.

I get as answer
    '{"ok":true,"id":"test5","rev":"1-f40e52919735ae6775af3d388361b3da"}
    '

How is this conversion supposed to be done?

Hints and comments are appreciated.

--Hannes

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Hannes Hirzel
The test case made simpler

WebClient httpPut: host, '/notes/test7' content:
'{"content":"\uC3\uA4s"}' type: 'text/plain'.

gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8 JSON"}
'

whereas

WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
type: 'text/plain'.

gives back
'{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
'

--Hannes

On 5/10/10, Hannes Hirzel <[hidden email]> wrote:

> Hello
>
> On 5/7/10, Andreas Raab <[hidden email]> wrote:
>> On 5/6/2010 3:32 PM, [hidden email] wrote:
>>> But it looks like there is a bug with the encoding !
>>
>> The behavior you are seeing is intentional. WebClient doesn't interpret
>> the content of a response. It simply gives you the bytes that you can
>> deal with in any way you see fit. That includes dealing with encodings.
>
>>> I have tried a quick test by replacing "ByteString" with "WideString" in
>>> WebResponse>>content without success.
>>
>> Yes, you need to decode the input. You can for example use the method
>> #utf8ToSqueak to decode the string into UTF-8.
>
>
> I have a similar problem, I want to post a document to a couchDB
> instance. Without German umlauts it works fine but if I add them I get
> an error message.
>
> host :='<a href="http://localhost:5984'">http://localhost:5984'.
> r := WriteStream on: (String new: 1000).
> d := Dictionary new. d at: 'content' put: 'äs' squeakToUtf8.
> (JsonObject newFrom: d) jsonWriteOn: r.
> WebClient httpPut: host, '/notes/test5' content:  r contents  type:
> 'text/plain;charset=utf-8'.
>
> The error message of the WebResponse is
> '{"error":"bad_request","reason":"invalid UTF-8 JSON"}
> '
>
> The same thing happens if I omit #squeakToUtf8
>
> If I inspect
>    r contents
> I get
>   '{"content":"\uC3\uA4s"}'
>
>
> Contrariwise if I post
>
>     host :='<a href="http://localhost:5984'">http://localhost:5984'.
>     r := WriteStream on: (String new: 1000).
>     d := Dictionary new. d at: 'content' put: 'abc' squeakToUtf8.
>     (JsonObject newFrom: d) jsonWriteOn: r.
>     WebClient httpPut: host, '/notes/test5' content:  r contents
> type: 'text/plain;charset=utf-8'.
>
> I get as answer
>     '{"ok":true,"id":"test5","rev":"1-f40e52919735ae6775af3d388361b3da"}
>     '
>
> How is this conversion supposed to be done?
>
> Hints and comments are appreciated.
>
> --Hannes
>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Hannes Hirzel
Ok, the following works

WebClient httpPut: host, '/notes/test11' content: '{"content":"',
(UTF8TextConverter encodeByteString: 'äbc'), '"}' type: 'text/plain'.

So I can manage.

--Hannes

On 5/10/10, Hannes Hirzel <[hidden email]> wrote:

> The test case made simpler
>
> WebClient httpPut: host, '/notes/test7' content:
> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>
> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
> JSON"}
> '
>
> whereas
>
> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
> type: 'text/plain'.
>
> gives back
> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
> '
>
> --Hannes
>
> On 5/10/10, Hannes Hirzel <[hidden email]> wrote:
>> Hello
>>
>> On 5/7/10, Andreas Raab <[hidden email]> wrote:
>>> On 5/6/2010 3:32 PM, [hidden email] wrote:
>>>> But it looks like there is a bug with the encoding !
>>>
>>> The behavior you are seeing is intentional. WebClient doesn't interpret
>>> the content of a response. It simply gives you the bytes that you can
>>> deal with in any way you see fit. That includes dealing with encodings.
>>
>>>> I have tried a quick test by replacing "ByteString" with "WideString"
>>>> in
>>>> WebResponse>>content without success.
>>>
>>> Yes, you need to decode the input. You can for example use the method
>>> #utf8ToSqueak to decode the string into UTF-8.
>>
>>
>> I have a similar problem, I want to post a document to a couchDB
>> instance. Without German umlauts it works fine but if I add them I get
>> an error message.
>>
>> host :='<a href="http://localhost:5984'">http://localhost:5984'.
>> r := WriteStream on: (String new: 1000).
>> d := Dictionary new. d at: 'content' put: 'äs' squeakToUtf8.
>> (JsonObject newFrom: d) jsonWriteOn: r.
>> WebClient httpPut: host, '/notes/test5' content:  r contents  type:
>> 'text/plain;charset=utf-8'.
>>
>> The error message of the WebResponse is
>> '{"error":"bad_request","reason":"invalid UTF-8 JSON"}
>> '
>>
>> The same thing happens if I omit #squeakToUtf8
>>
>> If I inspect
>>    r contents
>> I get
>>   '{"content":"\uC3\uA4s"}'
>>
>>
>> Contrariwise if I post
>>
>>     host :='<a href="http://localhost:5984'">http://localhost:5984'.
>>     r := WriteStream on: (String new: 1000).
>>     d := Dictionary new. d at: 'content' put: 'abc' squeakToUtf8.
>>     (JsonObject newFrom: d) jsonWriteOn: r.
>>     WebClient httpPut: host, '/notes/test5' content:  r contents
>> type: 'text/plain;charset=utf-8'.
>>
>> I get as answer
>>     '{"ok":true,"id":"test5","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>     '
>>
>> How is this conversion supposed to be done?
>>
>> Hints and comments are appreciated.
>>
>> --Hannes
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Hannes Hirzel
Unfortunately UTF8TextConverter cannot deal with non-Latin1
characters. So it's usefulness is limited.

--Hannes

On 5/10/10, Hannes Hirzel <[hidden email]> wrote:

> Ok, the following works
>
> WebClient httpPut: host, '/notes/test11' content: '{"content":"',
> (UTF8TextConverter encodeByteString: 'äbc'), '"}' type: 'text/plain'.
>
> So I can manage.
>
> --Hannes
>
> On 5/10/10, Hannes Hirzel <[hidden email]> wrote:
>> The test case made simpler
>>
>> WebClient httpPut: host, '/notes/test7' content:
>> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>>
>> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
>> JSON"}
>> '
>>
>> whereas
>>
>> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
>> type: 'text/plain'.
>>
>> gives back
>> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
>> '
>>
>> --Hannes
>>
>> On 5/10/10, Hannes Hirzel <[hidden email]> wrote:
>>> Hello
>>>
>>> On 5/7/10, Andreas Raab <[hidden email]> wrote:
>>>> On 5/6/2010 3:32 PM, [hidden email] wrote:
>>>>> But it looks like there is a bug with the encoding !
>>>>
>>>> The behavior you are seeing is intentional. WebClient doesn't interpret
>>>> the content of a response. It simply gives you the bytes that you can
>>>> deal with in any way you see fit. That includes dealing with encodings.
>>>
>>>>> I have tried a quick test by replacing "ByteString" with "WideString"
>>>>> in
>>>>> WebResponse>>content without success.
>>>>
>>>> Yes, you need to decode the input. You can for example use the method
>>>> #utf8ToSqueak to decode the string into UTF-8.
>>>
>>>
>>> I have a similar problem, I want to post a document to a couchDB
>>> instance. Without German umlauts it works fine but if I add them I get
>>> an error message.
>>>
>>> host :='<a href="http://localhost:5984'">http://localhost:5984'.
>>> r := WriteStream on: (String new: 1000).
>>> d := Dictionary new. d at: 'content' put: 'äs' squeakToUtf8.
>>> (JsonObject newFrom: d) jsonWriteOn: r.
>>> WebClient httpPut: host, '/notes/test5' content:  r contents  type:
>>> 'text/plain;charset=utf-8'.
>>>
>>> The error message of the WebResponse is
>>> '{"error":"bad_request","reason":"invalid UTF-8 JSON"}
>>> '
>>>
>>> The same thing happens if I omit #squeakToUtf8
>>>
>>> If I inspect
>>>    r contents
>>> I get
>>>   '{"content":"\uC3\uA4s"}'
>>>
>>>
>>> Contrariwise if I post
>>>
>>>     host :='<a href="http://localhost:5984'">http://localhost:5984'.
>>>     r := WriteStream on: (String new: 1000).
>>>     d := Dictionary new. d at: 'content' put: 'abc' squeakToUtf8.
>>>     (JsonObject newFrom: d) jsonWriteOn: r.
>>>     WebClient httpPut: host, '/notes/test5' content:  r contents
>>> type: 'text/plain;charset=utf-8'.
>>>
>>> I get as answer
>>>     '{"ok":true,"id":"test5","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>>     '
>>>
>>> How is this conversion supposed to be done?
>>>
>>> Hints and comments are appreciated.
>>>
>>> --Hannes
>>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Levente Uzonyi-2
On Mon, 10 May 2010, Hannes Hirzel wrote:

> Unfortunately UTF8TextConverter cannot deal with non-Latin1
> characters. So it's usefulness is limited.

UTF8TextConverter can deal with non-latin1 characters. I
think you're trying to pass a WideString to #encodeByteString: which
obviously doesn't work.


Levente

>
> --Hannes
>
> On 5/10/10, Hannes Hirzel <[hidden email]> wrote:
>> Ok, the following works
>>
>> WebClient httpPut: host, '/notes/test11' content: '{"content":"',
>> (UTF8TextConverter encodeByteString: 'äbc'), '"}' type: 'text/plain'.
>>
>> So I can manage.
>>
>> --Hannes
>>
>> On 5/10/10, Hannes Hirzel <[hidden email]> wrote:
>>> The test case made simpler
>>>
>>> WebClient httpPut: host, '/notes/test7' content:
>>> '{"content":"\uC3\uA4s"}' type: 'text/plain'.
>>>
>>> gives back as answer: '{"error":"bad_request","reason":"invalid UTF-8
>>> JSON"}
>>> '
>>>
>>> whereas
>>>
>>> WebClient httpPut: host, '/notes/test8' content: '{"content":"abc"}'
>>> type: 'text/plain'.
>>>
>>> gives back
>>> '{"ok":true,"id":"test8","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>> '
>>>
>>> --Hannes
>>>
>>> On 5/10/10, Hannes Hirzel <[hidden email]> wrote:
>>>> Hello
>>>>
>>>> On 5/7/10, Andreas Raab <[hidden email]> wrote:
>>>>> On 5/6/2010 3:32 PM, [hidden email] wrote:
>>>>>> But it looks like there is a bug with the encoding !
>>>>>
>>>>> The behavior you are seeing is intentional. WebClient doesn't interpret
>>>>> the content of a response. It simply gives you the bytes that you can
>>>>> deal with in any way you see fit. That includes dealing with encodings.
>>>>
>>>>>> I have tried a quick test by replacing "ByteString" with "WideString"
>>>>>> in
>>>>>> WebResponse>>content without success.
>>>>>
>>>>> Yes, you need to decode the input. You can for example use the method
>>>>> #utf8ToSqueak to decode the string into UTF-8.
>>>>
>>>>
>>>> I have a similar problem, I want to post a document to a couchDB
>>>> instance. Without German umlauts it works fine but if I add them I get
>>>> an error message.
>>>>
>>>> host :='<a href="http://localhost:5984'">http://localhost:5984'.
>>>> r := WriteStream on: (String new: 1000).
>>>> d := Dictionary new. d at: 'content' put: 'äs' squeakToUtf8.
>>>> (JsonObject newFrom: d) jsonWriteOn: r.
>>>> WebClient httpPut: host, '/notes/test5' content:  r contents  type:
>>>> 'text/plain;charset=utf-8'.
>>>>
>>>> The error message of the WebResponse is
>>>> '{"error":"bad_request","reason":"invalid UTF-8 JSON"}
>>>> '
>>>>
>>>> The same thing happens if I omit #squeakToUtf8
>>>>
>>>> If I inspect
>>>>    r contents
>>>> I get
>>>>   '{"content":"\uC3\uA4s"}'
>>>>
>>>>
>>>> Contrariwise if I post
>>>>
>>>>     host :='<a href="http://localhost:5984'">http://localhost:5984'.
>>>>     r := WriteStream on: (String new: 1000).
>>>>     d := Dictionary new. d at: 'content' put: 'abc' squeakToUtf8.
>>>>     (JsonObject newFrom: d) jsonWriteOn: r.
>>>>     WebClient httpPut: host, '/notes/test5' content:  r contents
>>>> type: 'text/plain;charset=utf-8'.
>>>>
>>>> I get as answer
>>>>     '{"ok":true,"id":"test5","rev":"1-f40e52919735ae6775af3d388361b3da"}
>>>>     '
>>>>
>>>> How is this conversion supposed to be done?
>>>>
>>>> Hints and comments are appreciated.
>>>>
>>>> --Hannes
>>>>
>>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Andreas.Raab
In reply to this post by laza
On 5/10/2010 2:40 AM, Alexander Lazarević wrote:

> I seem to have some issues while using a proxy with WebClient and
> WebClient-HTTP.
>
> I had a proxy configuration set in the Preferences Browser (localhost
> 8888). To not use any proxyserver it was sufficient before to just
> erase the server name from the preferences. You just can't set an
> empty string for the proxy port preference anyway.
> Now WebClient only seems to check if the proxyserver name is nil but
> not if it is empty and obviously fails to connect to some proxy with
> blank hostname.

Thanks for testing. The issue should be fixed in the repository  in
WebClient-Core-ar.18.

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Squeaksource 401 handling (Re: [ANN] WebClient and WebServer 1.0 for Squeak)

Andreas.Raab
In reply to this post by Levente Uzonyi-2
On 5/10/2010 3:29 AM, Levente Uzonyi wrote:
> On Mon, 10 May 2010, Alexander Lazarević wrote:
>>
>> GET /trunk/?C=M;O=D HTTP/1.1
>> User-Agent: Squeak4.1alpha-10129
>> Authorization: Basic xxxxxxxxxxxxxx
>
> If that's your real password, you better change it. ;)

BTW, does anyone know how to fix this problem? The issue can be
illustrated like this: When having a private repository (global: no
access) Squeaksource does not respond with a 401 when trying to read it.
Instead, it returns an empty listing:

   WebClient httpGet: 'http://www.squeaksource.com/Secret/'

On the other hand, when you try to put data into the server, it does
respond correctly with a 401:

   WebClient
        httpPut: 'http://www.squeaksource.com/Secret/test'
        content: 'foo' type: 'text/plain'.

Because the initial get doesn't properly respond with a 401 the client
is *forced* to send the initials upfront with basic authentication.
Otherwise there'd be no need to use basic auth.

Any ideas anyone? There must be a way by which the get can be protected
like the put operation.

Cheers,
   - Andreas

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] WebClient and WebServer 1.0 for Squeak

Hannes Hirzel
In reply to this post by Levente Uzonyi-2
On 5/10/10, Levente Uzonyi <[hidden email]> wrote:

> On Mon, 10 May 2010, Hannes Hirzel wrote:
>
>> Unfortunately UTF8TextConverter cannot deal with non-Latin1
>> characters. So it's usefulness is limited.
>
> UTF8TextConverter can deal with non-latin1 characters. I
> think you're trying to pass a WideString to #encodeByteString: which
> obviously doesn't work.
>
>
> Levente
>

Yes I am passing aWideString to
    #encodeByteString:

as this is the only conversion method UTF8TextConverter.

And you're right I should pass a ByteString.

However as the case
   ('ä', 8220 asCharacter asString) asByteString   "A"
shows in comparison to
  ('ä', 65 asCharacter asString) asByteString      "B"

I get only in case "B" a ByteString, in case "A" it remains a WideString.

So the question is: How do I convert a WideString to UTF8 as
UTF8TextConverter is limited to code points from 0...255 and I want
the full Unicode range?

Or put the question otherwise: Is there a textconverter which
implements the following algorithm
http://dsc.sun.com/dev/gadc/technicalpublications/articles/utf8.html

-Hannes

123