Accented characters

Hi,

I would like to enter accented characters in textarea and textinput. Is
this known to be working on 2.6b1 ? I've tried with WAKom and
WAKomEncoded without success. How can I do this please ?


Thank you very much
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Hi

Yes, this should work as long as you tell the truth about the
content-encoding. What does telling the truth mean? It means what you
tell the content-encoding is and what it actually is must match.

So what are your options

external: utf-8
internel: utf-8 (ByteString)
kom-adapter: WAKom
This is probably the easiest and most performant, remember to convert
your literals with non-ASCII characters to utf-8 (#squeakToIso
#isoToUtf8). The disadvantage is that you can't directly index into
such a string or ask for its size (besides beeing empty).
Concatenation however works as long as you concatenate only utf-8
Strings.

external: latin-1
internal: latin-1 (ByteString)
kom-adapter: WAKom
This also easy to set up. You need to make sure the content-type of
WAResponse gets initialized to iso-8859-1. The easiest way is probably
a custom session subclass that overrides #contentType. Remember to
convert your literals with non-ASCII characters to latin-1. This is
not needed for >= 3.8 (have a look at #squeakToIso to be sure, in this
case you also don't need to send #squeakToIso in the case above).
Performance should be the same as above. This approach lets you
directly index into strings and ask for their size. However, now that
we have Unicode, Latin-1 is a legacy encoding.

external: utf-8
internal: unicode (WideString)
kom-adapter: WAKomEncoded
This is probably the most sophisticated solution and requires >= 3.8.
I don't know wheter you need to convert you literals but there's any
easy way to find out ;). There were once said to be performance issues
with this approach. I don't know wheter this holds still true. This
approach also lets you directly index into strings and ask for their
size.
In gerenal this could use any external encoding supported by squeak
(eg. utf-16) but WAKomEncoded would need to be adapted.

The only possible show stopper that comes in my mind is:
http://bugs.impara.de/view.php?id=3784
But is affects only outdated 3.9 versions.

Philippe


2006/7/22, Damien Cassou <[hidden email]>: _______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall wrote:
> Hi

Hi Philippe,

thank you very much for this very interesting resume of the different
possibilities.

Suppose I would like to use the third option which seems to be the
future ones:


First I need to do is use WAKomEncoded. But I do not understand what I
need to do to use it in my application. Can you explain me please ?


Thanks
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> First I need to do is use WAKomEncoded. But I do not understand what I
> need to do to use it in my application. Can you explain me please ?

Well instead of WAKom you use WAKomEncoded.

WAKom stop.
WAKomEncoded startOn: 8080.

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall wrote:
>> First I need to do is use WAKomEncoded. But I do not understand what I
>> need to do to use it in my application. Can you explain me please ?
>
> Well instead of WAKom you use WAKomEncoded.
>
> WAKom stop.
> WAKomEncoded startOn: 8080.

Ok, I wasn't clear at all, sorry. I understand I need to use
WAKomEncoded and how to use it. What I miss now is how to convert
strings from Seaside input to Squeak model and back. Thank you Philippe.
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2006/7/24, Damien Cassou <[hidden email]>:
You don't need to convert anything. If you use WAKomEncoded it does
the converting. You just have to make sure that the strings that you
give to seaside but don't come from seaside (literals) have the right
encoding.

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall wrote:
It doesn't work. I wrote a very simple task:

TestTask>>go
   | value |
   value _ self request: 'A string'.
   self inform: value.


I executed this:

WAKom stop.
WAKomEncoded  startOn: 8080.


And then I entered a simple 'ö' in the text input within firefox in
which utf-8 is selected.

This is the result of clicking the 'Ok' button:

"
   Internal Server Error
   Error: Invalid utf8 input detected
"


My image is 3.9b-7044 and I use Seaside2.6b1-dc.62

Any advice ?


Thank you
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2006/7/24, Damien Cassou <[hidden email]>:
You're right it's broken. I was not aware of this Also external: utf-8
internal: utf-8 doesn't work in 3.9 although it does in 3.7.

> My image is 3.9b-7044 and I use Seaside2.6b1-dc.62
>
> Any advice ?

Switch to an older Squeak (3.8 or 3.7) or fix it yourself. Sorry if
this sounds harsh but these are the two most likely ways to get it
working.

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall wrote:

How is it possible that nobody cares about accented characters within
Seaside ?
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

My little experiment on Squeak3.9b-7044 using WAKom without any conversion:

- seaside input method works correctly and the model get the right value
(at least, if I inspect it in squeak I get the right accented character)
- during the answering, the WAResponse contained the valid accented
character printable within Squeak debugger
- the only problem is in html source code I can read in my web browser.
There is a '?' instead of the letter.

What is the encoding used by Squeak3.9b-7044 ? It seems to be UTF-8
because WAKom does not convert anything if I understand correctely.

Can somebody help me and make accented characters work on Seaside ?
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

On Jul 24, 2006, at 7:45 AM, Damien Cassou wrote:
>
>
> How is it possible that nobody cares about accented characters  
> within Seaside ?

Speaking for myself: I certainly care about them, but I use Squeak  
3.7.  The UTF-8 support from 3.8 mostly tends to complicate the issue  
(of course, accented characters don't show up correctly in inspectors  
etc, but that's a price I'm willing to pay).

I'm surprised things are now broken in 3.8/3.9, however -  
WAKomEncoded *used* to work, didn't it?

Avi
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Avi Bryant wrote:
Everything seems to work in 3.8 if you use WAKomEncoded. However, in 3.9
neither WAKomEncoded (which answers Invalid UTF-8 input) nor WAKom
(which answers a '?') are working.

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

I saw this problem when I was preparing Seaside for the 3.9 kernel
image. But I expected that it was problem only of this image, not the
general problem of Squeak 3.9. I use unicode characters (Czech) in
production every day for long time (WAKomEncoded code is partly mine)
without problems but this images are still based on version 3.8.

-- Pavel

On 7/24/06, Damien Cassou <[hidden email]> wrote: _______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

It would be great if we could pindown the problem with a couple of  
tests.
Sorry guys that we broke some code.

Stef

On 24 juil. 06, at 20:11, Pavel Krivanek wrote:

_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2006/7/24, Damien Cassou <[hidden email]>:
Ok, the problem is that WAKom already does the utf-8 decoding of the
input fields (what WAKomEncoded was supposed to do). This is because
he sends #unescapePercents and this already does the utf-8 decoding in
3.9.

Happy fixing.

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

2006/7/25, Philippe Marschall <[hidden email]>:
There is now Seaside2.6b1-pmm.65 a with an experimental WAKomEncoded
for Squeak 3.9 called WAKomEncoded39. At least your example works.

A few notes about this:
- does (should) not break any existing, working code. This includes 3.8 and 3.7.
- no changes in Squeak or Kom
- no additional code, only this additional class

Getting WAKom to work is a bit trickier. I requires hacks, you can choose where.

if Squeak:
change: #unescapePercents to use latin-1 instead of utf-8
This might break SqueakLand or SmallLand code.

if Kom:
Changes sends from #unescapePercents to
unescapePercentsWithTextEncoding: 'latin-1'
This is not at all portable to 3.8 or 3.7.

Note that these are really just hacks. You might want to use them
internally (they get the job done, for now at least) but they are no
general solution.

And this works for any encoding not just Latin-1. The
Latin1TextConverter just happens to the text converter that does not
decoding.
(Yeah, really great name)

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Philippe Marschall wrote:

Hey Philippe,

very great job. I tried lots of UTF-8 characters and everything worked
perfectly. Really great.


> A few notes about this:
> - does (should) not break any existing, working code. This includes 3.8
> and 3.7.
> - no changes in Squeak or Kom
> - no additional code, only this additional class


If I understand correctly your code, your WAKomEncoded39 use half of the
WAKom (for the answer) and half of WAKomEncoded (for the request).


> Getting WAKom to work is a bit trickier. I requires hacks, you can
> choose where.


Why one would need this if WAKomEncoded39 works ?

How can I install my image in seasidehosting using WAKomEncoded39 ? Is
it possible ?


Thanks
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> If I understand correctly your code, your WAKomEncoded39 use half of the
> WAKom (for the answer) and half of WAKomEncoded (for the request).

No, the other way around. It uses WAKomEncoded for the response and
WAKom for the request since WAKom already automatically decodes the
request in 3.9

> > Getting WAKom to work is a bit trickier. I requires hacks, you can
> > choose where.
>
>
> Why one would need this if WAKomEncoded39 works ?

Well there are several possible reasons. Three come into my mind
- You already have a lot of utf-8 strings in the image, for example if
you're migrating from 3.7
- Your backend uses utf-8 (eg. PostgreS). utf-8 <-> WideString <->
utf-8 doesn't really make sense
- You don't believe in WideStrings

> How can I install my image in seasidehosting using WAKomEncoded39 ? Is
> it possible ?

Good question. Probably the best way is to way until Lukas is back
from holidays (next week) and hear what he and Avi (and other people)
think about it.

But then maybe we just ship it and revert it later if he has any
problems with it.

If you want it now rename WAKomEncoded to WAKomEncoded38 and
WAKomEncoded39 to WAKomEncoded and then select WAKomEncoded in the web
interface.

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

Thank you very much Philippe. There is still something I do not understand:

- What is the encoding used in WideString ? You told it is unicode, but
what is unicode if not UTF-{8,16,32} ?
- If the web browser accept and answers in UTF-8 and if Squeak stores
UTF-8 un bytestrings, why one need any conversion ?

In fact, my problem is I don't understand why conversions are necessary.
Everything should be UTF-8 and no question has to be asked.


Thank you


Philippe Marschall wrote:
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside

> - What is the encoding used in WideString ? You told it is unicode, but
> what is unicode if not UTF-{8,16,32} ?

It stores the code point directly. So if you want, you can think of it
as utf-32. Except that it's not 32 bit ;)

> - If the web browser accept and answers in UTF-8 and if Squeak stores
> UTF-8 un bytestrings, why one need any conversion ?

You don't. That's why some people would like to see WAKom working.

> In fact, my problem is I don't understand why conversions are necessary.
> Everything should be UTF-8 and no question has to be asked.

No.

Philippe
_______________________________________________
Seaside mailing list
[hidden email]
http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside