How do we get a string with a specific encodings

> Hi guys
>
> I would like to know how I can specify the encodings of my strings.
>
> Stef
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
>
>
>

> Could you be a bit more specific, please? Do you mean encodings  
> within your smalltalk environment, or do you mean specify the  
> encoding type for the final html page (such as the type received  
> from textInput callbacks)?
>
> John.
>
> www.pinesoft.co.uk
>
> stephane ducasse wrote:
>> Hi guys
>> I would like to know how I can specify the encodings of my strings.
>> Stef
>> _______________________________________________
>> Seaside mailing list
>> [hidden email]
>> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>
>
> Pinesoft Computers are registered in England, Registered number:  
> 2914825. Registered office: 266-268 High Street, Waltham Cross,  
> Herts, EN8 7EA
>
>
>
> This message has been scanned for viruses by BlackSpider  
> MailControl - www.blackspider.com
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

> On Thu, 2007-08-02 at 13:09 +0200, stephane ducasse wrote:
> > Hi john
> >
> > first I would like to know how I can get a string in latin1 or utf my
> > image?
>
> That depends on the image version and WAKom variant you are using.
> If you use a squeak 3.9 image and the WAKomEncoded39 server than
> everything is converted to utf-8 before it is being transfered
> to the client.

> >> That depends on the image version and WAKom variant you are using.
> >> If you use a squeak 3.9 image and the WAKomEncoded39 server than
> >> everything is converted to utf-8 before it is being transfered
> >> to the client.
> >
> > It also depends where you get the content from. A from field? A file
> > upload? A file on disk?
> > Really more information would help.
>
> In fact it comes from two sources.
>         - one on a file (that normally I read with a converter to be sure)

> Now I was also wondering how I can simply specify a string with an
> encoding in Squeak
> in the image since I have no clue about that.
>
> Stef
> >
> > Cheers
> > Philippe
> > _______________________________________________
> > Seaside mailing list
> > [hidden email]
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> >
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

>>>> That depends on the image version and WAKom variant you are using.
>>>> If you use a squeak 3.9 image and the WAKomEncoded39 server than
>>>> everything is converted to utf-8 before it is being transfered
>>>> to the client.
>>>
>>> It also depends where you get the content from. A from field? A file
>>> upload? A file on disk?
>>> Really more information would help.
>>
>> In fact it comes from two sources.
>>         - one on a file (that normally I read with a converter to  
>> be sure)
>
> What encoding does the file have?

> >>>> That depends on the image version and WAKom variant you are using.
> >>>> If you use a squeak 3.9 image and the WAKomEncoded39 server than
> >>>> everything is converted to utf-8 before it is being transfered
> >>>> to the client.
> >>>
> >>> It also depends where you get the content from. A from field? A file
> >>> upload? A file on disk?
> >>> Really more information would help.
> >>
> >> In fact it comes from two sources.
> >>         - one on a file (that normally I read with a converter to  
> >> be sure)
> >
> > What encoding does the file have?
>
> utf8 but I could change that.
> It seems that latin1 would be better.
>
> >>         - from input fields.
> >
> > Are you running an encoded server adapter or WAKom?
>
> Just WAKom but I should move to use the WAKomEncoded39 version
>
> >> Now I was also wondering how I can simply specify a string with an
> >> encoding in Squeak
> >> in the image since I have no clue about that.
>
> do you know the anwser that the last question?

> On Thu, 2007-08-02 at 15:49 +0200, stephane ducasse wrote:
> > >>>> That depends on the image version and WAKom variant you are using.
> > >>>> If you use a squeak 3.9 image and the WAKomEncoded39 server than
> > >>>> everything is converted to utf-8 before it is being transfered
> > >>>> to the client.
> > >>>
> > >>> It also depends where you get the content from. A from field? A file
> > >>> upload? A file on disk?
> > >>> Really more information would help.
> > >>
> > >> In fact it comes from two sources.
> > >>         - one on a file (that normally I read with a converter to
> > >> be sure)
> > >
> > > What encoding does the file have?
> >
> > utf8 but I could change that.
> > It seems that latin1 would be better.
> >
> > >>         - from input fields.
> > >
> > > Are you running an encoded server adapter or WAKom?
> >
> > Just WAKom but I should move to use the WAKomEncoded39 version
> >
> > >> Now I was also wondering how I can simply specify a string with an
> > >> encoding in Squeak
> > >> in the image since I have no clue about that.
> >
> > do you know the anwser that the last question?
>
> You better not. I know, Phillippe will disagree :)

> If you have trouble with this
> you can ask me and I can provide you with a fix. Or your searching
> the archives with subjects:
>
> - [Seaside] 3.9 and encoding in multipart fields
>
> Norbert
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

>
>> My opinion
>> is that you should convert every string you get from outside
>> the image to the internal format squeak uses. That is a controlled
>> setting and you are able to use size and other methods on those
>> strings without worry.
>
> Unfortunately even if you have WideStrings you can not rely on that
> all the methods for Strings work since some are broken for
> WideStrings.

>> The get your file to the web with the
>> correct encoding you read in the file convert it from utf-8 to
>> the internal squeak encoding. Then you should use WAKomEncoded39
>> that will do the conversion to utf-8 when the string is about
>> to leave the image.
>
> That is of course only true if you get your Strings into the image via
> Seaside and not via FileStream. And of course WAKomEncoded39 does not
> work in Squeak 3.9 if you have the special version of Kom distributed
> with the SeasideInstaller.

> >
> > Sorry, I didn't get the question. Could you please rephrase and be
> > very exact about:
> > - where your strings come from
>
> either from a file with an encoding (utf8 but could be latin1)
>
> > - what encoding these strings have
> utf-8
>
> > - what encoding these strings should have in the end
>
> I do not know.

> Now when the user type text in input fields I would like to have the
> same encoding.
>
> Then I was also curious about the use in the image of encoded strings
> (ie how to create them, convert....)
>
>
> >
> > Just "I have an encoding problem" is not really helpful at all
> > especially if asking for answers before supplying any information.
>
> sorry
>
> >
> >> My opinion
> >> is that you should convert every string you get from outside
> >> the image to the internal format squeak uses. That is a controlled
> >> setting and you are able to use size and other methods on those
> >> strings without worry.
> >
> > Unfortunately even if you have WideStrings you can not rely on that
> > all the methods for Strings work since some are broken for
> > WideStrings.
>
> How can I see that?

> >> The get your file to the web with the
> >> correct encoding you read in the file convert it from utf-8 to
> >> the internal squeak encoding. Then you should use WAKomEncoded39
> >> that will do the conversion to utf-8 when the string is about
> >> to leave the image.
> >
> > That is of course only true if you get your Strings into the image via
> > Seaside and not via FileStream. And of course WAKomEncoded39 does not
> > work in Squeak 3.9 if you have the special version of Kom distributed
> > with the SeasideInstaller.
>
> arghhhh :)

> >
> >> From input fields it is nearly the same except
> >> the fact that WAKomEncoded39 doesn't do conversion for fields if
> >> the request is multipart (that means mostly you uploaded something
> >> like image along with the text fields).
> >
> > This is fixed in 2.8. The only place where WAKomEncoded39 does not
> > conversion is for fileuploads because we have no way of telling what
> > encoding the file has or if it's even a text file.
>
> ok
> so I should wait and use 2.8.

> You need to make a decision wheter you want utf8 or widestrings/"new
> squeak encoding" in your image.
> utf8 strings -> WAKom
> WideString -> WAKomEncoded39
>
> Make sure you convert the strings you read from disk to the machting
> encoding. You can convert a string to widestring/"new squeak encoding"
> by sending #convertFromEncoding: and can convert a widestring/"new
> squeak encoding" to something else by sending #convertToEncoding:
>
> WideStrings have a history of slow performance and bugs (there are
> still known, open bugs). There is no way of telling which methods and
> primitives don't work. For utf-8 strings you can rely on pretty much
> everything but concatenation being broken (including #size). As for
> inspector support both of them don't shine. Your choice.
>
>> Now when the user type text in input fields I would like to have the
>> same encoding.
>>
>> Then I was also curious about the use in the image of encoded strings
>> (ie how to create them, convert....)
>>
>>
>>>
>>> Just "I have an encoding problem" is not really helpful at all
>>> especially if asking for answers before supplying any information.
>>
>> sorry
>>
>>>
>>>> My opinion
>>>> is that you should convert every string you get from outside
>>>> the image to the internal format squeak uses. That is a controlled
>>>> setting and you are able to use size and other methods on those
>>>> strings without worry.
>>>
>>> Unfortunately even if you have WideStrings you can not rely on that
>>> all the methods for Strings work since some are broken for
>>> WideStrings.
>>
>> How can I see that?
>
> Stuff like #match: throwing an exception if the argument is a  
> WideString.
>
>> It would be good to report (if this is not already done)
>
> Done.
>
>>> Additionally you won't be able to inspect your non-latin
>>> Squeak WideStrings because they have no language tag and there is no
>>> way of adding one.
>>
>> ahhh.
>> What do you mean is that we cannot inspect utf-8 wideStrings ?
>
> There is no such thing as utf-8 wideStrings. There are either utf-8
> strings or widestrings. You can inspect both, but neither will display
> correctly for non-latin characters if created from Seaside.
>
>>>> The get your file to the web with the
>>>> correct encoding you read in the file convert it from utf-8 to
>>>> the internal squeak encoding. Then you should use WAKomEncoded39
>>>> that will do the conversion to utf-8 when the string is about
>>>> to leave the image.
>>>
>>> That is of course only true if you get your Strings into the  
>>> image via
>>> Seaside and not via FileStream. And of course WAKomEncoded39 does  
>>> not
>>> work in Squeak 3.9 if you have the special version of Kom  
>>> distributed
>>> with the SeasideInstaller.
>>
>> arghhhh :)
>
> Well it would be much simpler if Kom was maintained. It's not that
> we're asking for much. Just integrating changes to three methods.
> There are some people working on an updated Kom release and we are
> looking forward to it.
>
>>>
>>>> From input fields it is nearly the same except
>>>> the fact that WAKomEncoded39 doesn't do conversion for fields if
>>>> the request is multipart (that means mostly you uploaded something
>>>> like image along with the text fields).
>>>
>>> This is fixed in 2.8. The only place where WAKomEncoded39 does not
>>> conversion is for fileuploads because we have no way of telling what
>>> encoding the file has or if it's even a text file.
>>
>> ok
>> so I should wait and use 2.8.
>
> Only if you have multipart fields and use WAKomEncoded39
>
> Cheers
> Philippe
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

>
> > You need to make a decision wheter you want utf8 or widestrings/"new
> > squeak encoding" in your image.
> > utf8 strings -> WAKom
> > WideString -> WAKomEncoded39
> >
> > Make sure you convert the strings you read from disk to the machting
> > encoding. You can convert a string to widestring/"new squeak encoding"
> > by sending #convertFromEncoding: and can convert a widestring/"new
> > squeak encoding" to something else by sending #convertToEncoding:
> >
> > WideStrings have a history of slow performance and bugs (there are
> > still known, open bugs). There is no way of telling which methods and
> > primitives don't work. For utf-8 strings you can rely on pretty much
> > everything but concatenation being broken (including #size). As for
> > inspector support both of them don't shine. Your choice.
> >
> >> Now when the user type text in input fields I would like to have the
> >> same encoding.
> >>
> >> Then I was also curious about the use in the image of encoded strings
> >> (ie how to create them, convert....)
> >>
> >>
> >>>
> >>> Just "I have an encoding problem" is not really helpful at all
> >>> especially if asking for answers before supplying any information.
> >>
> >> sorry
> >>
> >>>
> >>>> My opinion
> >>>> is that you should convert every string you get from outside
> >>>> the image to the internal format squeak uses. That is a controlled
> >>>> setting and you are able to use size and other methods on those
> >>>> strings without worry.
> >>>
> >>> Unfortunately even if you have WideStrings you can not rely on that
> >>> all the methods for Strings work since some are broken for
> >>> WideStrings.
> >>
> >> How can I see that?
> >
> > Stuff like #match: throwing an exception if the argument is a
> > WideString.
> >
> >> It would be good to report (if this is not already done)
> >
> > Done.
> >
> >>> Additionally you won't be able to inspect your non-latin
> >>> Squeak WideStrings because they have no language tag and there is no
> >>> way of adding one.
> >>
> >> ahhh.
> >> What do you mean is that we cannot inspect utf-8 wideStrings ?
> >
> > There is no such thing as utf-8 wideStrings. There are either utf-8
> > strings or widestrings. You can inspect both, but neither will display
> > correctly for non-latin characters if created from Seaside.
> >
> >>>> The get your file to the web with the
> >>>> correct encoding you read in the file convert it from utf-8 to
> >>>> the internal squeak encoding. Then you should use WAKomEncoded39
> >>>> that will do the conversion to utf-8 when the string is about
> >>>> to leave the image.
> >>>
> >>> That is of course only true if you get your Strings into the
> >>> image via
> >>> Seaside and not via FileStream. And of course WAKomEncoded39 does
> >>> not
> >>> work in Squeak 3.9 if you have the special version of Kom
> >>> distributed
> >>> with the SeasideInstaller.
> >>
> >> arghhhh :)
> >
> > Well it would be much simpler if Kom was maintained. It's not that
> > we're asking for much. Just integrating changes to three methods.
> > There are some people working on an updated Kom release and we are
> > looking forward to it.
> >
> >>>
> >>>> From input fields it is nearly the same except
> >>>> the fact that WAKomEncoded39 doesn't do conversion for fields if
> >>>> the request is multipart (that means mostly you uploaded something
> >>>> like image along with the text fields).
> >>>
> >>> This is fixed in 2.8. The only place where WAKomEncoded39 does not
> >>> conversion is for fileuploads because we have no way of telling what
> >>> encoding the file has or if it's even a text file.
> >>
> >> ok
> >> so I should wait and use 2.8.
> >
> > Only if you have multipart fields and use WAKomEncoded39
> >
> > Cheers
> > Philippe
> > _______________________________________________
> > Seaside mailing list
> > [hidden email]
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> >
>
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

> writing stuff from it out to an utf8 page with seaside 2.8, and
> I still get funky characters instead of accents. I tried guessing what
> the actual output encoding is but no success. Normal characters seem
> to be ascii/latin1, accented characters use two bytes but are not
> correct utf8...
>
> >
> > > You need to make a decision wheter you want utf8 or widestrings/"new
> > > squeak encoding" in your image.
> > > utf8 strings -> WAKom
> > > WideString -> WAKomEncoded39
> > >
> > > Make sure you convert the strings you read from disk to the machting
> > > encoding. You can convert a string to widestring/"new squeak encoding"
> > > by sending #convertFromEncoding: and can convert a widestring/"new
> > > squeak encoding" to something else by sending #convertToEncoding:
> > >
> > > WideStrings have a history of slow performance and bugs (there are
> > > still known, open bugs). There is no way of telling which methods and
> > > primitives don't work. For utf-8 strings you can rely on pretty much
> > > everything but concatenation being broken (including #size). As for
> > > inspector support both of them don't shine. Your choice.
> > >
> > >> Now when the user type text in input fields I would like to have the
> > >> same encoding.
> > >>
> > >> Then I was also curious about the use in the image of encoded strings
> > >> (ie how to create them, convert....)
> > >>
> > >>
> > >>>
> > >>> Just "I have an encoding problem" is not really helpful at all
> > >>> especially if asking for answers before supplying any information.
> > >>
> > >> sorry
> > >>
> > >>>
> > >>>> My opinion
> > >>>> is that you should convert every string you get from outside
> > >>>> the image to the internal format squeak uses. That is a controlled
> > >>>> setting and you are able to use size and other methods on those
> > >>>> strings without worry.
> > >>>
> > >>> Unfortunately even if you have WideStrings you can not rely on that
> > >>> all the methods for Strings work since some are broken for
> > >>> WideStrings.
> > >>
> > >> How can I see that?
> > >
> > > Stuff like #match: throwing an exception if the argument is a
> > > WideString.
> > >
> > >> It would be good to report (if this is not already done)
> > >
> > > Done.
> > >
> > >>> Additionally you won't be able to inspect your non-latin
> > >>> Squeak WideStrings because they have no language tag and there is no
> > >>> way of adding one.
> > >>
> > >> ahhh.
> > >> What do you mean is that we cannot inspect utf-8 wideStrings ?
> > >
> > > There is no such thing as utf-8 wideStrings. There are either utf-8
> > > strings or widestrings. You can inspect both, but neither will display
> > > correctly for non-latin characters if created from Seaside.
> > >
> > >>>> The get your file to the web with the
> > >>>> correct encoding you read in the file convert it from utf-8 to
> > >>>> the internal squeak encoding. Then you should use WAKomEncoded39
> > >>>> that will do the conversion to utf-8 when the string is about
> > >>>> to leave the image.
> > >>>
> > >>> That is of course only true if you get your Strings into the
> > >>> image via
> > >>> Seaside and not via FileStream. And of course WAKomEncoded39 does
> > >>> not
> > >>> work in Squeak 3.9 if you have the special version of Kom
> > >>> distributed
> > >>> with the SeasideInstaller.
> > >>
> > >> arghhhh :)
> > >
> > > Well it would be much simpler if Kom was maintained. It's not that
> > > we're asking for much. Just integrating changes to three methods.
> > > There are some people working on an updated Kom release and we are
> > > looking forward to it.
> > >
> > >>>
> > >>>> From input fields it is nearly the same except
> > >>>> the fact that WAKomEncoded39 doesn't do conversion for fields if
> > >>>> the request is multipart (that means mostly you uploaded something
> > >>>> like image along with the text fields).
> > >>>
> > >>> This is fixed in 2.8. The only place where WAKomEncoded39 does not
> > >>> conversion is for fileuploads because we have no way of telling what
> > >>> encoding the file has or if it's even a text file.
> > >>
> > >> ok
> > >> so I should wait and use 2.8.
> > >
> > > Only if you have multipart fields and use WAKomEncoded39
> > >
> > > Cheers
> > > Philippe
> > > _______________________________________________
> > > Seaside mailing list
> > > [hidden email]
> > > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> > >
> >
> > _______________________________________________
> > Seaside mailing list
> > [hidden email]
> > http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
> >
>
>
> --
> Damien Pollet
> type less, do more [ | ] http://typo.cdlm.fasmz.org
> _______________________________________________
> Seaside mailing list
> [hidden email]
> http://lists.squeakfoundation.org/cgi-bin/mailman/listinfo/seaside
>

> On 12/09/2007, Philippe Marschall <[hidden email]> wrote:
> > > I think I have a similar problem in Citezen. Reading/parsing and utf8
> > > file,
> >
> > You you use WAKom or WAKomEncoded*. Do you set the converter for the
> > file stream? Do you use the Kom from SqueakMap or the one from the
> > SeasideInstaller?
>
> Oops, sorry for not seeing your answer earlier...
>
> I use WAKom, don't do anything particular to the file stream (just
> using FileDirectory>>readOnlyFileNamed:) and I'm using Kom from
> package universe (which in turn gets
> http://squeak.krampe.se/KomHttpServer-gk.14.mcz)