[squeak-dev] how to create an UTF-8 character

> Yoshiki Ohshima wrote:
>>   For that kind of web applications and servers that deals with stuff
>> outside of Squeak, it doesn't serve a good purpose, because editting,
>> displaying etc. are out of scope.  Needless to say, the original idea
>> was to make Squeak to be the dynamic, interactive, multilingualized,
>> environment so there is mismatch.  Web applications etc. historically
>> comes after the goal.
>
> Which wouldn't be a problem if the code was able to handle the data
> properly. Unfortunately, the effects of an "invalid" leading char are
> very, very strange (everything from crashing the scanner to raising
> weird errors in comparisons, character access etc). As it stands, an
> application that uses non-Latin characters off the web is best off by
> keeping everything in UTF-8.
>
> BTW, one way to deal with this properly is by providing a leading char
> upon input conversion (i.e., utf8ToSqueak would then insert the proper
> leading chars for each character). As a matter of fact, I think this is
> what Unicode class>>value: should do (instead of substituting the
> environmental leading char).
>
>>   If you need to retain these extra information, sending the strings
>> without going through UTF-8 conversion makes more sense.
>
> Or provide it via additional attributes. I still think that language
> information would best be modeled by a text attribute - in which case we
> have a plain Unicode implementation for strings as well as the ability
> to provide the disambiguation in text where required.
>
> Cheers,
>    - Andreas
>
>

> Yoshiki Ohshima wrote:
>>   For that kind of web applications and servers that deals with stuff
>> outside of Squeak, it doesn't serve a good purpose, because editting,
>> displaying etc. are out of scope.  Needless to say, the original idea
>> was to make Squeak to be the dynamic, interactive, multilingualized,
>> environment so there is mismatch.  Web applications etc. historically
>> comes after the goal.
>
> Which wouldn't be a problem if the code was able to handle the data
> properly. Unfortunately, the effects of an "invalid" leading char are
> very, very strange (everything from crashing the scanner to raising
> weird errors in comparisons, character access etc). As it stands, an
> application that uses non-Latin characters off the web is best off by
> keeping everything in UTF-8.
>
> BTW, one way to deal with this properly is by providing a leading char
> upon input conversion (i.e., utf8ToSqueak would then insert the proper
> leading chars for each character). As a matter of fact, I think this is
> what Unicode class>>value: should do (instead of substituting the
> environmental leading char).
>
>>   If you need to retain these extra information, sending the strings
>> without going through UTF-8 conversion makes more sense.
>
> Or provide it via additional attributes. I still think that language
> information would best be modeled by a text attribute - in which case we
> have a plain Unicode implementation for strings as well as the ability
> to provide the disambiguation in text where required.

>>>> Am I the only one using the generic en/decoding functionality in
>>> Squeak in the form of #convertTo/FromEncoding?
>>>
>>> Convert from "Squeak" to UTF-8
>>> aString convertToEncoding: 'utf-8'
>>
>>
>> do I understand correctly that such a aString is a sequence of  
>> unicode
>> codepoints?
>>>
> At first the utf-8 is a sequence of bytes. These bytes are a space
> optimzed encoding of a code point (utf-8). If you decode those bytes
> you get your code point (unicode). From a sequence of code points
> you can derive a character. In most cases (for us westerners) it will
> be a single code point AFAIK.

> >>>> Am I the only one using the generic en/decoding functionality in
> >>> Squeak in the form of #convertTo/FromEncoding?
> >>>
> >>> Convert from "Squeak" to UTF-8
> >>> aString convertToEncoding: 'utf-8'
> >>
> >>
> >> do I understand correctly that such a aString is a sequence of  
> >> unicode
> >> codepoints?
> >>>
> > At first the utf-8 is a sequence of bytes. These bytes are a space
> > optimzed encoding of a code point (utf-8). If you decode those bytes
> > you get your code point (unicode). From a sequence of code points
> > you can derive a character. In most cases (for us westerners) it will
> > be a single code point AFAIK.
>
> I'm trying to really understand in Squeak. :)
> What we call character is what then?
> Is it a codepoint? or the looked up glyph in a font table?
>

> On Mon, 2008-09-29 at 18:53 +0200, stephane ducasse wrote:
>>>>>> Am I the only one using the generic en/decoding functionality in
>>>>> Squeak in the form of #convertTo/FromEncoding?
>>>>>
>>>>> Convert from "Squeak" to UTF-8
>>>>> aString convertToEncoding: 'utf-8'
>>>>
>>>>
>>>> do I understand correctly that such a aString is a sequence of
>>>> unicode
>>>> codepoints?
>>>>>
>>> At first the utf-8 is a sequence of bytes. These bytes are a space
>>> optimzed encoding of a code point (utf-8). If you decode those bytes
>>> you get your code point (unicode). From a sequence of code points
>>> you can derive a character. In most cases (for us westerners) it  
>>> will
>>> be a single code point AFAIK.
>>
>> I'm trying to really understand in Squeak. :)
>> What we call character is what then?
>> Is it a codepoint? or the looked up glyph in a font table?
>>
> I don't know. I've never dealt with how squeak does those things