Squeak on UTF-8 system

Hi, I can't run squeak with correct Czech language keyboard input and
font output on Fedora Core 4 (UTF-8 enviroment - LANG=cs_CZ.UTF-8).

-textenc and -encoding parameters didn't works (squeak 3.8).

Is there any trick or how-to?

Thanks, pf

Petr Fischer wrote:
Try to workaround the problem. There is a Czech keyboard package for
Squeak at SqueakMap written
by Vladimir Janousek.

Elod

Petr Fischer wrote: This issue is a real problem... the national input seems not work on
*any* full UTF-8 linux distribution (Ubuntu), even if i try all locale
environment and squeak-vm parameters combinations. And AFAIK nobody was
able to solve this so far...

Elod Kironsky wrote: this doesn't work on 3.8+ and we really need the utf-8 environment...
(there may be a possibility to rewrite the keyboard-package for higher
versions, but i'm still kind of a squeak-beginner)

Viktor Svub wrote:
I am sure, it can be rewritten with a little effort. I will ask
Vladimir, he is on this mailing list, but I am not sure he is reading.
Another sollution is to use the developer version on www.comtalk.net.
The direct link is ftp://comtalk.net/Squeak-dev.zip It is natively
prepared for czech language I think. As an example, please see the
screenshots section at www.squeak.org. The first 2 images are from this
developer version as Pavel Krivanek is a great Seaside fan and an active
programmer in Seaside.

Elod

Elod Kironsky wrote: comtalk.net installed, and it works perfectly; the problem is't the
localization, but the keyboard input, cause it's written for win32 and
iso-8859-2 linux, and the utf-8 linux distros handle international
keyboards differently (i'm a linux-newbie, so i can't tell how)...

Hello guys,
I have been dealing with a similar issue for a few days already. Though,
my problem lies somewhere inside WAKom/WAKomEncoded or the underlying
widechar conversion classes. I have two half-working scenarios.
1) WAKom: I enter unicode text inside Squeak. When it is displayed in my
web app, it is in wrong encoding. However, text which I feed to Squeak
image through web browser is saved inside as UTF-8 and then displayed as
I want on the web page. When I print the text inside of my image, it has
some characters substituted with combinations of other characters.
2) WAKomEncoded: Unfortunately it means, I can have accented text
entered into my image only through web interface. So I tried
WAKomEncoded, which successfully recodes from Squeak to UTF-8 and
renders text properly in browser. BUT text entered in browser and
displayed back is not recoded properly. The UTF-8 conversion substitutes
the letters with accent with other combination of characters then it should.
This problem arises on system with any encoding.

Cheers,

Petr


Petr Fischer wrote:

Am Thursday, 16. March 2006 12:33 schrieb Viktor Svub:
About 1 and a half year ago I submitted a patch for the unix keyboard
input handler of the VM to make it utf8 ready. Unfortunately it never
got  included. Maybe it's time to try again.

Martin

Martin Kuball napsal(a): maybe you should just post the fix here, so anyone can build an own
utf-8 VM, without waiting for [a new release / a few years] (cross out
which does not apply)  ;)

> About 1 and a half year ago I submitted a patch for the unix keyboard
> input handler of the VM to make it utf8 ready. Unfortunately it never
> got  included. Maybe it's time to try again.
>
> Martin
>

Share it please. Thanks! pf

On Mar 16, 2006, at 10:23 PM, Petr wrote:

does your squeak uses UTF-8? As far as I know default system does not  
use this encoding and I (using Korean and UTF-8) have to modified  
squeak so that it use UTF-8 encoding and after this I can get correct  
display in both browser and web browser(seaside application). Hope  
this can help you.

PS)
I think Squeak should use UTF-8 (or any equivalent unicode based  
encoding scheme, I think UTF-8 is best choice) exclusively.

  I don't have such environments so that I cannot test things by
myself, but the rule of thumb is to disable the conversion built in
the VM.  I believe that the VM (do you mean that the VM is 3.8a1 or
such?) has "noconv" or such for the -textenc and -encoding.

  The encoding conversion in the VM is basically written before the
m17n stuff, and doesn't work well if combined with it.

  Hope this helps in some way...

-- Yoshiki

At Thu, 16 Mar 2006 10:34:29 +0100,
Viktor Svub wrote:

  Hello,

> does your squeak uses UTF-8? As far as I know default system does not  
> use this encoding and I (using Korean and UTF-8) have to modified  
> squeak so that it use UTF-8 encoding and after this I can get correct  
> display in both browser and web browser(seaside application). Hope  
> this can help you.

  I think I asked this once before, and I apologize if I have missed
the response from you, but can you tell me a bit about the
modification you made?

> PS)
> I think Squeak should use UTF-8 (or any equivalent unicode based  
> encoding scheme, I think UTF-8 is best choice) exclusively.

  The internal encoding of current Squeak is Unicode based, and it can
communicate with the external world in many variety of encodings,
including UTF-8.

  Using exclusively UTF-8 (for the external encoding) will be too
limiting in many applications on many platforms.  It is a good idea to
use a single internal representation, and Unicode is a compromization
today for this purpose (and Squeak uses it, too.)  But there is no
good reason to limit the external encoding.

-- Yoshiki

Yoshiki Ohshima wrote: thank you very much ^_^ ... "squeak -encoding noconv <image>" is the way
it goes - the national input is OK now (the VM thorows a few
"iconv_open: Invalid argument" messages on startup, but this can be
ignored, i think)

Do I have use modified VM or some special UTF-8 enabled image?

Petr



Sungjin Chun wrote:

Am Thursday, 16. March 2006 21:10 schrieb Petr Fischer:
> > About 1 and a half year ago I submitted a patch for the unix
> > keyboard input handler of the VM to make it utf8 ready.
> > Unfortunately it never got  included. Maybe it's time to try
> > again.
> >
> > Martin
>
> Share it please. Thanks! pf

I don't have much time this weekend. But I will try to dig up the
stuff and post it here.

Martin

For VM, maybe, I think there should be an option for specifying I/O  
encoding. For my case, I did modified VM code (actually my system,  
Mac OS X is configured to use UTF-8 and VM code is modified so that  
it does no conversion).

For Image, yes you have to create/modify some classes and code so  
that it use UTF-8 as default encoding instead of default iso-8859-1.  
Again, for my case, I did modified image so that it does perform I/O  
with UTF-8.

On Mar 18, 2006, at 3:07 AM, Petr wrote:

Am Friday, 17. March 2006 19:51 schrieb Martin Kuball: I'm not sure if the patches are still needed. Some previous mails
suggested that it's possible to make it work by specifing the right
parameters on the comandline. Well, it does not work for me. So find
attached to this mail the patches I made over a year ago.
Unfortunately I wasn't able to test them with a recent VM. I tried
with the svn version but it failed to compile because of some missing
methods (like fetchLong32ofObject).

Martin

On Mar 17, 2006, at 4:41 PM, Yoshiki Ohshima wrote:
>
>   I think I asked this once before, and I apologize if I have missed
> the response from you, but can you tell me a bit about the
> modification you made?

My modification is very simple - hard part is finding/importing good  
quality korean font - I only did create UnicodeEnvironment (subclass  
of LanguageEnvironment) because all other subclasses of  
LanguageEnvironment uses different charset from Unicode. And I  
configured squeak to use UnicodeEnvironment instead of default  
Latin1Environment.

>
>   Using exclusively UTF-8 (for the external encoding) will be too
> limiting in many applications on many platforms.  It is a good idea to
> use a single internal representation, and Unicode is a compromization
> today for this purpose (and Squeak uses it, too.)  But there is no
> good reason to limit the external encoding.

I totally agree on this.

PS)
I've encountered new problem. If file name is UTF-8 encoded Korean, I  
can only see ????.txt instead of 한글.txt in squeak. (FileDirectory  
class) I think this is related to VM part but not sure.

Hope this can help you.

  Hello,

> My modification is very simple - hard part is finding/importing good  
> quality korean font - I only did create UnicodeEnvironment (subclass  
> of LanguageEnvironment) because all other subclasses of  
> LanguageEnvironment uses different charset from Unicode. And I  
> configured squeak to use UnicodeEnvironment instead of default  
> Latin1Environment.

  I must have forgotten the changes I made while ago.  Please find
the following and attachment.  If you can share your work with us,
that would be really great, by the way!

-- Yoshiki
-------------------------------
To: [hidden email]
Cc: Takashi Yamamiya <[hidden email]>, [hidden email]
Subject: Re: How to build korean image of Squeak
From: Yoshiki Ohshima <[hidden email]>
Date: Tue, 01 Nov 2005 03:52:53 -0800
Message-ID: <uvezcwpnu.wl%[hidden email]>
In-Reply-To: <[hidden email]>
User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.6
 (Marutamachi) APEL/10.6 Emacs/21.1 (i386-mingw-nt5.1.2600) MULE/5.0
 (SAKAKI) Meadow/2.00 (KIKYOU)
MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka")
X-UIDL: 1130846129.25675.mailbox

  Hello, Seungbum,

  I did some attempt to support Korean in the latest Squeakland
image.  It is doing something seemingly right, but as always, I cannot
tell that for sure.  Can you try it and how it is doing?

  To try it:
  * Download an image from:
    http://www.squeakland.org/installers/SqueakPluginImage-dev.zip
    and unpack it.
  * Download the 'font file' from:
    http://www.is.titech.ac.jp/~ohshima/tmp/uKoreanFont.out
    and place the file in the same folder as
    SqueakPluginImage-dev.image.
  * Launch the image and choose 'update code from server' from the
    help menu in the World menu.
  * (From the file list), load the attached into the image.  It may
    take some time, but should work.
  * Switch to Korean from World menu -> 'help...' -> 'set language...'
    menu.  This may also take some time, but should work.

  Now, you should be able to input Korean characters on Windows.  I
don't know if it works on Linux in some settings, but I would imagine
that you will need a custom VM, as the Japanese support does.

  Let me know how it goes.

  Thank you,

-- Yoshiki

The way I created the uKoreanFont.out is:

  * Download efont-unicode-bdf-0.4.2, shinonome-0.9.11p1, and
    intlfonts-1.2.1 and copy the necessary files to the same folder as
    SqueakPluginImage-dev.image
  * Evaluate:
    StrikeFontSet createExternalFontFileForUnicodeKorean: 'uKoreanFont.out'.