International text input on X11

We-eell, I’ve built a VM with the ImmX11 plugin. And added your suggested hacks to the two methods. And started my vm with -vm-sound-alsa -vm-display-X11 -compositioninput etc and … nothing much happens. It starts, it runs it behaves just as normal. The plugin is loaded and the CompositionWindowManager var is set. No extra pop-up windows. What am I missing?


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Hipatitis (n): Terminal coolness

On Wed, Jun 1, 2016 at 6:53 PM, tim Rowledge <[hidden email]> wrote:
> We-eell, I’ve built a VM with the ImmX11 plugin. And added your suggested hacks to the two methods. And started my vm with -vm-sound-alsa -vm-display-X11 -compositioninput etc and … nothing much happens. It starts, it runs it behaves just as normal. The plugin is loaded and the CompositionWindowManager var is set. No extra pop-up windows. What am I missing?

If the settings described below is not properly set, the behavior you
mentioned is what is expected.  (Even the settings below are there,
you can still activate and deactivate the XIM client so it still can
happen.)

- The LC_ALL environment variable (and LANGUAGE, if I'm not mistaken)
has to be set.  The variable I believe has to be set for the X server
so it has to be done before launching the X server.
- an XIM client, such as ibus, scim, and Matchbox if I'm not mistaken,
has to be running.
- and the XIM client is configured to handle the language set to
LC_ALL. (On the Raspbian environment, you see an icon in the menu bar,
where you can see what languages it handles.)

With those, start a Squeak session.  You can activate/deactivate the
XIM client by its own key combination.  When you try to type into a
text field in Squeak, the overlay would be displayed around the text
field.

If you have a spare Raspian installation, you can do the following:
(But I am writing this from my memory, so it may not be exactly
correct. Sorry).

- From raspi-config, go to the "Internationalisation Options" and add
ja_JP.UTF8.

- From terminal install a few things:

sudo apt-get install ibus
sudo apt-get install ibus-anthy

This will add a new menu item to the Raspian menu.  IIRC, you need to
"add language" and select Japanese from the ibus configuration menu.

- Add the following lines to .bashrc
  export LANG=ja_jp.UTF8
  export LC_ALL=$LANG
  export LANGUAGE=$LANG

close X, do something like ". .bashrc" and restart X.  You should see
that Raspian menus are shown in Japanese.  (You can still start
raspi-config later and remove Japanese, so don't panic)

- Start Scratch, activate ibus from the ibus icon in the Raspian menu
(choose Japanese/日本語), click on somewhere such as the sprite name
field.
- type in some strings with enough vowels in it.  Hit enter and
composed characters should go into the text field.






--
-- Yoshiki

I tried to set my Pi as specified and it didn’t go all that well. I think we may need *much* more thorough instructions for us non-non-en speakers …


> On 01-06-2016, at 7:13 PM, Yoshiki Ohshima <[hidden email]> wrote:
> {snip}
> If you have a spare Raspian installation, you can do the following:
> (But I am writing this from my memory, so it may not be exactly
> correct. Sorry).
>
> - From raspi-config, go to the "Internationalisation Options" and add
> ja_JP.UTF8.

OK, no problem
>
> - From terminal install a few things:
>
> sudo apt-get install ibus
> sudo apt-get install ibus-anthy

Had to do that twice since the first time complained about problems configuring the ‘anthy’ tool

>
> This will add a new menu item to the Raspian menu.  IIRC, you need to
> "add language" and select Japanese from the ibus configuration menu.

The only new item I could find was the ibus config under the preferences heading. This tool did not seem to have any option to set Japanese at first; it was only after rebooting that it became available.

>
> - Add the following lines to .bashrc
>  export LANG=ja_jp.UTF8
>  export LC_ALL=$LANG
>  export LANGUAGE=$LANG
>
> close X, do something like ". .bashrc" and restart X.  You should see
> that Raspian menus are shown in Japanese.  (You can still start
> raspi-config later and remove Japanese, so don't panic)

After rebooting *some* of the menu text was rendered in nasty unicode-block form rather than Japanese or anything that might look like it to my untrained eyes.

>
> - Start Scratch, activate ibus from the ibus icon in the Raspian menu
> (choose Japanese/日本語), click on somewhere such as the sprite name
> field.
> - type in some strings with enough vowels in it.  Hit enter and
> composed characters should go into the text field.

Nah, no luck there. Nothing happened that could pass for an extra window opening up.
A `ps aux | grep ibus` shows the ibus daemon appears to be running.

I’m out of ideas on this. Help!

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DSO: Do Something or Other

On Thu, Jun 2, 2016 at 12:50 PM, tim Rowledge <[hidden email]> wrote:
Ah... I'm sorry to forgot to mention:

apt-get install ttf-kochi-gothic xfonts-intl-japanese
xfonts-intl-japanese-big xfonts-kaname

to add extra fonts.

>> - Start Scratch, activate ibus from the ibus icon in the Raspian menu
>> (choose Japanese/日本語), click on somewhere such as the sprite name
>> field.
>> - type in some strings with enough vowels in it.  Hit enter and
>> composed characters should go into the text field.
>
> Nah, no luck there. Nothing happened that could pass for an extra window opening up.
> A `ps aux | grep ibus` shows the ibus daemon appears to be running.

Ok.  One test is to launch the web browser and put the keyboard focus
on the search field on google or something.  Then click on the ibus
menu in the Raspbian title bar.  If you have a choice to input
Japanese there, ibus is ready to handle it.

Sorry again that I don't have the pi handy right now.  If it does not
resolve tomorrow, I'll make some screenshots to explain it.

> On 02-06-2016, at 1:42 PM, Yoshiki Ohshima <[hidden email]> wrote:
>>
> Ok.  One test is to launch the web browser and put the keyboard focus
> on the search field on google or something.  Then click on the ibus
> menu in the Raspbian title bar.  If you have a choice to input
> Japanese there, ibus is ready to handle it.

No happiness here. At least loading all those fonts improved the rendering!

>
> Sorry again that I don't have the pi handy right now.  If it does not
> resolve tomorrow, I'll make some screenshots to explain it.
>

Please! This is very strange stuff to me; much guidance required.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful Latin Phrases:- Canis meus id comedit = My dog ate it.

Let us try it with environment variables LC_ALL and LANG to be en_GB.UTF8 first.

export LC_ALL=en_GB.UTF8
export LANG=$LC_ALL

(and reboot.)  You should see menus in English (again)

With ibus installed, you should see something like
'raspi-menu' (http://tinlizzie.org/~ohshima/ibus/raspi-menu.png):
there should be ibus Preferences item there.  Pick that
one.

Then in the panel you select "Input Method" tab.  "Add" and select
Japanese, and choose 'anthy'.  as in 'ibus-config'
(http://tinlizzie.org/~ohshima/ibus/ibus-config.png).

There, I believe you get the menu in the bar that looks like
'ibus-selection'
(http://tinlizzie.org/~ohshima/ibus/ibus-selection.png).

Choose Japanese there and typing into terminal or browser should go
through composition.

Scratch, when launched with LC_ALL being en_GB.UTF8 should come up in
English here, but we can try switching to Japanese once the above
works.

(Okay...  I sent a similar email earlier with attached pictures but it
went to the moderation because the attachment is "too big".  The thing
is that it feels like 100k is a little bit too small...  How do others
deal with the limit?)

On Fri, Jun 3, 2016 at 10:07 AM, tim Rowledge <[hidden email]> wrote:


--
-- Yoshiki

So, some good news at last.

> On 03-06-2016, at 11:36 AM, Yoshiki Ohshima <[hidden email]> wrote:
>
> Let us try it with environment variables LC_ALL and LANG to be en_GB.UTF8 first.
>
> export LC_ALL=en_GB.UTF8
> export LANG=$LC_ALL
>
> (and reboot.)  You should see menus in English (again)

I actually had to reset the locale in raspi-config to get it clean. Do we have the same OS version? I have `uname -a` returning 4.4.9-v7+ #884 SMP Fri May 6 etc


>
> With ibus installed, you should see something like
> 'raspi-menu' (http://tinlizzie.org/~ohshima/ibus/raspi-menu.png):
> there should be ibus Preferences item there.  Pick that
> one.
>

Almost; I had to choose a plain ‘Japanese’ entry, add it and then choose an input method for it before I could see anything to do with ‘anthy’. Once that was ok I could use the menubar icon to choose input method and at last I get a pop-up composition window. Hooray!

Of course, it could be inserting Elvish or Klingon (please tell me somebody has made a Klingon input method…) for all I know.


If we want to make this accessible we’re going to need a much more precise description of what to do, or better yet a script.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful Latin Phrases:- Furnulum pani nolo = I don't want a toaster.

On Mon, Jun 6, 2016 at 10:35 AM, tim Rowledge <[hidden email]> wrote:

>
> Almost; I had to choose a plain ‘Japanese’ entry, add it and then choose an input method for it before I could see anything to do with ‘anthy’. Once that was ok I could use the menubar icon to choose input method and at last I get a pop-up composition window. Hooray!
>
> Of course, it could be inserting Elvish or Klingon (please tell me somebody has made a Klingon input method…) for all I know.

Great!

> If we want to make this accessible we’re going to need a much more precise description of what to do, or better yet a script.

Sure.  My instruction was mostly from memory and might confuse you
more than necessary.  (Sorry) There are books and web pages that
people (including myself) look at but they are not still a single
script that anybody can execute to have the environment going.

Now you seem to have the composition window show up, you can try it in
Scratch that is running under Japanese locale, and with the changes to
the image, the composition window should show up where the cursor is,
basically.  (We can tweak the exact location of it; perhaps it might
better go one line worth of height below where the cursor is.)

Again I have not looked into all possible places in the Scratch
related morphs where it makes sense to have it, but supporting those
should now be just a matter of adding a line or two to those morphs'
implementations.

Thank you!


--
-- Yoshiki

> On 06-06-2016, at 10:35 AM, tim Rowledge <[hidden email]> wrote:

> method and at last I get a pop-up composition window. Hooray!
>

… except it doesn’t actually seem to input anything to the squeak text morph. Sigh.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: IAM: Increase Amperage Above Maximum

On Mon, Jun 6, 2016 at 10:47 AM, tim Rowledge <[hidden email]> wrote:
>
>> On 06-06-2016, at 10:35 AM, tim Rowledge <[hidden email]> wrote:
>
>> method and at last I get a pop-up composition window. Hooray!
>>
>
> … except it doesn’t actually seem to input anything to the squeak text morph. Sigh.

 I think Squeak Text Morphs are broken for sometime, beside the fact
that Squeak does have to load its own fonts.  Scratch does the
rendering with the OS feature (via UnicodePlugin) so that somehow
works.


--
-- Yoshiki

So far as I can tell even the Scratch code is simply not getting any input from the composition window; well except for very occasionally and not in any way that I can make a pattern out of yet. When it does the JIS-to-unicode convertor complains about the character value not being in its table. It’s very odd!

I think all that I can do for now is look at making the the ST code you added a bit cleaner and wait for a proper config sequence/script.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- One clown short of a circus.

> On 06-06-2016, at 11:27 AM, tim Rowledge <[hidden email]> wrote:
>
> So far as I can tell even the Scratch code is simply not getting any input from the composition window; well except for very occasionally and not in any way that I can make a pattern out of yet. When it does the JIS-to-unicode convertor complains about the character value not being in its table. It’s very odd!

Sigh. Now it’s working ‘properly’ to the best of my knowledge. I can type stuff, it looks like probably Japanese script and goes into the sprite name field. Even adding text to the sprite seems to do the right thing, project notes.... OK, after hitting everything I can think of that might involve text entry in Scratch it looks good.

So, how on earth did I get here… and can I ever get back to plain old en_GB?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: UDF: Use Disk for Frisbee

On Mon, Jun 6, 2016 at 11:34 AM, tim Rowledge <[hidden email]> wrote:
>
>> On 06-06-2016, at 11:27 AM, tim Rowledge <[hidden email]> wrote:
>>
>> So far as I can tell even the Scratch code is simply not getting any input from the composition window; well except for very occasionally and not in any way that I can make a pattern out of yet. When it does the JIS-to-unicode convertor complains about the character value not being in its table. It’s very odd!
>
> Sigh.

I'd say "Yay!".

Scratch specific Morphs such as StringFieldMorphs works okay, right?
That convertor complains sounds like it is about the Squeak Text
Morphs, that used to have m17n support (as opposed to i18n) but over
the course of years it got broken.

> Now it’s working ‘properly’ to the best of my knowledge. I can type stuff, it looks like probably Japanese script and goes into the sprite name field. Even adding text to the sprite seems to do the right thing, project notes.... OK, after hitting everything I can think of that might involve text entry in Scratch it looks good.

Great!

> So, how on earth did I get here… and can I ever get back to plain old en_GB?

Do you have to?

--
-- Yoshiki

> On 06-06-2016, at 11:38 AM, Yoshiki Ohshima <[hidden email]> wrote:
>
>> So, how on earth did I get here… and can I ever get back to plain old en_GB?
>
> Do you have to?

Well, yes. When I was a mid-teen (around the time that electricity was invented) I could make a faltering conversation in English, Welsh, French, German, Russian or Latin. These days I can mange English ok and can just about remember how to pronounce Welsh. You’ll note that coping with Japanese is not in either list …

It seems to me that we’ve fixed the VM ok, and we know how to start it correctly to use the new option. I’m not at all sure whether it is sensible to use the -compositioninput flag for anyone not wanting the popup window; should the scratch startup script do some working out to decide this? Should it be an option people can set? Automatic if the LANG is set to Japanese (and other languages)? It seems there is a fair bit of package loading required for anyone to have the required fonts and the ibus tools, so perhaps there should be a script/package to handle that and replace/modify the /bin/scratch script. I’m not at all sure about the script changes you posted May 25th, not because I have some reason to dislike them but I just don’t understand.

I think my process was -
- `sudo apt-get install ibus ibus-anthyttf-kochi-gothic xfonts-intl-japanese xfonts-intl-japanese-big xfonts-kaname`
- use raspi-config to set locale to ja_JP.UTF
- edit /home/pi/.bashrc to add
"export LANG=ja_jp.UTF8
 export LC_ALL=$LANG
 export LANGUAGE=$LANG”
to the end.
 -reboot.
- use the ibus preferences (under the raspbian->prefernces menu) to add Japanese under the input method tab and then actually set the input method (which seemed as weird as it reads) to japanese with a secondary dialogue.
- start Scratch, set Japanese as language
- use the ibus icon at the right end of the raspbian menubar to set ‘Anthy’ (NOT Japanese) as the input

And I *think* that was it.

We have outline image code to make use of it, at least for Scratch (though I see that the composition window appears for me in ordinary browsers now) that needs tidying up. I can do that. Somebody what knows what they’re doing needs to produce a nice little shell script I can stick next to the timidity install script in /usr/share/scratch that will load everything and configure it correctly. I can’t imagine the above complications working too wwell in a classroom.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: IXM: Initiate X-rated error Messages

On Mon, Jun 6, 2016 at 1:19 PM, tim Rowledge <[hidden email]> wrote:
>
>> On 06-06-2016, at 11:38 AM, Yoshiki Ohshima <[hidden email]> wrote:
>>
>>> So, how on earth did I get here… and can I ever get back to plain old en_GB?
>>
>> Do you have to?
>
> Well, yes. When I was a mid-teen (around the time that electricity was invented) I could make a faltering conversation in English, Welsh, French, German, Russian or Latin. These days I can mange English ok and can just about remember how to pronounce Welsh. You’ll note that coping with Japanese is not in either list …

Oh, sorry.  I was facetious but not helpful.
>
> It seems to me that we’ve fixed the VM ok, and we know how to start it correctly to use the new option. I’m not at all sure whether it is sensible to use the -compositioninput flag for anyone not wanting the popup window; should the scratch startup script do some working out to decide this?

Yes.  The patch to the scratch script I sent to checks the environment
variable XIM, and passes the -compositioninput flag only when there is
a spec for it.  (connecting to XIM may still fail when a server like
ibus is not running, but I think Squeak VM is smart enough to handle
it.

> Should it be an option people can set? Automatic if the LANG is set to Japanese (and other languages)? It seems there is a fair bit of package loading required for anyone to have the required fonts and the ibus tools, so perhaps there should be a script/package to handle that and replace/modify the /bin/scratch script. I’m not at all sure about the script changes you posted May 25th, not because I have some reason to dislike them but I just don’t understand.

Hmm, ok.  The change to the shell script may look more complicated,
but that is because the logic in the (modified) /usr/bin/squeak was
included there and we can bypass /usr/bin/squeak altogether.
Otherwise it just checks the XIM environment variable.

There can be different state such as have ibus installed but not
running, have japanese fonts installed but LANG is not using them,
etc.  As long as LANG and LC_ALL are en_GB.UTF8, there is no harm for
English users I believe).

> We have outline image code to make use of it, at least for Scratch (though I see that the composition window appears for me in ordinary browsers now) that needs tidying up. I can do that. Somebody what knows what they’re doing needs to produce a nice little shell script I can stick next to the timidity install script in /usr/share/scratch that will load everything and configure it correctly. I can’t imagine the above complications working too wwell in a classroom.

Installing the ibus package would do it but for those who don't do
that, it should be fine.  (And you can remove it by apt-get uninstall
ibus).

--
-- Yoshiki

That’s ok - I was being a little sarcastic too. It amazes me that humans can handle such a range of language features. I mean the the only two things I can think of the Japanese and English have in common is that humans use them and both are time-forward grammar. Welsh is fun because it has a few extra items in the alphabet but nothing scary looking - and it’s perfectly phonetic in both directions. Russian was interesting to try out because of the different alphabet, though it’s still nothing like the change to ideographic languages. Scratch does a decent job of handling the fonts via CairoPango & your UTF work, and even deals with right-to-left text ok.

Please, don’t anyone ask me for boustrophedon support.

>>
>> It seems to me that we’ve fixed the VM ok, and we know how to start it correctly to use the new option. I’m not at all sure whether it is sensible to use the -compositioninput flag for anyone not wanting the popup window; should the scratch startup script do some working out to decide this?
>
> Yes.  The patch to the scratch script I sent to checks the environment
> variable XIM, and passes the -compositioninput flag only when there is
> a spec for it.

That would be the -
if echo $XMODIFIERS | grep '@im=' > /dev/null; then
+     VMOPTIONS="$VMOPTIONS -compositioninput"
+ fi
+
changes?

Could you send me the plain file? I just don’t get the unix diff format and it’s much easier for me to compare manually.

I’ve just noticed that attempting numeric input appears not to work because instead of a 0 character I get a 65296 (apparently unicode ‘full width 0’) char. www.codetable.net tells me there are fullwidth digits and halfwidth digits and I bet buried in there somewhere are half-wits. Thankfully it seems {big number} asCharacter digitValue does a good enough job of getting the number value and then we just have to more explicitly check for - and . If you take a look at StringFieldMorph>insertCharacter: you’ll see where the problem lies easily enough. To my amazement I can actually enter numbers and the arithmetic gets sorted out ok. Good job somebody!



tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Dukedom: aristocratic birth control

On Mon, Jun 6, 2016 at 5:36 PM, tim Rowledge <[hidden email]> wrote:
Yes.  (And for that matter, the part that figures out the
LD_LIBRARY_PATH may be simplified as there is only one architecture
involved on RasPi).

> Could you send me the plain file? I just don’t get the unix diff format and it’s much easier for me to compare manually.

I don't have the access to the hardware right away...  You can:

cp /usr/bin/scratch ~/
cd ~
patch -p0 < scratch.diff

and ~/scratch should become the file it is intended to be.

> I’ve just noticed that attempting numeric input appears not to work because instead of a 0 character I get a 65296 (apparently unicode ‘full width 0’) char. www.codetable.net tells me there are fullwidth digits and halfwidth digits and I bet buried in there somewhere are half-wits. Thankfully it seems {big number} asCharacter digitValue does a good enough job of getting the number value and then we just have to more explicitly check for - and . If you take a look at StringFieldMorph>insertCharacter: you’ll see where the problem lies easily enough. To my amazement I can actually enter numbers and the arithmetic gets sorted out ok. Good job somebody!

That was my old attempt to keep the DecimalProperty of Unicode table...


--
-- Yoshiki

Here is a simpler version of the scratch shell script that is not in
the diff form:

On Mon, Jun 6, 2016 at 6:22 PM, Yoshiki Ohshima <[hidden email]> wrote:

--
-- Yoshiki

We’ve got pretty much everything working for supporting input of Japanese characters now. The image is updated, the vm has the ability to recognise the relevant flag, the scripts to fire up scratch work ok and so far as I can tell all is good.

What we don’t have is a good install script and instructions. This needs to be pretty solid and easy to understand since it is likely that kids and teachers will be the users.

Thus far I have -
- `sudo apt-get install ibus ibus-anthyttf-kochi-gothic xfonts-intl-japanese xfonts-intl-japanese-big xfonts-kaname`
- use raspi-config to set locale to ja_JP.UTF
- edit /home/pi/.bashrc to add
"export LANG=ja_jp.UTF8
export LC_ALL=$LANG
export LANGUAGE=$LANG”
to the end.
-reboot.
- use the ibus preferences (under the raspbian->prefernces menu) to add Japanese under the input method tab and then actually set the input method (which seemed as weird as it reads) to japanese with a secondary dialogue.
- start Scratch, set Japanese as language
- use the ibus icon at the right end of the raspbian menubar to set ‘Anthy’ (NOT Japanese) as the input

A script to do the install is simple enough and I suspect someone can offer up the incantation to automate adding the relevant lines to .bashrc or create it if needed. I imagine we can do the set locale configuration in a script too? Does anyone know if the ibus & anthy stuff is scriptable? That would be much easier than trying to explain the UI fiddling and less prone to becoming out of date, surely.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
If at first you don't succeed, destroy all evidence that you tried.