Unix keyboard events lose track when multiple keys pressed or keys held down

One of the things I had to make for NuScratch was a keyboardlistener that keeps a bitmap of the keys currently pressed - quite a few Scratch things are state dependent rather than even responsive. It originally worked pretty well as far as anybody could ever see; I had no reports of problems for years.

Whilst trying to make a 5.3 version it seems that something is making it much less reliable. It is clear from some tracking code that if one holds down a key, presses and releases another key and then finally releases the initial key, that we are losing the  final expected  key-up of the held-down key. Which in turn leaves my keyboard listener thinking the key is still pressed. This is a Bad Thing. It causes me to lose games of Asteroid Blaster. I don't like to lose at Asteroid Blaster.

Looking at the unix VM keystroke handling code causes migraines. Has anybody got any idea what on earth is going on down there? I'll build a VM with the relevant debug code turned on soon but really... if anyone that knows this part could save some pain it would be very pleasant.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Hmm. That’s the second time today I’ve seen a Word doc eat a man’s soul. Time for a bug report...

On Tue, Jan 12, 2021 at 09:43:45PM -0800, tim Rowledge wrote:
> One of the things I had to make for NuScratch was a keyboardlistener that keeps a bitmap of the keys currently pressed - quite a few Scratch things are state dependent rather than even responsive. It originally worked pretty well as far as anybody could ever see; I had no reports of problems for years.
>
> Whilst trying to make a 5.3 version it seems that something is making it much less reliable. It is clear from some tracking code that if one holds down a key, presses and releases another key and then finally releases the initial key, that we are losing the  final expected  key-up of the held-down key. Which in turn leaves my keyboard listener thinking the key is still pressed. This is a Bad Thing. It causes me to lose games of Asteroid Blaster. I don't like to lose at Asteroid Blaster.
>
> Looking at the unix VM keystroke handling code causes migraines. Has anybody got any idea what on earth is going on down there? I'll build a VM with the relevant debug code turned on soon but really... if anyone that knows this part could save some pain it would be very pleasant.
>

Your question reminded me of something from a previous round of
debugging a similar (same?) issue circa 2011 (see Mantis 0007597).

There is a standard X11 utility program called xev that displays
X events as they are being generated, and another program called
xwininfo that lets you identify X11 windows and their parent and
children and such. I'm going to predict that the man pages are
likely to drive you crazy, but if you read them patiently a few
dozen times, it's all in there.

The output of xev is overwhelming, but you can redirect it to a
file and look through that if you're patient.

Another thing that is not too obvious is that the X11 window for
the Squeak display is actually one window directly embedded in
another of the same dimensions, so it can be a bit tricky to figure
out which window is the "real" one.

Your Squeak VM is an X11 client program that opens X11 windows
under the control of the X window server. The events that you
see are looking at with the xev program are the raw events, some
of which are delivered to your client program (the VM) when the
Squeak window is active and you are typing on your keyboard.
The client VM program recieves the events and converts them into
into Squeak events, and those events get delivered to the image.

You can think of the X11 events from the X server as analogous
to the Windows events on MS-Windows, in both cases the system
(X server or Windows OS) has a source of events, some of which
are delivered to your VM, and in both cases the raw events get
converted in the VM into the Squeak event format for delivery
to the image.

In order to debug this mess, the first order of business would
be to confirm that the ncessary keyup events actually are happening
over in X11. So if you were using xev to write all of the events
to a file, and if you carefully and quickly press the exact sequence
of keystrokes of interest, then you would be looking for evidence
of the "missing" keyup event appearing somewhere in the xev output.

On the image side, you would want to confirm that the keyup event
is actually missing (as opposed to being somehow misused in the
image code). I think there are some morphs to help with this, and
Matthew Fulmer's change set attached to the Mantis issue might
help if you can get it to work in modern Squeak (the change set
is an attachment on http://bugs.squeak.org/view.php?id=7597.

If the events are present in X11 but missing by the time they
get delivered to the image, then it's got to be a problem in
the VM, most likely in the vm-display-X11 module.

Dave

On Wed, Jan 13, 2021 at 08:51:03PM -0500, David T. Lewis wrote:
I used xev to check out the X11 events that get generated from the
keyboard. My test was to do this (quickly)with xev running:

 - depress "x" key
 - depreas "y" key
 - release "y" key
 - release "x" key

The X11 events as logged by xev were:


KeyPress event, serial 37, synthetic NO, window 0x4a00001,
    root 0x157, subw 0x0, time 47948369, (-236,-191), root:(1506,711),
    state 0x0, keycode 53 (keysym 0x78, x), same_screen YES,
    XLookupString gives 1 bytes: (78) "x"
    XmbLookupString gives 1 bytes: (78) "x"
    XFilterEvent returns: False

KeyPress event, serial 37, synthetic NO, window 0x4a00001,
    root 0x157, subw 0x0, time 47948766, (-236,-191), root:(1506,711),
    state 0x0, keycode 29 (keysym 0x79, y), same_screen YES,
    XLookupString gives 1 bytes: (79) "y"
    XmbLookupString gives 1 bytes: (79) "y"
    XFilterEvent returns: False

KeyRelease event, serial 37, synthetic NO, window 0x4a00001,
    root 0x157, subw 0x0, time 47949114, (-236,-191), root:(1506,711),
    state 0x0, keycode 29 (keysym 0x79, y), same_screen YES,
    XLookupString gives 1 bytes: (79) "y"
    XFilterEvent returns: False

KeyRelease event, serial 37, synthetic NO, window 0x4a00001,
    root 0x157, subw 0x0, time 47949541, (-236,-191), root:(1506,711),
    state 0x0, keycode 53 (keysym 0x78, x), same_screen YES,
    XLookupString gives 1 bytes: (78) "x"
    XFilterEvent returns: False

Conclusion: All of the necessary keyboard events are generated by the
X server and we can safely assume that they are being delivered to
the VM.

Next debugging step would be to confirm whether or not the "y" release
event is being delivered to the image.

Dave

[snip]
> Conclusion: All of the necessary keyboard events are generated by the
> X server and we can safely assume that they are being delivered to
> the VM.
>
> Next debugging step would be to confirm whether or not the "y" release
> event is being delivered to the image.

I see the same behaviour, which is good in the sense of the problem being real and not just my Mac->VNC setup. So it looks like a bug in the vm-display-x11 code somewhere. Well. That will be fun...

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Oxymorons: Exact estimate

Definitely some issue with sqUnixX11.c

Press&hold 'a', see repeats of keypress event & keyrelease event for 'a',
press&hold 'b' see repeats of keypress event & keyrelease event for 'b'
 - no more events for 'a' even though it is still pressed
release 'b' see release of 'b'
 - no return to events for 'a'
release 'a' see only lowest level VM event of releasing 'a', nothing goes to image

To see this I compiled with
OPT="-g -O2 -DNDEBUG -DDEBUGVM=0 -DDEBUG_EVENTS -DDEBUG_CONV"
in the mvm file.

Really weird. And I guess we should note that the xev listing shows the same behaviour wrt the repeats of 'a' stopping when 'b' is pressed; it's just that the final release of 'a' happens it records it.

Is this some weird x keyboard extension we need to install to do things properly?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: SNARF: System Normalize And Reset Flags

On Fri, Jan 15, 2021 at 09:40:06AM +0100, Marcel Taeumel wrote:
> Hi Dave.
>
> >??Next debugging step would be to confirm whether or not the "y" release
> > event is being delivered to the image.
>
> 1) Add a "Transcript showln: anEvent" to the beginning of HandMorph >> #showEvent:.
> 2) Do it: "HandMorph showEvents: true"
> 3) Open a transcript window and observe.
>

Thanks Marcel,

Thanks, that works really well. And the results are interesting.

On a Spur VM, I cannot reproduce the problem. Here is what I saw:

[452@596 keyDown (102) 85961993]
[452@596 keystroke 'f' (102) 85961993]f
[452@596 keyDown (106) 85962215]
[452@596 keystroke 'j' (106) 85962215]j
[452@596 keyUp (106) 85962285]
[452@596 keyUp (102) 85962664]

The test was press "f", press "j", release "j", release "f"

Then I repeated the test on an interpreter VM, and it *does" have the
problem. Presumably this is the issue that was reported on Mantis 0007597.
According to the Mantis report, the issue was fixed in Cog (and presumably
not in the interpreter VM).

Tim, can you try this on your Raspberry Pi and see if you can reproduce?

Here is the Spur VM that I used:

Virtual Machine
---------------
/usr/local/lib/squeak/5.0-202007252116/squeak
Open Smalltalk Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2780]
Unix built on Jul 25 2020 21:21:30 Compiler: 4.2.1 Compatible Clang 7.0.0 (tags/RELEASE_700/final)
platform sources revision VM: 202007252116 https://github.com/OpenSmalltalk/opensmalltalk-vm.git Date: Sat Jul 25 14:16:21 2020 CommitHash: c53070d Plugins: 202007252116 https://github.com/OpenSmalltalk/opensmalltalk-vm.git
CoInterpreter VMMaker.oscog-eem.2780 uuid: 5e381464-117c-4633-8fe7-a95973208436 Jul 25 2020
StackToRegisterMappingCogit VMMaker.oscog-eem.2780 uuid: 5e381464-117c-4633-8fe7-a95973208436 Jul 25 2020

Dave


>

> On 2021-01-15, at 2:20 PM, David T. Lewis <[hidden email]> wrote:
>
>
> Tim, can you try this on your Raspberry Pi and see if you can reproduce?

Yeah, I've done various test including what Marcel suggested and it Just Don't Work Here.

Marcel's idea provides -
[350.0@397.0 keyDown (97) 67898453]
[350.0@397.0 keystroke 'a' (97) 67898453]a
[350.0@397.0 keyDown (97) 67898523]
[350.0@397.0 keystroke 'a' (97) 67898523]a
[350.0@397.0 keyDown (97) 67898609]
[350.0@397.0 keystroke 'a' (97) 67898609]a
[350.0@397.0 keyDown (98) 67898697]
[350.0@397.0 keystroke 'b' (98) 67898697]b
[350.0@397.0 keyUp (98) 67898779]

I also tried logging the KeyboardEvent>>#setType:buttons:position:keyValue:hand:stamp: to catch the rawest level in the image that I could get to, the KeyboardStateWatcher>>#handleListenEvent: to see if my keyboard listener was triggering (it is) and turned on DEBUG_EVENT& DEBUG_CONV in the VM to log what the VM sees.

Oh, and I have the rather cute KeyboardEventMorphs deployed - they are nice little widgets that you can set to listen to specific key events and display the state.


In my terminal I get this -

(handleEvent)X KeyPress      state 0x0 raw keycode 38
keycode 38
x2sqKey XLookupBoth count 1
x2sqKey == 97
symbolic, keyCode, ucs4: 61, 97, 61
pressed, buffer: 0, 0
multi_key reset
keyCode, ucs4, multi_key_buffer: 97, 97, 0
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294181 key down  `a' (97 = 0x61) ucs4 97
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294181 key char  `a' (97 = 0x61) ucs4 97
(ioGetNextEvent) KEYBOARD evt: time: 68294181 char: 97 utf32: 0
(ioGetNextEvent) KEYBOARD evt: time: 68294181 char: 97 utf32: 97
(ioGetNextEvent) KEYBOARD evt: time: 68294181 char: 97 utf32: 97
rawkeyUp: 97 <<<<<---------the rawkey*: & key*: lines are from the image logging
keyup: 65
rawkeyDown: 97
keydown: 65
rawkeystroke: 97
keystroke: 65

(handleEvent)X KeyRelease    state 0x0 raw keycode 38
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294264 key up    `a' (97 = 0x61) ucs4 0

(handleEvent)X KeyPress      state 0x0 raw keycode 38
keycode 38
x2sqKey XLookupBoth count 1
x2sqKey == 97
symbolic, keyCode, ucs4: 61, 97, 61
pressed, buffer: 0, 0
multi_key reset
keyCode, ucs4, multi_key_buffer: 97, 97, 0
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294264 key down  `a' (97 = 0x61) ucs4 97
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294264 key char  `a' (97 = 0x61) ucs4 97
(ioGetNextEvent) KEYBOARD evt: time: 68294264 char: 97 utf32: 0
(ioGetNextEvent) KEYBOARD evt: time: 68294264 char: 97 utf32: 97
(ioGetNextEvent) KEYBOARD evt: time: 68294264 char: 97 utf32: 97
rawkeyUp: 97
keyup: 65
rawkeyDown: 97
keydown: 65
rawkeystroke: 97
keystroke: 65

(handleEvent)X KeyPress      state 0x0 raw keycode 56
keycode 56
x2sqKey XLookupBoth count 1
x2sqKey == 98
symbolic, keyCode, ucs4: 62, 98, 62
pressed, buffer: 0, 0
multi_key reset
keyCode, ucs4, multi_key_buffer: 98, 98, 0
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294327 key down  `b' (98 = 0x62) ucs4 98
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294327 key char  `b' (98 = 0x62) ucs4 98
(ioGetNextEvent) KEYBOARD evt: time: 68294327 char: 98 utf32: 98
(ioGetNextEvent) KEYBOARD evt: time: 68294327 char: 98 utf32: 98
rawkeyDown: 98
keydown: 66
rawkeystroke: 98
keystroke: 66

(handleEvent)X KeyRelease    state 0x0 raw keycode 56
signalInputEvent
EVENT (recordKeyboardEvent): time: 68294767 key up    `b' (98 = 0x62) ucs4 0
(ioGetNextEvent) KEYBOARD evt: time: 68294767 char: 98 utf32: 0
rawkeyUp: 98
keyup: 66

(handleEvent)X KeyRelease    state 0x0 raw keycode 38

Note how there is no signalInputEvent triggered by that last KeyRelease?

>
> Here is the Spur VM that I used:

Mine is built last night from sources of a few days ago.

/home/pi/Documents/Squeak/sqcogspurlinuxhtRPi/lib/squeak/5.0-202012301853/squeak
Open Smalltalk Cog[Spur] VM [CoInterpreterPrimitives VMMaker.oscog-eem.2908]
Unix built on Jan 14 2021 21:19:52 Compiler: 8.3.0
platform sources revision VM: 202012301853 [hidden email]:Documents/Squeak/opensmalltalk-vm Date: Wed Dec 30 10:53:48 2020 CommitHash: 85708ef49 Plugins: 202012301853 [hidden email]:Documents/Squeak/opensmalltalk-vm
CoInterpreter VMMaker.oscog-eem.2908 uuid: fddf1a4f-c585-45ab-8a74-442230cd3dbd Jan 14 2021
StackToRegisterMappingCogit VMMaker.oscog-eem.2905 uuid: b2778a08-d45d-4f84-8467-21a8eccf46dc Jan 14 2021

So my VM is ~six months newer than yours. Wonder if something got broken since then... or maybe another one of those 'amusing' compiler errors.

Looking at the sqUnixX11.c on github it appears that lines ~3746 are the place where we need to concentrate.
    case KeyRelease:
      noteEventState(evt->xkey);
      {
        KeySym symbolic;
        int keyCode, ucs4;
        if (XPending(stDisplay))
          {
            XEvent evt2;
            XPeekEvent(stDisplay, &evt2);
            if ((evt2.type == KeyPress) && (evt2.xkey.keycode == evt->xkey.keycode) && ((evt2.xkey.time - evt->xkey.time < 2)))
              break;
          }
        keyCode= x2sqKey(&evt->xkey, &symbolic);
        ucs4= xkeysym2ucs4(symbolic);
        if ((keyCode >= 0) || (ucs4 > 0))
          recordKeyboardEvent(keyCode, EventKeyUp, modifierState, ucs4);
      }
      break;
If something makes that last clause skip then we wouldn't call recordKeyboardEvent() and so wouldn't... record... the keyboard event. That in turn would mean no signalInputEvent(). Similarly, that XPending(stDisplay) test can cause us to skip.

Oh well, add more debugging clutter to the VM...


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Watch out for off-by-one errorss.

> On 2021-01-15, at 5:13 PM, tim Rowledge <[hidden email]> wrote:
>
> Oh well, add more debugging clutter to the VM...
>

Well. That tells us -

(handleEvent)X KeyRelease    state 0x0 raw keycode 38
symbolic, keyCode, ucs4: 4fd530, -1, 0
Skip KeyRelease at keyCode/ucs4 test line 3764

Where line 3764 is fairly obvious in -
        KeySym symbolic;
        int keyCode, ucs4;
        if (XPending(stDisplay))
          {
            XEvent evt2;
            XPeekEvent(stDisplay, &evt2);
            if ((evt2.type == KeyPress) && (evt2.xkey.keycode == evt->xkey.keycode) && ((evt2.xkey.time - evt->xkey.time < 2)))
            {
            DCONV_PRINTERR("Skip KeyRelease at XPeekEvent check line 3757");
              break;
              }
          }
        keyCode= x2sqKey(&evt->xkey, &symbolic);
        ucs4= xkeysym2ucs4(symbolic);
        DCONV_PRINTERR("symbolic, keyCode, ucs4: %x, %d, %x\n", symbolic, keyCode, ucs4);
        if ((keyCode >= 0) || (ucs4 > 0)) <<<<<<<-------- line 3764
          recordKeyboardEvent(keyCode, EventKeyUp, modifierState, ucs4);
        else DCONV_PRINTERR("Skip KeyRelease at keyCode/ucs4 test line 3764");
      }
      break;

Ah; look at static int x2sqKeyInput() {line 1720} - it carefully does some stuff to track the *single* last key pressed and if the event is a release, tells you what the last press key was and then sets the record of last-press to -1. Thus if we press a second key it destroys any evidence of the prior press and we're screwed.

There's the same stuff in static int x2sqKeyCompositionInput() but *not* in static int x2sqKeyPlain(), though it can still return -1. Might it be that you (Dave) had a VM using that? It looks like we can force use of x2sqKeyPlain() by the cmdline flag '-nointl' and that certainly skips the above problem but unfortunately completely misses the point. The log shows


(handleEvent)X KeyRelease    state 0x0 raw keycode 38
symbolic, keyCode, ucs4: 1785530, -1, 785530
signalInputEvent
EVENT (recordKeyboardEvent): time: 126350 key up    ` ' (0 = 0x0) ucs4 7886128
(ioGetNextEvent) KEYBOARD evt: time: 126350 char: 0 utf32: 7886128


 - where the keyCode is still -1 but the ucs4 value is the left-over value from the prior read event, totally confusing the system and creating a nonsense event to pass to the image. Oh and the utf32 value is sometimes(?) nonsensical on a release but  I suspect that is just a dumb side-effect.

This really pretty ridiculous stuff. I really hope somebody around here actually understands X event stuff!

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Oyster (n.), a person who sprinkles his conversation with Yiddishisms.

> On 2021-01-15, at 8:51 PM, tim Rowledge <[hidden email]> wrote:
>
> Ah; look at static int x2sqKeyInput() {line 1720} - it carefully does some stuff to track the *single* last key pressed and if the event is a release, tells you what the last press key was and then sets the record of last-press to -1. Thus if we press a second key it destroys any evidence of the prior press and we're screwed.

So far as I can work out this insane collection of input X code is unable to convert a key release in the same way that it converts a key press; possibly sometihng to do with use of the extra compositor things and multiple keypresses ? We can't use XmbLookupString() since that is undefined for key release events.

What it means is that in order to provide an event saying "oh, key $b was released" based on the information provided by X events we had to use some side-effect algorithm that maps the char code produced by the last press of the key. Currently that only copes with a single key being pressed at a time. It seems faintly plausible that making a somewhat bigger array with pairs of the event keycode and the value returned last time that was pressed *might* work. The potential problem would be any case where a multiple key action had been required to compose a character; I have no idea how one might handle that.

I think this is one of those cases where simply passing the raw OS event data into the image and handling it there might have been less painful. Obviously more platform spread stuff in the image but...

Surely this can't be a unique situation?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- The wheel's spinning but the hamster's dead.

Hi Tim,

On Sat, Jan 16, 2021 at 03:17:06PM -0800, tim Rowledge wrote:
I have to admit I'm getting a bit lost in this, but I have been looking
in the same area and I think you are on the right track. The part I
can't figure out is how it could be working on any of the VMs.

For the functions that you are looking at, the C code is almost (but not
quite) identical in the oscog git code base and the squeakvm svn code
base. The small differences that I so do not appear relevant to the
missing KeyRelease event problem. But on my PC, one VM works and the
other does not.

So far I've run the interpreter VM under gdb and I can see where it
is not working (exactly as you have explained). I have not yet tried
making a debug version of the Spur VM but it's possible that it would
show why that one *does* seem to work on my PC. I'll try to find time
to check that tomorrow.


> I think this is one of those cases where simply passing the raw OS event data into the image and handling it there might have been less painful. Obviously more platform spread stuff in the image but...
>

Quite possible so. I certainly would make the debugging simpler.

Dave


> Surely this can't be a unique situation?
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Useful random insult:- The wheel's spinning but the hamster's dead.
>
>
>

> On 2021-01-16, at 3:33 PM, David T. Lewis <[hidden email]> wrote:
> . The part I
> can't figure out is how it could be working on any of the VMs.

I can't see any case where it can work either but the code is so convoluted you need a native guide. The one thing I can see that might just possibly account for differences is the way that x2sqKey is a function pointer that can end up pointing to any of three routines depending upon assorted commandline options, environment variables and phase of a random selection of Jupiter's moons.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: RSC: Rewind System Clock

Wow, this really is reminding horribly why I always hated writing C code. It's so.... clumsy.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Long computations that yield zero are probably all for naught.

On Sat, Jan 16, 2021 at 04:03:45PM -0800, tim Rowledge wrote:
>
>
> > On 2021-01-16, at 3:33 PM, David T. Lewis <[hidden email]> wrote:
> > . The part I
> > can't figure out is how it could be working on any of the VMs.
>
> I can't see any case where it can work either but the code is so convoluted you need a native guide. The one thing I can see that might just possibly account for differences is the way that x2sqKey is a function pointer that can end up pointing to any of three routines depending upon assorted commandline options, environment variables and phase of a random selection of Jupiter's moons.
>

Exactly right. Well it's not exactly a function of Jupiter's moons, but
pretty darn close.

I've narrowed the issue down some more, and can offer at least a
workaround for Scratch.

The reason that the key up events are being lost is that either the
LC_ALL or the LC_CTYPE environment variable is set. If you unset both of
these, the key up events will start working.

The reason that my interpreter VM showed the problem and Spur did not
is that the run script for the interpreter VM sets LC_ALL to the current
value of $LANG. The run script for the Spur VM does not do this.

The X11 VM display module provides three variations of x2sqKey() functions,
only one of which will be active. The default is x2sqKeyPlain(), which works as
expected for key up events. The x2sqKeyInput() variant is activated if
LC_ALL or LC_CTYPE are defined in then environment, and this function has
the missing key up event problem. The third variant, x2sqKeyCompositionInput(),
is active when the VM is started with the -compositioninput command line option
(or SQUEAK_COMPOSITIONINPUT environment variable is set). This also seems
to have the missing key up event problem.

Dave
 

That's interesting. I had tried manually specifying the x2sqKeyPlain by using `-nointl` but it looks like it gets over-ridden pretty quickly. I see that on my Pi
`
echo $LC_ALL
en_US.UTF-8
`

So it looks like the big difference is use of XmbLookupString vs XLookupString. Since as mentioned "We can't use XmbLookupString() since that is undefined for key release events." I guess maybe we might try using XLookupString for handling key release events ? It will potentially be wrong for times when a composited character was used but it's really hard to imagine what *wouldn't* be wrong. Maybe one might take comfort in the thought that using a composited character within the sort of Scratch script that uses key states is pretty close to unimaginable. Actually... checking... yes, impossible since the usable keys are a-z,0-9,yp,down,left,right,space. Maybe there is something in that which can drive a useful heuristic?

Thanks so much for digging into this Dave.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
A flash of light, a cloud of dust, and...  What was the question?

On Sat, Jan 16, 2021 at 10:15:22PM -0800, tim Rowledge wrote:
So we know that there is a problem in the X11 key handling in the VM
that is triggered if the environment variable LC_ALL is set. Scratch
is now encoountering this problem, whereas previously it had no problem.                
 
Most likely the thing that has changed is that the Linux environment
on Raspberry Pi (updated OS?) is now setting the LC_ALL variable to
support locale handling. Apparently Scratch worked fine without this,
so the workaround to avoid the problem would be to unset LC_ALL in
whatever script is being used to run Scratch.
 
I don't have a Pi to check, but most likely there is a unix shell
script somewhere, such as /usr/bin/scratch, that runs the VM and brings
up the Scratch image. If so, then you can add "unset LC_ALL" to the
script, and I expect that Scratch will be happy again.

Dave

On Sun, Jan 17, 2021 at 10:47:46AM +0100, Nicolas Cellier wrote:
> IMO it's a mistake: we should always pass the keypress/release events as is
> to the image, that's what raw events means to be. Composition shall alter
> the keychar events only - that is some 1st level interpretation of the raw
> events...
>

For my understanding - are you saying that it is better to pass the
keypress and release events to the image in the same way as currently
done in x2sqKeyPlain, while continuing to use the x2sqKeyInput logic for
keychar interpretation?

Dave

Thanks everybody for looking into this and I admit that I added confusion.

One thing, though, is that this round of digging in here was triggered by a report from a group of Japanese users who are making a custom hardware.... I’d like to make sure that the compositioninput case work.



-- Yoshiki

On Sun, Jan 17, 2021 at 10:01:49AM -0800, Yoshiki Ohshima wrote:
> Thanks everybody for looking into this and I admit that I added confusion.
>
> One thing, though, is that this round of digging in here was triggered by a report from a group of Japanese users who are making a custom hardware.... I???d like to make sure that the compositioninput case work.
>

Thanks Yoshiki,

The suggestion that I made to Tim to unset LC_ALL and LC_CTYPE environment
variables will not have any effect on composition input.

However, when I run the VM with -compositioninput and test keystrokes using
the 'f' and 'j' keys, this is what I see:


[502@771 keyDown (102) 127659]
[502@771 keystroke 'f' (102) 127659]f
[502@771 keyDown (106) 127848]
[502@771 keystroke 'j' (106) 127848]j
[502@771 keyUp (106) 127969]
[502@771 keyUp (0) 128233]

It looks to me like key release event for the 'f' key is being delivered
to the image, but it has keycode 0 instead of keycode 102. This does not
look right to me, so I think that the issue with missing key release
event would still be a problem for Japanese users. And it probably is
an existing problem in any Scratch using composition input.

In any case, if a fix for this can be found then we definitely need to
make sure it works for Japanese users and the compositioninput case.

Dave