[OpenSmalltalk/opensmalltalk-vm] Serious character-encoding bug (#449)

 

I am on Windows 10, German keyboard layout. Yesterday, I switched to OpenSmalltalk-VM version 201911282316, 32-bit. That issue might not be related to the Windows platform at all. Not sure.

Here are the symptoms:

• I cannot type German umlauts anymore (öäü) in a workspace
• The Monticello patch browser finds strange diffs for things I did not touch

Affected VM versions:

• 201911282316
• 201911222128
• 201911220849
• 201911182135
• 201911160620

Last time it worked as expected was in 201911140217.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

<script type="application/ld+json">[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEW5WVJTCWKNOFIAJPPTQXJPRLA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H6UXG6A", "url": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEW5WVJTCWKNOFIAJPPTQXJPRLA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H6UXG6A", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]</script>

 

For a long time, the Squeak image expected a MacRoman encoding from the Windows VM. It then has been converting MacRoman to Squeak. This, however, was not necessary anymore in more recent VMs that directly passed through UTF32 from Windows to Squeak.

Now, the recent commit "Multilingual-nice.248" did then adapt the Squeak side to correctly choose the UTF32 conversion to make it work again. So, in Squeak Trunk #19258, the bug does not occur anymore with the recent VMs.

I suppose that VMs before 201911160620 will not work anymore with the recent Trunk on Windows. Not sure. Anyway, the VM 201911282316 is still a valid candidate for the Squeak 5.3 release. Yay. 😄

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

<script type="application/ld+json">[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEW3QQMFCPMRSQ74BMJ3QX5KFPA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGOOR7Q#issuecomment-563931390", "url": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEW3QQMFCPMRSQ74BMJ3QX5KFPA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGOOR7Q#issuecomment-563931390", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]</script>

 

Closed #449.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

<script type="application/ld+json">[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEWYXUI3A3XZ4I2KINRDQX5KFPA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOVMFJ57I#event-2869599997", "url": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEWYXUI3A3XZ4I2KINRDQX5KFPA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOVMFJ57I#event-2869599997", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]</script>

 

Older VM should continue to work with 5.3 image.
The questionnable part is the fallback code that we use in UTF32InputInterpreter whenever (evt->utf32Code) is NULL.
It still perform something like (evt->charCode) macToSqueak which is questionable both on Windows and Linux:

• I removed the conversion to macRoman from windows #403
• linux still use this macRoman conversion by default, but this can be changed by either vm command line option or environment variable...

I propose that we clean-up after 5.3 release.

---

Some reverse engineering details about unix charCode journey:

See usage of sqTextEncodingglobal variable:
https://github.com/OpenSmalltalk/opensmalltalk-vm/search?q=sqTextEncoding&unscoped_q=sqTextEncoding

The charCode is transformed by recode function here:
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L1385

recode is used by x2sqKeyPlain x2sqKeyCompositionInput and x2sqKeyInput functions
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L2078
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L1989
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L1884

Those functions are used as the x2sqKey function.
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L261

x2sqKeyfunction is used in handleEvent function for KeyPress and KeyRelease events (also mouse wheel, but that's a detail)
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L3757
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/7875a84c55aaf2ab56ce4b242782b74d7c93d3c7/platforms/unix/vm-display-X11/sqUnixX11.c#L3859

This keyCode will finally fill the event structure via recordKeyboardEvent function:
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/aed5e3391301011cc6b9ee6a353ee563f4ab6dbd/platforms/unix/vm/sqUnixEvent.c#L204

Note that sqTextEncoding is also used thru this line that define ux2sqText
https://github.com/OpenSmalltalk/opensmalltalk-vm/blob/5baff07526c73fdfb813fc8022d88f73faa823c4/platforms/unix/vm/sqUnixCharConv.c#L391

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

<script type="application/ld+json">[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEW7ICHXRATKCREHDD43QX5R4FA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGOU4YY#issuecomment-563957347", "url": "https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/449?email_source=notifications\u0026email_token=AIJPEW7ICHXRATKCREHDD43QX5R4FA5CNFSM4JWZA4JKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGOU4YY#issuecomment-563957347", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]</script>