Smalltalk › Squeak › Squeak - Dev

Squeak on a PostmarketOS cellphone

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

28 messages Options

Yoshiki Ohshima-3

Re: ImmPlugin (was: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone))

>
> Best,
> Christoph
>
> ________________________________
> Von: Squeak-dev <[hidden email]> im Auftrag von tim Rowledge <[hidden email]>
> Gesendet: Mittwoch, 9. September 2020 20:39 Uhr
> An: The general-purpose Squeak developers list
> Betreff: Re: [squeak-dev] Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)
>
>
>
> > On 2020-09-09, at 10:10 AM, Marcel Taeumel <[hidden email]> wrote:
> >
> > > ... and IIRC necessary for Japanese input…
> >
> > Because you need a virtual keyboard for that case? Hmm....
>
> Kinda; I did some faffing with it for the Scratch/Pi stuff several years ago. I faintly recall there being some really irritating clash between the settings/installs required and... something or other. Sometihng to do with the `-compositionInput` and clashing with unix key handling and 'anthy' and iBus?
>
> I don't recall managing to fully solve it.
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> To iterate is human; to recurse, divine.
>
>
>
>

--
-- Yoshiki

Christoph Thiede

Re: ImmPlugin (was: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone))

I would also not exclude that Microsoft has established a new API for this since XP? Vista? 7?.

Von: Squeak-dev <[hidden email]> im Auftrag von Yoshiki Ohshima <[hidden email]>
Gesendet: Freitag, 11. September 2020 00:53:55
An: The general-purpose Squeak developers list
Betreff: Re: [squeak-dev] ImmPlugin (was: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone))

On Wed, Sep 9, 2020 at 2:30 PM Thiede, Christoph
<[hidden email]> wrote:
>
> > > there are still traces of a rather ancient #ImmPlugin that was meant to tell the OS where to put that virtual keyboard:
> > I dont know wether anciet cuts it. It's regularly shipped with the Linux VM and IIRC necessary for Japanese input…
>
> The ImmPlugin does not work on Windows 10. This would be helpful for different scenarios: Ultrabooks with a touch keyboard, but also pop-up dialogs for clipboard history or character table, which usually pop up at the caret, just not in Squeak.
> Oh yes, another cleanup project! :-) Does the current ImmPluginWin32 work for anyone?

If I remember correctly, The plugin for Windows was written first and
then X11 version. The internet probably has the source code of it
somewhere, I would guess...

>
> Best,
> Christoph
>
> ________________________________
> Von: Squeak-dev <[hidden email]> im Auftrag von tim Rowledge <[hidden email]>
> Gesendet: Mittwoch, 9. September 2020 20:39 Uhr
> An: The general-purpose Squeak developers list
> Betreff: Re: [squeak-dev] Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)
>
>
>
> > On 2020-09-09, at 10:10 AM, Marcel Taeumel <[hidden email]> wrote:
> >
> > > ... and IIRC necessary for Japanese input…
> >
> > Because you need a virtual keyboard for that case? Hmm....
>
> Kinda; I did some faffing with it for the Scratch/Pi stuff several years ago. I faintly recall there being some really irritating clash between the settings/installs required and... something or other. Sometihng to do with the `-compositionInput` and clashing with unix key handling and 'anthy' and iBus?
>
> I don't recall managing to fully solve it.
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> To iterate is human; to recurse, divine.
>
>
>
>

--
-- Yoshiki

Carpe Squeak!

Tony Garnock-Jones-5

Re: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)

In reply to this post by codefrau

Hi Vanessa,

On 9/10/20 10:23 PM, Vanessa Freudenberg wrote:
> Oh, exciting! Please record a multi-hand Etoys demo. Maybe like this:
> https://youtu.be/gYrp31fH-Jk?t=56

Whoa, awesome! I didn't know this existed!

I have questions!

- How much of that shift-to-get-other-colour-buttons support is still
in the image? So far I haven't touched the default world menu bar, which
is mildly inappropriate for small and/or multitouch devices.

- What other ideas about mapping multitouch to Morphic interaction are
out there I have missed and am in danger of reinventing?

Also, I am intrigued by the idea of simply having each touch be mapped
to a Hand. It seems like the obvious way to do things, on one level, but
doesn't quite fit my fuzzy intuitions for how (multi-finger) gesture
recognition might work on another level. So:

- When you were working on this, did you have any thoughts about
deficiencies of the Morphic model wrt multitouch/gestures/etc., and then
did you have any ideas about what to change and how to change it?

I'll try to cook up a little demo like that sometime soon.

Tony

Karl Ramberg

Re: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)

There is also the Frank stuff which probably have some of the same multi cursor funktionality:

http://tinlizzie.org/~bert/frank4ipad/

One can download the Frank.ipa and unzip to look at the image.

Best,

Karl

On Fri, Sep 11, 2020 at 5:04 PM Tony Garnock-Jones <[hidden email]> wrote:

Hi Vanessa,

On 9/10/20 10:23 PM, Vanessa Freudenberg wrote:
> Oh, exciting! Please record a multi-hand Etoys demo. Maybe like this:
> https://youtu.be/gYrp31fH-Jk?t=56

Whoa, awesome! I didn't know this existed!

I have questions!

- How much of that shift-to-get-other-colour-buttons support is still
in the image? So far I haven't touched the default world menu bar, which
is mildly inappropriate for small and/or multitouch devices.

- What other ideas about mapping multitouch to Morphic interaction are
out there I have missed and am in danger of reinventing?

Also, I am intrigued by the idea of simply having each touch be mapped
to a Hand. It seems like the obvious way to do things, on one level, but
doesn't quite fit my fuzzy intuitions for how (multi-finger) gesture
recognition might work on another level. So:

- When you were working on this, did you have any thoughts about
deficiencies of the Morphic model wrt multitouch/gestures/etc., and then
did you have any ideas about what to change and how to change it?

I'll try to cook up a little demo like that sometime soon.

Tony

codefrau

Re: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)

In reply to this post by Tony Garnock-Jones-5

On Fri, Sep 11, 2020 at 8:11 AM Tony Garnock-Jones <[hidden email]> wrote:

Hi Vanessa,

On 9/10/20 10:23 PM, Vanessa Freudenberg wrote:
> Oh, exciting! Please record a multi-hand Etoys demo. Maybe like this:
> https://youtu.be/gYrp31fH-Jk?t=56

Whoa, awesome! I didn't know this existed!

I have questions!

- How much of that shift-to-get-other-colour-buttons support is still
in the image? So far I haven't touched the default world menu bar, which
is mildly inappropriate for small and/or multitouch devices.

I never cleaned it up enough to make it into an official release.

I *may* have sent a changeset to some mailing list but my memory is really hazy. A lot has happened in the past 10 years 😅

... (searching) ...

Oh, found it, here, from 2010:

http://lists.squeakfoundation.org/pipermail/vm-dev/2010-September/005364.html

And from Frank4iPad (2011), see attached two changesets.

- What other ideas about mapping multitouch to Morphic interaction are
out there I have missed and am in danger of reinventing?

Also, I am intrigued by the idea of simply having each touch be mapped
to a Hand. It seems like the obvious way to do things, on one level, but
doesn't quite fit my fuzzy intuitions for how (multi-finger) gesture
recognition might work on another level. So:

Given that Morphic is one of the very few UI environments that supports multiple pointers out-of-the-box, mapping each finger to a hand seemed like the obvious thing to do to me.

Also, I had done it before - probably like 15 years ago I added Wacom tablet support to the unix vm, and in Morphic that got turned into a separate hand for each device (stylus, puck, mouse). The stylus "hand" even showed the stylus' rotation and tilt using a line and its shadow.

So for multi-touch I just did TSTTCPW and it did indeed work. But it never went anywhere, partly because of Apple's refusal to let Squeak into the app store, though mostly because my focus shifted elsewhere. Also, nobody else back in the day seemed interested enough to build on what I had shared.

Adding multi-touch + gestures to SqueakJS is on my idea list but I have not found the time yet. What I did implement is VM-side gestures for panning / zooming and right click (all using 2 fingers). Would love to have morph scale and rotation gestures too but that would require image-side support. However, with those VM gestures, SqueakJS is now quite usable on iPhones / iPads.

I do like the big "Cmd" button for two-handed interaction, and that could be implemented either in the image (see touchCmdMorph attached) or in the VM. For SqueakJS I'd do it in the VM because one goal for SqueakJS is to be able to run as many unmodified images as possible. But that means until I get around to implement it, there is no way to do "Cmd-p" using the keyboard like you saw in the iPad video. I can only invoke print-it using the menu.

In SqueakJS there is an explicit button for showing / hiding the keyboard. I always planned to add IMM plugin support to automate showing/hiding the keyboard but never got around to do it.

Another thought was to invoke the character recognizer that is still in the image. At some point its cmd-r key binding got re-used though, so I don't think there is a way to invoke it now.

- When you were working on this, did you have any thoughts about
deficiencies of the Morphic model wrt multitouch/gestures/etc., and then
did you have any ideas about what to change and how to change it?

If I did, I don't recall those thoughts now.

Agreeing on a multi-touch event format for VM events would be a good step forward—the one John McIntosh did for the iOS VM is pretty much a 1-to-1 mapping of the iOS API. I'd prefer something more platform-neutral, like mimicking the Web's Pointer Event API https://developer.mozilla.org/en-US/docs/Web/API/Pointer_events (or the Touch Events API, but I like the unified API for all kinds of pointing devices).

I'm excited you're working on this!

- Vanessa -

2596touchCmdMorph-bf.cs (2K) Download Attachment

2595touchEvents-bf.cs (14K) Download Attachment

codefrau

Re: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)

In reply to this post by Karl Ramberg

Ha, talk about a trip down memory lane! I had no idea that still existed :)

- Vanessa -

On Sat, Sep 12, 2020 at 5:42 AM karl ramberg <[hidden email]> wrote:

There is also the Frank stuff which probably have some of the same multi cursor funktionality:
http://tinlizzie.org/~bert/frank4ipad/

One can download the Frank.ipa and unzip to look at the image.

Best,
Karl

On Fri, Sep 11, 2020 at 5:04 PM Tony Garnock-Jones <[hidden email]> wrote:
Hi Vanessa,

On 9/10/20 10:23 PM, Vanessa Freudenberg wrote:
> Oh, exciting! Please record a multi-hand Etoys demo. Maybe like this:
> https://youtu.be/gYrp31fH-Jk?t=56

Whoa, awesome! I didn't know this existed!

I have questions!

- How much of that shift-to-get-other-colour-buttons support is still
in the image? So far I haven't touched the default world menu bar, which
is mildly inappropriate for small and/or multitouch devices.

- What other ideas about mapping multitouch to Morphic interaction are
out there I have missed and am in danger of reinventing?

Also, I am intrigued by the idea of simply having each touch be mapped
to a Hand. It seems like the obvious way to do things, on one level, but
doesn't quite fit my fuzzy intuitions for how (multi-finger) gesture
recognition might work on another level. So:

- When you were working on this, did you have any thoughts about
deficiencies of the Morphic model wrt multitouch/gestures/etc., and then
did you have any ideas about what to change and how to change it?

I'll try to cook up a little demo like that sometime soon.

Tony

Beckmann, Tom

Re: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)

Hi everyone,

just wanted to mention that I experimented with bringing XInput2 [4] support to the X11 plugin for Squeak. With this, we get smooth (pixel perfect) scrolling and multitouch support. The code is lacking the device management part, so it may not work correctly when new devices are attached, for example.

Some pointers to code:
- Here are the changes to the X11 event handling loop [1]
- For the touch event tuple [2], I stayed rather close to the info XI2 provides, meaning you get the x,y coordinate, a sequence number that identifies a continuous gesture in the stream of events, and a touch type (begin/end/update/cancel). In terms of the range of information (not its structure) this also mirrors what you get in a browser if we omit experimental features such as radius of the touch area [3].

On the image side, I just simply added new #touch[Begin,Update,Update,Cancel]: callbacks to Morph, to which I forwarded the deconstructed event tuples as TouchEvent objects for demonstration purposes. Nothing fancy there just yet. I agree that a concept of multiple hands will likely provide a better abstraction.

Hope this can be of some interest :)

Best,
Tom

[1] https://github.com/tom95/opensmalltalk-vm/blob/xi-experiment/platforms/unix/vm-display-X11/sqUnixX11.c#L3638
[2] https://github.com/tom95/opensmalltalk-vm/blob/xi-experiment/platforms/unix/vm/sqUnixEvent.c#L188
[3] https://developer.mozilla.org/en-US/docs/Web/API/Touch
[4] https://www.x.org/releases/X11R7.7/doc/inputproto/XI2proto.txt
________________________________________
From: Squeak-dev <[hidden email]> on behalf of Vanessa Freudenberg <[hidden email]>
Sent: Saturday, September 12, 2020 11:25:24 PM
To: The general-purpose Squeak developers list
Subject: Re: [squeak-dev] Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)

Ha, talk about a trip down memory lane! I had no idea that still existed :)

- Vanessa -

On Sat, Sep 12, 2020 at 5:42 AM karl ramberg <[hidden email]<mailto:[hidden email]>> wrote:
There is also the Frank stuff which probably have some of the same multi cursor funktionality:
http://tinlizzie.org/~bert/frank4ipad/

One can download the Frank.ipa and unzip to look at the image.

Best,
Karl

On Fri, Sep 11, 2020 at 5:04 PM Tony Garnock-Jones <[hidden email]<mailto:[hidden email]>> wrote:
Hi Vanessa,

On 9/10/20 10:23 PM, Vanessa Freudenberg wrote:
> Oh, exciting! Please record a multi-hand Etoys demo. Maybe like this:
> https://youtu.be/gYrp31fH-Jk?t=56

Whoa, awesome! I didn't know this existed!

I have questions!

- How much of that shift-to-get-other-colour-buttons support is still
in the image? So far I haven't touched the default world menu bar, which
is mildly inappropriate for small and/or multitouch devices.

- What other ideas about mapping multitouch to Morphic interaction are
out there I have missed and am in danger of reinventing?

Also, I am intrigued by the idea of simply having each touch be mapped
to a Hand. It seems like the obvious way to do things, on one level, but
doesn't quite fit my fuzzy intuitions for how (multi-finger) gesture
recognition might work on another level. So:

- When you were working on this, did you have any thoughts about
deficiencies of the Morphic model wrt multitouch/gestures/etc., and then
did you have any ideas about what to change and how to change it?

I'll try to cook up a little demo like that sometime soon.

Tony

timrowledge

Re: Squeak on a cellphone responds to touchscreen input now (was Re: Squeak on a PostmarketOS cellphone)

> On 2020-09-13, at 8:22 AM, Beckmann, Tom <[hidden email]> wrote:
>
> Hi everyone,
>
> just wanted to mention that I experimented with bringing XInput2 [4] support to the X11 plugin for Squeak. With this, we get smooth (pixel perfect) scrolling and multitouch support.

Excellent news!

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: IA: Illogical And