The cooked and the raw [RE: Touch/MultiTouch Events]
Locked
 
|
My intuition is that some window systems will give cooked/composite gesture events, where with others we will need optional Smalltalk code or a plugin to recognize and compose gesture events. One thing that has bothered me for some time is the difficulty in explaining how users interact with input events and the amount of required cooperation agreed to between components. [E.g. drag 'n drop]. I think some of this is elegant ("I want her/him & she/he wants me") but what I am looking for is a way to express interest in pattern roles. I want to specify and recognize gesture patterns and object roles within each pattern. So match (composed) gesture to pattern within a sensitive area to get: open/close drag 'n drop (draggable=source, droppable=target; object-for-drag, object-for-drop) expand/collapse (maximize/minimize) grow/shrink (pinch, press+drag) rescale (out/in) rotate stretch/adjust reposition scroll (swipe) select (tap, double-tap, select+tap) The "same" gesture could map differently depending on the "sensitive area", e.g. open/close vs maximize/minimize; grow/shrink vs rescale vs stretch vs reposition. Sensitive areas could compose as with mouse sensitivity. Sensitivity & role(s) given to any morph. Redo pluggable buttons/menus/.. in new pattern. I know this is both a code and a cognitive change, but I think easier to explain = more comprehensible. I think it could be more compactly expressive. -KenD
-KenD
|
Re: The cooked and the raw [RE: Touch/MultiTouch Events]
|
Hi Ken, just commenting on the availability of gestures - do you know of any windowing systems that provide high level touch gestures? The systems I'm familiar with all defer this to a UI library. libinput, which you linked to in an earlier thread, also explains why they cannot, generally, provide high-level gestures on absolute touch devices [1]. They do provide gestures for touchpads, however. These could also be of interest, but I think that's a different event type altogether that should not mingle with touch events. Similarly, OS X appears to provide high-level gesture for touchpads/trackpads [2], but for their touchscreen APIs they appear to defer to UI libraries again [3] (these are just my takeaways from 5min of googling). Best, Tom On Sat, Sep 19, 2020 at 9:37 PM <[hidden email]> wrote:
|
|
|
In reply to this post by KenDickey
It's been a while, but when I was doing iOS and Android development, you typically got the raw events and sent them off to a handler depending on what you needed them to do. That's why I believe it should be in the image, not in the VM. It's all very context sensitive: whether a single finger dragging across the screen represents a swipe, scroll, drag or something else depends on context (not just events that came before and/or after the current event, but also the GUI object being manipulated, any interactions between GUI objects as well as the application using it/them) The VM simply doesn't have enough context to make an informed, or even reasonable, guess as to what a sequence of touch events mean at a higher level. The image does only in the sense that it has a set of use cases it anticipates for a limited set of widgets. For example, if I'm implementing a drawing application I may be very interested in the raw events to determine if the user is using a stylus or a finger (perhaps you want a stylus to represent a fine-tipped brush while the finger represents blobby finger painting etc. Does the user want multiple touches to represent higher level gestures or simply multiple simultaneous painting brushes? etc. etc.) The gestures in a drawing application are likely to be different, and more nuanced, than those for a general purpose GUI. Touch-based 3D apps are another example where you can pretty much throw out the rule book. Same with touch-based games. It may not even be specific to the application: let's say I wanted to have a transparent overlay cover the entire display that intercepts all touch events and, if the touch appears to be from a stylus (or some other means of relaying higher precision touches), route the events to a handwriting recognition system which then has to pass the recognized string to the widget underneath where the text was written. Yes, you can pretend that touches are mouse events with a handful of gestures... if you don't mind restricting yourself to very limited, lowest-common-denominator mobile UIs. I would agree that the image could and should provide some (optional) default behavior(s) to 'cook'/synthesize these events into higher level events that get passed to Morphs (or whatever other UI framework) that wants them. This would probably work very well for many use cases. Just implement it in such a way that it's easy to bypass / override so that at a widget/application/world/image level they can say 'no, just give me the raw events and I'll figure it out' or 'here, use my handler instead'. This also leaves the door open to someone else coming along and saying 'here's a better way to abstract this' without reinventing the (mouse)wheel with N buttons. Perhaps morphs could have a gestureHandler that, if present, takes care of things and optionally dispatches #swipeGestureEvent..., #dragGestureEvent... etc. for you. If a given morph doesn't provide a handler, use the default handler at the world level that deals with a majority of 'typical' use cases. Just thinking out loud. I can say from experience that trying to derive uncooked events (as I did for mouse wheel events in Cuis) is problematic both from an implementation and performance standpoint... and the code is ugly. Separating overcooked pasta is more fun. At the same time, dealing with raw events (as I tried for a period of time on Android before I wised up) for a 'typical' UI isn't normally the way you want to go either. So yes, you often want a handler to do this for you... but it needs to be easy to override/replace/bypass for when you hit the unusual use cases which are more common than you might imagine for touch. These aren't one-size-fits-all solutions... your application's need for cooked touch events may be very different from mine. To me, this screams 'do it in the image!' (optionally with a plugin for performance... down the road... maybe.[1]) [1] I think you're going to find that getting 64-bit ARM JIT support, some sort of process level multi-core support and taking advantage of the GPU are far more important if you want your image and application to feel like it's from this century on mobile devices. I've been trying to use Squeak/Cuis on mobile devices since the CogDroid days... it's not been a good experience due to these issues. (utilizing the GPU helps, but that alone often isn't sufficient) On Sat, Sep 19, 2020 at 3:37 PM <[hidden email]> wrote:
|
|
|
In reply to this post by KenDickey
Thanks much for the additional input! As I noted, I am coming up from zero on this and have many references to digest. To clarify, my thought is to have raw inputs packaged by the vm and have St code which does gesture recognition with initial states based on screen areas (morphs), parses the events, giving dynamic user feedback (morph and finger/stylus tip highlighting, finger/stylus trails), and following the state machines to do the recognition. I would like declarative gesture-pattern descriptions and a way to discover roles and bind objects to them so that when a recognition event is complete, the corresponding method is invoked with the roles as receiver and arguments. At this point I am still "drawing on clouds", which is prior to the "cast in jello" stage. Given the gestures in common use, I would like a naming and compact recognition pattern akin to enumerators for collections. What view of this gives the simplest, most comprehensible explanations? Still reading and cogitating. Thanks much for references and clarifying thoughts! -KenD
-KenD
|
Re: The cooked and the raw [RE: Touch/MultiTouch Events]
|
|
In reply to this post by Tom Beckmann
Am Sa., 19. Sept. 2020 um 22:07 Uhr schrieb Tom Beckmann <[hidden email]>: > > just commenting on the availability of gestures - do you know of any windowing systems that provide high level touch gestures? The systems I'm familiar with all defer this to a UI library. > Well, since you did not say cross-platform: let's not forget Windows. ;-D https://docs.microsoft.com/en-us/windows/win32/user-interaction (I have no coding experience with any of these though.) They seem to have come up with new APIs replacing the old ones a few times. Looking at them and comparing might help to not repeat mistakes others have made. But oftentimes you cannot use something in Squeak because it is just a single opaque window to Windows, not one window per control/widget, as in regular Win32 apps. Kind regards, Jakob |
«
Return to Squeak VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |