Mouseover on a touch screen

For many years now I have created artistic content that makes intensive use
of mouseovers. About a year ago I got a Nokia N810, and played a bit with
Squeak on that device. I've still not recovered from the shock of realizing
that on a touch screen, there is no such thing as mouseover!: you cannot
get the mouse cursor to move without putting it down. What to do.

I can think of two potential approaches to this problem.

1. Subclass or modify HandMorph to "do the right thing in the right
circumstances". Clearly some work has to be done to clarify how this should
behave. HandMorph seems quite complex -- I don't suppose someone has
written a tutorial on it somewhere?

2. A new class which acts as a mouse event proxy. This could either be a
"screen" -- something that would fit over an entire PasteUpMorph -- or a
"sled" that would be carried around by a HandMorph.

In any case, the obvious question is: how should it behave. I haven't
really specified this yet, but a rough idea would be something like this:
"it" absorbs mouseDown, and records what morph would receive mouseDown if
it weren't there. If a mouseUp occurs that would have been given to the
same morph then mouse events are replayed and sent to that morph.
Otherwise, "it" has to relay mouse events to whatever underlying morph
would have gotten them if it were not there. This may not be simple to
achieve.

Now I've only poked around a little, and it looks to me as though the code
for deciding what morph should get an event is in HandMorph. So, if I'm
serious about doing something, regardless of how I do this it looks like I
will need to get deep into the workings of HandMorph.

I've tended to want to do my custom stuff in Squeak by subclassing, and am
inclined to do approach 2 above -- at this point I kind of like the idea of
a "screen". Am I reinventing a wheel here? If someone knows of work I
should look at, I'd appreciate any references.

There's a major psychological difference between a mouseOver and a click.
The click is a kind of commitment, which a mouseOver isn't. It would be a
real shame to have to say that we just have to give up mouseOvers on touch
screens. I'm not prepared to do that.

-Thanks, Jim

---
 Jim Rosenberg                      http://www.well.com/user/jer/
     Internet: [hidden email]

ST-80 was predicated on a particular set of UI metaphors. Touch
screens (especially "multi-touch") handily disregard some of these
metaphors. It will be interesting to see what we do to accommodate
these new input devices.

I'm sure I'm not the only person here that thinks Squeak/Morphic on a
tablet could be fantastic:)


On Thu, Dec 24, 2009 at 11:19 AM, Jim Rosenberg <[hidden email]> wrote:


--
Ron

--On Thursday, December 24, 2009 14:19:35 -0500 I wrote:

> For many years now I have created artistic content that makes intensive
> use of mouseovers. About a year ago I got a Nokia N810, and played a bit
> with Squeak on that device. I've still not recovered from the shock of
> realizing that on a touch screen, there is no such thing as mouseover!:
> you cannot get the mouse cursor to move without putting it down. What to
> do.

I've made some progress at understanding how I think this should work, but
am still deep in the swamp at understanding how to accomplish it. Here is
my idea, in the simplest possible terms:

If there is a mouseDown event, and a morph is available that should handle
it, there is no change to how Morphic event handling works. However: if no
handling morph is available, then the event system should not change state
and should act as though the event never occurred.

I'm having some trouble following how "state" is maintained. There is
mouseFocus, of course. Is there more to mouse state than this? Where should
I look to find where state is changed when the mouse goes down?

-Thanks, Jim

---
 Jim Rosenberg                      http://www.well.com/user/jer/
     Internet: [hidden email]

I wrote:

> I've still not recovered from the shock of realizing that on a touch
> screen, there is no such thing as mouseover!: you cannot get the mouse
> cursor to move without putting it down. What to do.

OK, I have a solution, which works fine for me, though it isn't necessarily
generic. The mouseovers I need to preserve on a touch screen are all in my
own code. So, all I have to do is add handlesMouseOverDragging:,
mouseEnterDragging:, and mouseLeaveDragging: to the appropriate classes and
have it just send the non-dragging messages to self. This is so simple it's
a bit embarrassing. I've tried it, and it works!

---
 Jim Rosenberg                      http://www.well.com/user/jer/
     Internet: [hidden email]

--On Wednesday, December 30, 2009 22:40:36 -0500 I wrote:

> OK, I have a solution, which works fine for me, though it isn't
> necessarily generic. The mouseovers I need to preserve on a touch screen
> are all in my own code. So, all I have to do is add
> handlesMouseOverDragging:, mouseEnterDragging:, and mouseLeaveDragging:
> to the appropriate classes and have it just send the non-dragging
> messages to self. This is so simple it's a bit embarrassing. I've tried
> it, and it works!

For the record, if someone else happens to be following down this path, I
need to amplify this. There is some extra secret sauce you need besides
using the *Dragging: methods. You *do* have to intervene in the normal
mouseDown process to "turn it off" when a morph does not really need to
handle it. I had implemented this in my test class, and it wasn't working
until I implemented the *Dragging: methods, so I hadn't realized this was
also necessary until I tried to get it all working in a "normal"
PasteUpMorph.

Below is the hack I put in my subclass of PasteUpMorph. Any morph that
would normally handle mouseDown that needs to ignore it when the screen is
touched for the mere purpose of moving the mouse -- i.e. for an
"exploratory drag" -- needs to "reset" mouse state as at the end of the
method below.

*DISCLAIMER*: This all seems to be working for me, but I am still a newbie
at the internals of hand and mouse event processing. There is a very good
chance this method is not sufficiently general, and there's some subtlety
somewhere I'm overlooking. It's pretty easy to think you've got things
working, and then find there's a gap when the mouse is dragged over some
morph you haven't thought of and then a new "exploratory drag" is started.
This all is very much a cheap and dirty hack, and the problem really should
be thought through at a fundamental level by someone who really knows the
event processing code extremely well.

UI design is full of assumptions that are stood on their head on a touch
screen. Mouse-down, dragging -- these are taken for granted in UI design as
deeply *intentional* acts. You tend not to think about the implications of
"exploratory drag" until you actually have a touch screen in your hands.

Ideally, there would be some central Squeak preference that governs how all
this works.

----

mouseDown: anEvent
        "Touch screen behavior hack: if a submorph needs to respond to the event
as a button,
        keep normal behavior. Otherwise reset hand for 'no mouse down' behavior"

        self submorphs do: [:m|
                (m containsPoint: anEvent cursorPoint event: anEvent) ifTrue: [
                        (m respondsTo: #handlesMouseUp) ifTrue: [
                                (m handlesMouseUp: anEvent) ifTrue: [
                                        ^ super mouseDown: anEvent
                                ]
                        ]
                ]
        ].
        anEvent hand
                newMouseFocus: nil;
                resetClickState.


---
 Jim Rosenberg                      http://www.well.com/user/jer/
     Internet: [hidden email]