Smalltalk › Squeak › Squeak - Dev

Dynabook and interactive content

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

6 messages Options

HilaireFernandes

Dynabook and interactive content

Some thoughts on Dynabook and interactive content design by programming
http://blog.drgeo.eu/post/2019/Dynabook-user-interface-point-of-view

I am interested on opinions and related readings on the subject.

Thanks

Hilaire

--
Dr. Geo
http://drgeo.eu

http://drgeo.eu

Ron Teitelbaum

Re: Dynabook and interactive content

Hi Hilaire,

Thanks for the post. Very interesting reading. There is a third option between dense GUI and code editing.

You mentioned an assistant but I think there is the potential to have an advanced agent that builds interactive diagrams based on user input. As an example drawing a large cross builds a cartesian coordinate. Pulling arrows changes the scale, drawing a circle creates a circle, pulling the edge scales the circle, drawing a line staps to grid. Adding a Point creates a point. Typing a letter labels the point.

Having this interaction create code that can be edited and maybe also show similar templates available would also be very useful. As we get better at recognizing user intention [1] using those techniques to enhance user computer interaction should enable us to design much easier interfaces for teachers.

1. https://www.autodraw.com/ something like this. Draw an image and select a match from the top to change your image.

All the best,

Ron Teitelbaum

On Sat, Jan 12, 2019 at 6:46 AM Hilaire <[hidden email]> wrote:

Some thoughts on Dynabook and interactive content design by programming
http://blog.drgeo.eu/post/2019/Dynabook-user-interface-point-of-view

I am interested on opinions and related readings on the subject.

Thanks

Hilaire

--
Dr. Geo
http://drgeo.eu

HilaireFernandes

Re: Dynabook and interactive content

Hi Ron, thanks for your feedbacks

In 2005, when I ported Dr. Geo from C++ to Squeak, I investigated shapes
gesture recognition with the help of the Genie package. As you wrote,
you can recognize drawn shapes as circle and line, move the canvas in
one or another direction. Gesture was not based on neural network layer,
I don't think it can recognize shapes as shown with autodraw.

However in the context of Dr. Geo, most of the time you will not
instantiate a circle out of nothing, so it will mean to recognize
additional gestures to designate the center (a free point, a point on a
line, or on two lines) and another point on the circle, or a value as
the radius, or a segment as radius base, etc.

All perfectly doable, it will require the user to learn the gestures and
to have the ad-hoc pen based interface, though. I found these two
situations not likely to happen: user are mainly casual user (kids at
school) operating with a mouse driven computer. Another issue I found:
the gestures are implicit, the user has no way to discover the gestures
without going through a tutorial. In an iconic UI, you convey explicit
information with tooltips and toolbar messages when the user specifies
he wants to draw a circle. Ok, you can go through this process with
gesture based UI, with a kind of hybrid UI. Anyway I did not go very far
although the idea was appealing.

Now you wrote about gesture to create code or to retrieve a template
from a corpus; for example drawing a map will retrieve a code template
to create interactive geographic map, drawing a time line will retrieve
a template code to create an interactive temporal/chronological line,
drawing a triangle will retrieve a template code to create an
interactive geometric sketch, drawing a speaker/wave will retrieve a
template code to create a sound recorder/player and so on.

Was it what you meant?

Hilaire

Le 12/01/2019 à 21:27, Ron Teitelbaum a écrit :

> Thanks for the post. Very interesting reading. There is a third
> option between dense GUI and code editing.
>
> You mentioned an assistant but I think there is the potential to have
> an advanced agent that builds interactive diagrams based on user
> input. As an example drawing a large cross builds a
> cartesian coordinate. Pulling arrows changes the scale, drawing a
> circle creates a circle, pulling the edge scales the circle, drawing a
> line staps to grid. Adding a Point creates a point. Typing a letter
> labels the point.
>
> Having this interaction create code that can be edited and maybe also
> show similar templates available would also be very useful. As we get
> better at recognizing user intention [1] using those techniques to
> enhance user computer interaction should enable us to design much
> easier interfaces for teachers.
>
> 1. https://www.autodraw.com/ something like this. Draw an image and
> select a match from the top to change your image.
>

--
Dr. Geo
http://drgeo.eu

http://drgeo.eu

Ron Teitelbaum

Re: Dynabook and interactive content

Hi Hilaire,

Yes much like the interface for autodraw as you draw it selects the most likely template and displays it for you.

There are still ways to discover properties on an object using hover, highlighting, and snap. Getting closer to an arrowhead might highlight it, hovering over it would make it bigger, dragging it would extend a line. A slight distance away in the direction of the arrow exposes a rotation indicator that users can use to change the line's angle. Getting close to a circle might add an arrow from the center to show its radius, getting close to the arrow would highlight it and moving your mouse to the arrowhead would make it bigger and dragging it would extend the radius. Getting closer to the center would highlight the center point. Clicking it would establish the point on, clicking it again would turn it off.

I'm not saying this is a perfectly intuitive interface but you can make it more discoverable by clearly understanding what a user might want to do and understanding how the user might go about trying to add that functionality. (an arch drawn across two connected lines adds an angle marker). You could even have a menu of physical items to represent the cursor a ruler, a compass, a protractor that would change what properties are exposed based on what tool is used but still, the properties are more physically exposed directly on the drawing using hover, highlighting, and snap.

Having code that always displays and is modified as the user interacts would also be a very useful tool to allow users to identify changes and make adjustments not easily expressed physically.

All the best,

Ron Teitelbaum

On Thu, Jan 17, 2019 at 6:08 AM Hilaire <[hidden email]> wrote:

Hi Ron, thanks for your feedbacks

In 2005, when I ported Dr. Geo from C++ to Squeak, I investigated shapes
gesture recognition with the help of the Genie package. As you wrote,
you can recognize drawn shapes as circle and line, move the canvas in
one or another direction. Gesture was not based on neural network layer,
I don't think it can recognize shapes as shown with autodraw.

However in the context of Dr. Geo, most of the time you will not
instantiate a circle out of nothing, so it will mean to recognize
additional gestures to designate the center (a free point, a point on a
line, or on two lines) and another point on the circle, or a value as
the radius, or a segment as radius base, etc.

All perfectly doable, it will require the user to learn the gestures and
to have the ad-hoc pen based interface, though. I found these two
situations not likely to happen: user are mainly casual user (kids at
school) operating with a mouse driven computer. Another issue I found:
the gestures are implicit, the user has no way to discover the gestures
without going through a tutorial. In an iconic UI, you convey explicit
information with tooltips and toolbar messages when the user specifies
he wants to draw a circle. Ok, you can go through this process with
gesture based UI, with a kind of hybrid UI. Anyway I did not go very far
although the idea was appealing.

Now you wrote about gesture to create code or to retrieve a template
from a corpus; for example drawing a map will retrieve a code template
to create interactive geographic map, drawing a time line will retrieve
a template code to create an interactive temporal/chronological line,
drawing a triangle will retrieve a template code to create an
interactive geometric sketch, drawing a speaker/wave will retrieve a
template code to create a sound recorder/player and so on.

Was it what you meant?

Hilaire

Le 12/01/2019 à 21:27, Ron Teitelbaum a écrit :
> Thanks for the post. Very interesting reading. There is a third
> option between dense GUI and code editing.
>
> You mentioned an assistant but I think there is the potential to have
> an advanced agent that builds interactive diagrams based on user
> input. As an example drawing a large cross builds a
> cartesian coordinate. Pulling arrows changes the scale, drawing a
> circle creates a circle, pulling the edge scales the circle, drawing a
> line staps to grid. Adding a Point creates a point. Typing a letter
> labels the point.
>
> Having this interaction create code that can be edited and maybe also
> show similar templates available would also be very useful. As we get
> better at recognizing user intention [1] using those techniques to
> enhance user computer interaction should enable us to design much
> easier interfaces for teachers.
>
> 1. https://www.autodraw.com/ something like this. Draw an image and
> select a match from the top to change your image.
>
--
Dr. Geo
http://drgeo.eu

Karl Ramberg

Re: Dynabook and interactive content

In reply to this post by HilaireFernandes

Have you seen Jupyter.org ? I have briefly looked at it, but is seem to be quite nice.

Best,

Karl

On Sat, Jan 12, 2019 at 12:47 PM Hilaire <[hidden email]> wrote:

Some thoughts on Dynabook and interactive content design by programming
http://blog.drgeo.eu/post/2019/Dynabook-user-interface-point-of-view

I am interested on opinions and related readings on the subject.

Thanks

Hilaire

--
Dr. Geo
http://drgeo.eu

HilaireFernandes

Re: Dynabook and interactive content

Yes, that's the idea. Looks nice.

Le 17/01/2019 à 20:08, karl ramberg a écrit :
> Have you seen Jupyter.org ? I have briefly looked at it, but is seem
> to be quite nice.
>
> Best,
> Karl

--
Dr. Geo
http://drgeo.eu

http://drgeo.eu