We need to talk about graphics

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

We need to talk about graphics

timrowledge
Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.

We had Balloon2D & 3D.
We had 'Pooh graphics' to do vector forms.
We have some excellent stuff being done by the Cuis crew.
We have some amazing ideas from Nile/Gezira.
We have a number of potential 3rd party graphics libraries we might adopt.
We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above.
We probably have other options I've not even heard of.

Maybe we should actually do something? Can we at least talk about it?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: SVE: Skip on Vernal Equinox



Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

Jecel Assumpcao Jr
Tim Rowledge wrote on Sun, 28 Jun 2020 17:53:21 -0700
> Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.
>
> We had Balloon2D & 3D.
> We had 'Pooh graphics' to do vector forms.
> We have some excellent stuff being done by the Cuis crew.
> We have some amazing ideas from Nile/Gezira.
> We have a number of potential 3rd party graphics libraries we might adopt.
> We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above.
> We probably have other options I've not even heard of.

You didn't mention the original BitBlt and WarpBlit, which is
understandable if your focus is on vector graphics.

> Maybe we should actually do something? Can we at least talk about it?

Though the survey I did for the Board pointed to OpenGL ES as our best
option (at the time, things have changed since), it is not something I
personally like. I want something that is not a black box so I can
single step from the highest level to individual pixels when debugging
issues. But I also want it to take advantage of all available resources
(including GPUs, extra processors) by translating debugged code into
something else. I just don't want to debug in that something else, which
would be the case for OpenGL ES shaders.

Nile/Gezira + JIT would be my vote, but it has been planned for a long
time and, as far as I know, is not happening right now.

-- Jecel

Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

darth-cheney
Hi everyone,

I am of particularly low expertise in these matters, but in the grand tradition of online discourse I thought I'd chime in anyway.
 
Tim Rowledge wrote on Sun, 28 Jun 2020 17:53:21 -0700
> Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.
>
> We had Balloon2D & 3D.
> We had 'Pooh graphics' to do vector forms.
> We have some excellent stuff being done by the Cuis crew.
> We have some amazing ideas from Nile/Gezira.
> We have a number of potential 3rd party graphics libraries we might adopt.
> We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above.
> We probably have other options I've not even heard of.

Not sure if everyone keeps up with the guys on the other side of the tracks, but the Bloc library in Pharo took the approach of using Skia (originally it used Moz2D). A lot of work went into it, and a year or two ago I was able to get some interesting things up in my own experiments. This work is now no longer being done on the mainline branch, but instead as part of the Feenk GToolkit project. I'm not sure if or when the Pharo people will pick it up for incorporation.
  
When talking about incorporating 3rd party libraries, do we mean as VM plugins or through FFI? Is there some advantage of one over the other? On an unrelated thread (in which I posted off-topic, shame) I mentioned that Vulkan provides a complete XML description of its API, which, as far as I'm able to understand these things, means the enormous pain of creating FFI bindings is somewhat ameliorated here. That said, I don't really know anything about Vulkan other than the fact it's supposed to be the "next generation" graphics api from the same producers that brought us OpenGL and OpenGL 2: The Reckoning.

On Sun, Jun 28, 2020 at 11:01 PM Jecel Assumpcao Jr <[hidden email]> wrote:
 
Nile/Gezira + JIT would be my vote, but it has been planned for a long
time and, as far as I know, is not happening right now.

Nile seems like a fascinating option and I've tried to read as much about it as I can. Unfortunately aside from the various posted talks online and the VPRI papers, there's not a lot of information about using it (unless I am missing something -- you can track a github issue about its status here). My guess is that the Gezira part, of which there is even less information, is very closely tied to the specific Frank environment in which it was used -- is that correct? Using plain Squeak how would one even incorporate Nile?

Though the survey I did for the Board pointed to OpenGL ES as our best
option (at the time, things have changed since), it is not something I
personally like. I want something that is not a black box so I can
single step from the highest level to individual pixels when debugging
issues. But I also want it to take advantage of all available resources
(including GPUs, extra processors) by translating debugged code into
something else. I just don't want to debug in that something else, which
would be the case for OpenGL ES shaders.

One thing about Squeak and friends that is very nice is the way it can abstract over essential "kinds" of activity in the computer. For example, the global `Processor` class manages everything I really need from my actual computer's processor: scheduling different things to run and then doing so, while providing me a no-nonsense interface into itself. The current BitBlt graphics stack is also a sweet-spot abstraction for the kinds of graphics it's working with. Once you dig down and get the hang of bits, forms, and canvases, it all comes together in a delightfully lively way. I see what Jecel is saying here and I think it's a worthy goal -- what would be the equivalent of the `Processor` global for the GPU (even in cases where the actual machine doesn't have a "real" GPU)?
--
Eric


Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

David T. Lewis
In reply to this post by Jecel Assumpcao Jr

Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

David Duke
In reply to this post by darth-cheney
A setof thoughts from'offscreen'.  I think a useful starting point is whatis  graphics for? It
has a big bearing 2/3D is an obvious distinction. However.
Is GPU& shader  support necessary/desirable: 
yes if real-time rendering for thighs like games/VR are important, but in that case there 
is a bigger problem jitter caused by GC,
no if the main driver is application like a GUI. visualisation.
Looking at the history of Smalltalk and graphics in ST the best results have been where there
has been parsimony between the drover, ST the language & ST the platform:
-BitBlt which was driven by the aims  of the LRG  and early applications. what is the vision today?
-Moose/Pharao driven by data vis applications
-Croquet driven by a vision of collaborative VR
Trying to be all things to all users might be a nice vision but the result may be unusable by any one
constituency.
Other issues:
what about input: morphic, tea-time, or something akin to FRP?
what about other Media(sound, video)

I like theidea of not having a blackboc so that you canunderstand/ the process, Balloon was a nice idea.
However I found it difficult to pick up as the documentation was sparse,
Other platforms to consider for inspiration:
- OpenGL was not the first graphics standard, There was also PHIGS and GKS, whilst dated these had some useful ideas,
e.g. PHIGS's structures and the ability to edit these. Their input handling via device abstractions was an improvement on their successors

- VTK The 'Visualization Tool kit' is an Object-Oriented Approach to3 graphics that has had some success in the (scientific)visualisation  community
If you want to also support other media, then PREMO (Presentation Environment for Multimedia Objects) might have some ideas that are worth looking at,
It is anISO/SC24 Standard that was designed with OOP in mind as the baseline for implementation( arguably premature). I'm familiar with  it as I worked 
on it for some years. Again comes back to that question of what is the goal and use cases?

The Nile work looks cool, would be nice to incorporate it or at least leave the door open.

Background. An occassional Squeak user. I did once try to use squeak for vis, explorong  VTK-like system in Squeak.
However I backed out in part due to lack of documentation, and not the  right platform community for the community I was working with.
Over the last two decades I've spent time working largely in pure FP (Haskell) doing large-scale vis.


I'd be interested in tracking any effort and possibly contributing.

--David





 

 




 


On Tue, Jun 30, 2020 at 12:15 AM Eric Gade <[hidden email]> wrote:
Hi everyone,

I am of particularly low expertise in these matters, but in the grand tradition of online discourse I thought I'd chime in anyway.
 
Tim Rowledge wrote on Sun, 28 Jun 2020 17:53:21 -0700
> Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.
>
> We had Balloon2D & 3D.
> We had 'Pooh graphics' to do vector forms.
> We have some excellent stuff being done by the Cuis crew.
> We have some amazing ideas from Nile/Gezira.
> We have a number of potential 3rd party graphics libraries we might adopt.
> We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above.
> We probably have other options I've not even heard of.

Not sure if everyone keeps up with the guys on the other side of the tracks, but the Bloc library in Pharo took the approach of using Skia (originally it used Moz2D). A lot of work went into it, and a year or two ago I was able to get some interesting things up in my own experiments. This work is now no longer being done on the mainline branch, but instead as part of the Feenk GToolkit project. I'm not sure if or when the Pharo people will pick it up for incorporation.
  
When talking about incorporating 3rd party libraries, do we mean as VM plugins or through FFI? Is there some advantage of one over the other? On an unrelated thread (in which I posted off-topic, shame) I mentioned that Vulkan provides a complete XML description of its API, which, as far as I'm able to understand these things, means the enormous pain of creating FFI bindings is somewhat ameliorated here. That said, I don't really know anything about Vulkan other than the fact it's supposed to be the "next generation" graphics api from the same producers that brought us OpenGL and OpenGL 2: The Reckoning.

On Sun, Jun 28, 2020 at 11:01 PM Jecel Assumpcao Jr <[hidden email]> wrote:
 
Nile/Gezira + JIT would be my vote, but it has been planned for a long
time and, as far as I know, is not happening right now.

Nile seems like a fascinating option and I've tried to read as much about it as I can. Unfortunately aside from the various posted talks online and the VPRI papers, there's not a lot of information about using it (unless I am missing something -- you can track a github issue about its status here). My guess is that the Gezira part, of which there is even less information, is very closely tied to the specific Frank environment in which it was used -- is that correct? Using plain Squeak how would one even incorporate Nile?

Though the survey I did for the Board pointed to OpenGL ES as our best
option (at the time, things have changed since), it is not something I
personally like. I want something that is not a black box so I can
single step from the highest level to individual pixels when debugging
issues. But I also want it to take advantage of all available resources
(including GPUs, extra processors) by translating debugged code into
something else. I just don't want to debug in that something else, which
would be the case for OpenGL ES shaders.

One thing about Squeak and friends that is very nice is the way it can abstract over essential "kinds" of activity in the computer. For example, the global `Processor` class manages everything I really need from my actual computer's processor: scheduling different things to run and then doing so, while providing me a no-nonsense interface into itself. The current BitBlt graphics stack is also a sweet-spot abstraction for the kinds of graphics it's working with. Once you dig down and get the hang of bits, forms, and canvases, it all comes together in a delightfully lively way. I see what Jecel is saying here and I think it's a worthy goal -- what would be the equivalent of the `Processor` global for the GPU (even in cases where the actual machine doesn't have a "real" GPU)?
--
Eric



--
David Duke
Professor of Computer Science
School of Computing University of Leeds UK


Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

Stéphane Rollandin
> Is GPU& shader  support necessary/desirable:
> yes if real-time rendering for thighs like games/VR are important, but
> in that case there
> is a bigger problem jitter caused by GC,
In the image I just uploaded two days ago and that one can get from
http://www.zogotounga.net/comp/squeak/rogue.htm the main feature is a 2D
real-time top-down game.

I found that the GC-caused jitter was indeed a problem, but also that
the main culprit was the time it takes to simply paint an image which is
a surprisingly (to me) slow operation.

That, plus the alpha-blending issue that was discussed on the VM list a
little ago, where Juan proposed two new primitives.

Stef

Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

Karl Ramberg
In reply to this post by Jecel Assumpcao Jr


On Mon, Jun 29, 2020 at 5:01 AM Jecel Assumpcao Jr <[hidden email]> wrote:
Tim Rowledge wrote on Sun, 28 Jun 2020 17:53:21 -0700
> Seriously. We've been sitting around twiddling thumbs about better graphics for decades now.
>
> We had Balloon2D & 3D.
> We had 'Pooh graphics' to do vector forms.
> We have some excellent stuff being done by the Cuis crew.
> We have some amazing ideas from Nile/Gezira.
> We have a number of potential 3rd party graphics libraries we might adopt.
> We have an advanced JIT that could be used to make on-demand graphics routines either on their own or working with pretty much any of the above.
> We probably have other options I've not even heard of.

You didn't mention the original BitBlt and WarpBlit, which is
understandable if your focus is on vector graphics.

> Maybe we should actually do something? Can we at least talk about it?

Though the survey I did for the Board pointed to OpenGL ES as our best
option (at the time, things have changed since), it is not something I
personally like. I want something that is not a black box so I can
single step from the highest level to individual pixels when debugging
issues.
 
I have the same  thoughts on this.
Faster, smoother graphics would be really good.
At the same time I like access to everything within the image.




Best,
Karl

 
But I also want it to take advantage of all available resources
(including GPUs, extra processors) by translating debugged code into
something else. I just don't want to debug in that something else, which
would be the case for OpenGL ES shaders.

Nile/Gezira + JIT would be my vote, but it has been planned for a long
time and, as far as I know, is not happening right now.

-- Jecel



Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

Stéphane Rollandin
>     Though the survey I did for the Board pointed to OpenGL ES as our best
>     option (at the time, things have changed since), it is not something I
>     personally like. I want something that is not a black box so I can
>     single step from the highest level to individual pixels when debugging
>     issues.
>
> I have the same  thoughts on this.
> Faster, smoother graphics would be really good.
> At the same time I like access to everything within the image.

+1

Stef

Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

Peter Crowther-2
In reply to this post by Karl Ramberg
On Thu, 2 Jul 2020 at 09:22, karl ramberg <[hidden email]> wrote:
Faster, smoother graphics would be really good.
At the same time I like access to everything within the image.

If you could only choose one of these, which would you choose? "Access to everything within the image" is a really large anchor to attach to a graphics subsystem these days.  We have a single-threaded Smalltalk that has deep assumptions about a single memory space and a single rectangular display area of consistent properties wired in over forty years of development.  Meanwhile, modern graphics subsystems are *very* heavily parallel, use custom hardware, and you try to offload everything you can to the coprocessor because you don't want it even in your memory space.

If we want a teaching and learning system that can be understood and modified by a normal programmer, we keep everything in the image and accept limited facilities and slowness.  If we want something that makes use of all of the varied and wonderful capabilities of modern graphics hardware then a) what's the smallest subset we'll accept (to make it easy to port Squeak to new platforms), b) what knowledge do we require of anyone who wishes to investigate the graphics subsystem?

Cheers,

- Peter


Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

Stéphane Rollandin
> If you could only choose one of these, which would you choose?

If new modern, faster, but opaque graphics system is implemented, then
it does not have to replace the existing, fully accessible one, it can
just complement it (being available via a specific Canvas subclass, and
able to deliver a Form on demand). There is no need to have an
opposition here. Just as we have Morphic *and* MVC, we can have BitBlt
*and* a bridge to an opaque backend for people interested in that kind
of performance.

As for me, I am interested in both.

Stef

Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

HilaireFernandes
In reply to this post by Stéphane Rollandin


Le 30/06/2020 à 08:26, Stéphane Rollandin a écrit :
Is GPU& shader  support necessary/desirable:
yes if real-time rendering for thighs like games/VR are important, but in that case there
is a bigger problem jitter caused by GC,
In the image I just uploaded two days ago and that one can get from http://www.zogotounga.net/comp/squeak/rogue.htm the main feature is a 2D real-time top-down game.

I found that the GC-caused jitter was indeed a problem, but also that the main culprit was the time it takes to simply paint an image which is a surprisingly (to me) slow operation.

That, plus the alpha-blending issue that was discussed on the VM list a little ago, where Juan proposed two new primitives.

Juan proposed these primitives to speed up the rendering of VectorGraphics Widget Morph.

Regarding graphics, Juan's work on Vector Graphics can be of interest for Squeak.

Hilaire

-- 
Dr. Geo
http://drgeo.eu
https://pouet.chapril.org/@hilaire


Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

Karl Ramberg
In reply to this post by Peter Crowther-2
Hi,
I have tried to use the OpenGL from Squeak and found it very opaque and hard to use.
Both because the Squeak side is not very well documented and because OpenGL is hard  itself.

I have also looked briefly at Dolphin Smalltalk which have quite fast graphics on Windows but
the way you get it to work is quite elaborate and convoluted.


I agree with what Stephane Rollandin wrote; keep BitBlt for simplicity and work on better way to
access whatever we choose to use as a fast backend.

Best,
Karl

On Fri, Jul 3, 2020 at 5:11 PM Peter Crowther <[hidden email]> wrote:
On Thu, 2 Jul 2020 at 09:22, karl ramberg <[hidden email]> wrote:
Faster, smoother graphics would be really good.
At the same time I like access to everything within the image.

If you could only choose one of these, which would you choose? "Access to everything within the image" is a really large anchor to attach to a graphics subsystem these days.  We have a single-threaded Smalltalk that has deep assumptions about a single memory space and a single rectangular display area of consistent properties wired in over forty years of development.  Meanwhile, modern graphics subsystems are *very* heavily parallel, use custom hardware, and you try to offload everything you can to the coprocessor because you don't want it even in your memory space.

If we want a teaching and learning system that can be understood and modified by a normal programmer, we keep everything in the image and accept limited facilities and slowness.  If we want something that makes use of all of the varied and wonderful capabilities of modern graphics hardware then a) what's the smallest subset we'll accept (to make it easy to port Squeak to new platforms), b) what knowledge do we require of anyone who wishes to investigate the graphics subsystem?

Cheers,

- Peter



Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

David T. Lewis
In reply to this post by Stéphane Rollandin
On Fri, Jul 03, 2020 at 05:58:58PM +0200, St??phane Rollandin wrote:

> >If you could only choose one of these, which would you choose?
>
> If new modern, faster, but opaque graphics system is implemented, then
> it does not have to replace the existing, fully accessible one, it can
> just complement it (being available via a specific Canvas subclass, and
> able to deliver a Form on demand). There is no need to have an
> opposition here. Just as we have Morphic *and* MVC, we can have BitBlt
> *and* a bridge to an opaque backend for people interested in that kind
> of performance.
>
> As for me, I am interested in both.
>

+1

Dave


Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

David T. Lewis
In reply to this post by HilaireFernandes
On Fri, Jul 03, 2020 at 06:15:17PM +0200, Hilaire wrote:

>
> Le 30/06/2020 ?? 08:26, St??phane Rollandin a ??crit??:
> >>Is GPU& shader ??support necessary/desirable:
> >>yes if real-time rendering for thighs??like games/VR are important,
> >>but in that case there
> >>is a bigger problem jitter caused by GC,
> >In the image I just uploaded two days ago and that one can get from
> >http://www.zogotounga.net/comp/squeak/rogue.htm the main feature is a
> >2D real-time top-down game.
> >
> >I found that the GC-caused jitter was indeed a problem, but also that
> >the main culprit was the time it takes to simply paint an image which
> >is a surprisingly (to me) slow operation.
> >
> >That, plus the alpha-blending issue that was discussed on the VM list
> >a little ago, where Juan proposed two new primitives.
>
> Juan proposed these primitives to speed up the rendering of
> VectorGraphics Widget Morph.

I remember a discussion about that, though I don't have a link to it. It
sounds like we need to follow up on this and make something happen.

>
> Regarding graphics, Juan's work on Vector Graphics can be of interest
> for Squeak.
>

Definitely yes. Juan is working towards *both* high quality rendering
*and* high performance. I think that there may be a perception that
Morphic 3 and VectorGraphics might sacrifice performance for rendering
quality, but I am not sure that is the case at all.

This recent post by Juan would be of interest to any of us attempting to
achieve graphics performance on modern hardware:

  https://lists.cuis.st/mailman/archives/cuis-dev/2020-June/001958.html

I also have to add that the recent work that you (Hilaire) have been
doing with DrGeo on Cuis really illustrates the potential. Your circular
menu will be of interest to anyone who has been involved with Morphic,
Etoys, and Scratch interfaces:

  https://mamot.fr/@drgeo/104310151994863180

Dave


Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

HilaireFernandes
Le 03/07/2020 à 22:21, David T. Lewis a écrit :
Juan proposed these primitives to speed up the rendering of 
VectorGraphics Widget Morph.
I remember a discussion about that, though I don't have a link to it. It
sounds like we need to follow up on this and make something happen.

Here it is:

https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/505

-- 
Dr. Geo
http://drgeo.eu
https://pouet.chapril.org/@hilaire


Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

timrowledge


> On 2020-07-03, at 1:32 PM, Hilaire <[hidden email]> wrote:
>
> Le 03/07/2020 à 22:21, David T. Lewis a écrit :
>>> Juan proposed these primitives to speed up the rendering of
>>> VectorGraphics Widget Morph.

> https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/505

I'm all for such improvements but I have no idea about updating the rather specialised ARM bitblt code that Ben Avison did for RPF a few years ago. That stuff is responsible for no small part of the really good UI performance we get on Pi.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
When all else fails, let a = 7.  If that doesn't help, then read the manual.



Reply | Threaded
Open this post in threaded view
|

Re: We need to talk about graphics

David T. Lewis
On Fri, Jul 03, 2020 at 03:08:28PM -0700, tim Rowledge wrote:

>
>
> > On 2020-07-03, at 1:32 PM, Hilaire <[hidden email]> wrote:
> >
> > Le 03/07/2020 ?? 22:21, David T. Lewis a ??crit :
> >>> Juan proposed these primitives to speed up the rendering of
> >>> VectorGraphics Widget Morph.
>
> > https://github.com/OpenSmalltalk/opensmalltalk-vm/issues/505
>
> I'm all for such improvements but I have no idea about updating the rather specialised ARM bitblt code that Ben Avison did for RPF a few years ago. That stuff is responsible for no small part of the really good UI performance we get on Pi.
>

Marcel added this comment to the issue on github:

> Would be nice to also have the Smalltalk fall-back code for those new rules. :-)

which seems important if some of the VMs may not yet implement the new rules.

Dave