Squeak VM on iPad/IPhone gets open/GL rendering

> On friday night I created an open/GL renderer to work with Squeak screen updates on the iPhone/iPad.
>
> Unfortunately it runs slows than the tiled Core Animation render that I had work out earlier in the year
> with help from the Apple graphics engineers.
>
> So if anyone has a fair amount of knowledge about Open/GL ES maybe they could contact me and
> we'll see if we can improve things?
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>
>
>
>
>

> Well ok, let's fix the problem with no touch events with the iPad eToy changes then as a first step.
> To do this.
>
> Fetch the Platform source tree from squeakvm.org
> Visit the iOS folder and find the SqueakPureObjc.xcodeproj
> double click that to open in xcode.
> Select the SqueakNoOGLIPhone project and build as debug against an iPad simulator.
>
> (a) At this point it should compile/link/install/run and you'll get a Pharo desktop. If not
> then I'm missing something in the platform tree.
>
> (b) Oh note you might get an ERROR because you have to choose between the CALayer and Open/GL
> render class and I wanted to ensure developers understood there was a choice to be made. Edit it to choose
> the CALayer one since it's faster and is the same logic as you find in the OS-X post 4.x and pre 5.8
>
>
> Ok, now examine the iPhone.image file, any closure based VM will open it.
>
> Open Monticello
> Visit the http://www.squeaksource.com/iPhone
> Consider the changes between the loaded
> iPhone-Events-John_M_McIntosh.8
> and the broken
>  iPhone-Events-John_M_McIntosh.11
>
> Now to help debugging you can use
>
> ObjectiveCBridge callNSLogWithSmalltalkString: 'foo'.
>
> to turn a smalltalk string into a UTF8 NSString that you feed to the NSLog() so it appears
> on the system console, which is accessible from xCode for the device.
>
> BTW you can turn that into a shortcut method
> say
> 'foo' sendToConsole
>
> where you stick a sendToConsole method on String to do the ObjectiveCBridge call.
>
> The Event logic between Squeak and Pharo is different, I tried to re-adapt the changes from Bert
> back to Pharo, but somewhere I missed a piece so after you load  iPhone-Events-John_M_McIntosh.11
> then UI interaction stops working on the iOS system.
>
> This will get you comfortable with working with the image & VM in a VM developer setting, then
> we can think over what to do about the open/GL.
>
>
> On 2010-09-11, at 4:31 PM, Kjell Godo wrote:
>
>> I don't know Open/GL ES but I am game to help you.  I am a fast
>> learner.  If you point me at what I need to know I will know it with
>> your help.  If you can't find anybody better you can try me.
>>
>> On Sat, Sep 11, 2010 at 3:37 PM, John M McIntosh
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>
>

> Wow , Thanks for all the work on all these instructions. I will start
> doing them and ask questions about everything I bounce off of as I go
> along.  Hopefully that will speed things up.
>
> First question :
> Do I need to get an iPad?  I think I can rent one.
> Do I need a Mac? I can rent it.  I have a Windows XP Pro PC.
> There seems to be an iOS emulator in your instructions.  So maybe I
> don't need an iPad or iMac?  I'll get them if it will help.
>
> Are there any books I can look at? spec pages?  I have access to 24x7
> via our Seattle library.  I'll search for "Open/GL ES".
>
> ( I like the VM.  I was/am making a virtual one inside of picoLARC on
> sourceforge.net .  I am interested to know about it. )
>
> On Sat, Sep 11, 2010 at 7:36 PM, John M McIntosh
> <[hidden email]> wrote:
>> Well ok, let's fix the problem with no touch events with the iPad eToy changes then as a first step.
>> To do this.
>>
>> Fetch the Platform source tree from squeakvm.org
>> Visit the iOS folder and find the SqueakPureObjc.xcodeproj
>> double click that to open in xcode.
>> Select the SqueakNoOGLIPhone project and build as debug against an iPad simulator.
>>
>> (a) At this point it should compile/link/install/run and you'll get a Pharo desktop. If not
>> then I'm missing something in the platform tree.
>>
>> (b) Oh note you might get an ERROR because you have to choose between the CALayer and Open/GL
>> render class and I wanted to ensure developers understood there was a choice to be made. Edit it to choose
>> the CALayer one since it's faster and is the same logic as you find in the OS-X post 4.x and pre 5.8
>>
>>
>> Ok, now examine the iPhone.image file, any closure based VM will open it.
>>
>> Open Monticello
>> Visit the http://www.squeaksource.com/iPhone
>> Consider the changes between the loaded
>> iPhone-Events-John_M_McIntosh.8
>> and the broken
>>  iPhone-Events-John_M_McIntosh.11
>>
>> Now to help debugging you can use
>>
>> ObjectiveCBridge callNSLogWithSmalltalkString: 'foo'.
>>
>> to turn a smalltalk string into a UTF8 NSString that you feed to the NSLog() so it appears
>> on the system console, which is accessible from xCode for the device.
>>
>> BTW you can turn that into a shortcut method
>> say
>> 'foo' sendToConsole
>>
>> where you stick a sendToConsole method on String to do the ObjectiveCBridge call.
>>
>> The Event logic between Squeak and Pharo is different, I tried to re-adapt the changes from Bert
>> back to Pharo, but somewhere I missed a piece so after you load  iPhone-Events-John_M_McIntosh.11
>> then UI interaction stops working on the iOS system.
>>
>> This will get you comfortable with working with the image & VM in a VM developer setting, then
>> we can think over what to do about the open/GL.
>>
>>
>> On 2010-09-11, at 4:31 PM, Kjell Godo wrote:
>>
>>> I don't know Open/GL ES but I am game to help you.  I am a fast
>>> learner.  If you point me at what I need to know I will know it with
>>> your help.  If you can't find anybody better you can try me.
>>>
>>> On Sat, Sep 11, 2010 at 3:37 PM, John M McIntosh
>>
>> --
>> ===========================================================================
>> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
>> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
>> ===========================================================================
>>
>>
>>
>>
>>
>

>
> In checking Instruments the 10% of the time taken in CALayer for memcpy of data out of the UIImages is gone when I do open/GL
> However if the drawing area becomes too big then the Open/GL drawing is slower, trade offs, trade offs.
>
> Also the byte alignment plays a part, that I have to explore more.  Still this is significant progress and we'll see what happens this week.
>
> A quick check on an iPhone 3GS shows CALayer pushes about 27fps, OpenGL does 43 fps. So much bigger win, I think we'll keep it.
>
> Sadly it doesn't fly on an iPod Gen 1 device (3.1.3) Zero clues.. It's possible of course there isn't isn't enough memory to go around, but
> given I've two different implementation classes I can choose the viable one at run time...
>
>
> On 2010-09-11, at 3:37 PM, John M McIntosh wrote:
>
>> On friday night I created an open/GL renderer to work with Squeak screen updates on the iPhone/iPad.
>>
>> Unfortunately it runs slows than the tiled Core Animation render that I had work out earlier in the year
>> with help from the Apple graphics engineers.
>>
>> So if anyone has a fair amount of knowledge about Open/GL ES maybe they could contact me and
>> we'll see if we can improve things?
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>
>
>
>
>

> I figured out we can't reasonably do open/GL arbitrary sized textures on the first gen iPhone or  iPod Touch.
> So I check for iOS 3.1.x and fall back to using the CALayer render, the wonders of polymorphism.
>
> I also commented out some of the nifty eToys on the iPad features lurking in the VM, like rotate me to get a keyboard,
> and also checked and confirmed that rotation of both types of renders with the view as a plain view or
> embedded in a scrolling view worked as expected.
>
> At this point I'm somewhat done the open/GL optimization.
>
> The implementation is to:
>
> Ccreate a texture and populate via glTexImage2D using apple's extension APPLE_texture_2D_limited_npot
> at view surface creation time.
>
> Then on a ioForceDisplayUpdate (implicit or explicit) we take the union of the rectangles observed
> in ioShowDisplayOnWindow and then use the
>
>  glTexSubImage2D
>
> to push the bytes one row at a time by calculating the offset into the bytes found in the Display special object,
> the texture then is drawn to the screen. I note we create the full sized glTexImage2D only once at startup time.
>
> An alternate choice was to use kEAGLDrawablePropertyRetainedBacking=YES and set the
> glTexImage2D & glTexSubImage2D pair for each ioForceDisplayUpdate using the subrectangle.
> But I found the Open/GL fps had a lot of jitter, so it seemed less animation friendly than creating the
> glTexImage2D once and doing  the glTexSubImage2D on each ioForceDisplayUpdate.
>
> Someone mutters, GL_FRAMEBUFFER_OES but at this point someone can pay me to explore faster
> alternatives.
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>
>