Playing Flash movies in-image

"I'm trying to play a flash movie (.flv - H.264, 320 x 216, Millions; AAC,
Stereo, 22.050 kHz). I imported and played it successfully in OpenSophie, but before I attempt to
port the code (which is Tweak-based) into Squeak, is there any currently
existing code that can do this?"

No. In fact, I don't know how you could be playing a Flash movie in a Sophie image, because the Flash player there is so old
that it only understands swf file animations. It couldn't play movies because that was still five years off.

As far as I can read the history, in the late 90s Squeakers explored existing multimedia, then they went
crazy with originality. What they built is so pervasive it's hard to see it, because it changed the
environment. Basically, this was a pre-Croquet/Etoys/Tweak period. They dropped Flash and never picked it up again,
while creating insane graphics using the SurfacePlugin and scripting (which uses time stamped MessageSends
in a queue to substitute single processes for multi-threading.) This era started with this memo:

http://tweakproject.org/ABOUT/FAQ/OriginalTweakMemo/

(I can't remember who wrote it. I don't think he's here anymore. :P)

How you're playing a Flash movie I can't imagine, because they'd dropped that technology over a decade ago and movies started
being played in Flash successfully five, six years ago.

"I see what you mean.  I got stuck even before this process - if you can't
bring up halos, how do you even know which class/method to *start* with?"

Well, yea. That's been a problem. For sure.
But I don't believe in anything unless I can see it. I can assure you I'll never want to EVER watch a filmed movie of Macbeth when
this project is over. I've heard Orson Wells intone too many times.

I know that the SophieMovie hierarchy is the place to start. And learning more about Tweak is slowly showing
me how that model is connected to the view. If you want to take something from Tweak and put it in Morphic, and you don't want
to have start it from a Workspace, then you need to know how to convert a CPlayer into a Player. And so it goes...


Chris

Chris Cunnington-3 wrote

I don't know how you could be playing a Flash movie in a Sophie image, because the Flash player there is so old that it only understands swf file animations. It couldn't play movies because that was still five years off.

I don't know either, but I'm watching it play :)

Chris Cunnington-3 wrote

But I don't believe in anything unless I can see it. I can assure you I'll never want to EVER watch a filmed movie of Macbeth when
this project is over. I've heard Orson Wells intone too many times.

I don't think it's a MacbethPlayer, I think it plays other movies too ;-)

Chris Cunnington-3 wrote

If you want to take something from Tweak and put it in Morphic... you need to know how to convert a CPlayer into a Player.

I filed Sophie-Movie into trunk.  I'll see what I can come up with.

Thanks.
Sean

On 12/29/2010 6:16 PM, Chris Cunnington wrote:
> No. In fact, I don't know how you could be playing a Flash movie in a
> Sophie image, because the Flash player there is so old
> that it only understands swf file animations. It couldn't play movies
> because that was still five years off.

Because Sophie uses Quicktime for all its media needs. Quicktime
supports Flash just fine.

> As far as I can read the history, in the late 90s Squeakers explored
> existing multimedia, then they went crazy with originality.
> What they built is so pervasive it's hard to see it, because it changed
> the environment. Basically, this was a pre-Croquet/Etoys/Tweak period.

I wrote the Flash player for Disney projects and we used it there. The
reason it was never updated was because Macromedia (which owned Flash at
this point) changed their policy on releasing the SWF spec - in order to
obtain the spec for SWF > 3.0 one had to agree to NOT write any player
for SWF files; you were only allowed to use that documentation to CREATE
output in SWF. So a complete no-go for us (I have not checked what
recent policy is).

> They dropped Flash and never picked it up again, while creating insane
> graphics using the SurfacePlugin and scripting (which uses time stamped
> MessageSends in a queue to substitute single processes for multi-threading.)
> This era started with this memo:
>
> http://tweakproject.org/ABOUT/FAQ/OriginalTweakMemo/
>
> (I can't remember who wrote it. I don't think he's here anymore. :P)

I wrote the memo, I own the site, and methinks I'm still around ...

Cheers,
   - Andreas

>Because Sophie uses Quicktime for all its media needs. Quicktime 
>supports Flash just fine.

I didn't know that at all. Thank you for pointing it out. 

> I wrote the Flash player for Disney projects and we used it there. The 
> reason it was never updated was because Macromedia (which owned Flash at 
> this point) changed their policy on releasing the SWF spec - in order to 
> obtain the spec for SWF > 3.0 one had to agree to NOT write any player 
> for SWF files; you were only allowed to use that documentation to CREATE 
> output in SWF. So a complete no-go for us (I have not checked what 
> recent policy is).

That's interesting. That explains to me how I perceived that all interest in Flash 
had stopped. The code goes up to that point and that's all I could see. 

> I wrote the memo, I own the site, and methinks I'm still around ...

I do appreciate the truth of that statement. Talking about these things here makes
me a bit self-conscious, as I like to describe things in historical terms, while all 
the players are right here. But silence teaches me nothing, so...

Your site is invaluable to anybody interested in Tweak, which includes anybody 
who has ever had doubts about Morphic. 

http://tweakproject.org

Chris 

Ok, well it is great that someone is attempting to drag that movie player out of sophie. 

Sean P. DeNigris wrote

I filed Sophie-Movie into trunk.  I'll see what I can come up with.

Okay, here's a first stab - Mac-only (platform temporarily hardcoded).  If you download:
* FFI including Examples
* SpdQT from http://www.squeaksource.com/SophieGoldDig

then you can evaluate the following:
        aURI := URI fromString: 'file:///path/to/aQuickTimeFile.flv'.
        manager := SophieResourceManagerFolderPackage new.
        resource := SophieResource basedOn: manager.
        resource fileURI: aURI.
        movie := QuickTimeMovie new.
        movie resource: resource.
        movie openMovieFromURI: aURI.
        "movie play."
        movie startMovieQT.
        "movie stopMovieQT."
when you do #startMovieQT, you will hear the movie audio, but drawing has not been implemented, so you won't see anything.  At least we're connected to the QT API :)

Sean

>The concept is that logic is handled a URL  for a media object (video or audio).  It must then determine what type of
>media that is, using features of Quicktime or MimeTypes to figure out a optimal guess at the concrete decoder class to use.
>For that a weighted matrix of choices is used because if we have quicktime, use that, but maybe not if it's an OGG movie since
>quicktime doesn't support OGG natively, but there is a plugin...  Frankly this is the important piece of the whole architecture.

If it's QT compatible, then it goes to NewMovieFromDataRef, which is broader than NewMovieFromFile. The file could be anywhere: on the disk or on the
Internet. I've been extremely QT centric at this point, but I see what you're saying about OGG. Though I haven't spent time yet looking for the
code that parses an URI for different file types.

>Now the decoder logic has a number of goals,  play, stop, rewind, close..  More interesting is play, what happens then is the
>decoder's objective is to start at a particular position in the movie, and on each frame change signal to tweak the frame has changed.

That's the infinite Tweak loop between SophieMovieFramePlayer and SophieQuickTimeMovieViaPlugin. Press #play and that delegates
to SophieQuickTimeMovie>>playMovieScript. It's an loop that Tweak has made into a Process to control. I've
seen it appear as a process in the ProcessBrowser, when the movie plays. It goes back to SophieMovieFramePlayer>>onMovieUpdated
to get the #invalidate to redraw the screen, which is in the #costume of that Player.

>When it gets that, it then has the responsibility to gather up the bits and ask tweak to splash them onto the tweak UI.

Yea... I'm not sure how that works: "splash onto the Tweak UI". A Form is full of bits. In an MPEGMoviePlayerMorph we have Forms full of
bits for each frame, I suppose. But with the SurfacePlugin, we just have a Form defined location that contains nothing. I thought the SurfacePlugin
created a surface that floated above Squeak controlled by the OS.

All I see are SophieQuickTimeMovieViaPlugin>>gWorldPtr pixMapHandle. I never see fields of bits or a binary array upon inspection.
I can't see how the bits fill the #rectangle or #offscreenForm. Because it's using FFI, it seems in a black box. How do I visualize what the
SurfacePlugin is doing?


Chris

Well I'm impressed. It does just what you said.

It falls juuuuust a little short of what we need it to do.
But seeing as I've been looking at this for months and haven't got that
far. Again - I'm impressed. Have an attaboy:

"Attaboy, Sean!"

Chris

Chris Cunnington-3 wrote

"Attaboy, Sean!"

I love compliments, so I went ahead and got the video working ;-)  If you download the latest package from the above mentioned repository, and evaluate:
        aURI := URI fromString: 'file:///path/to/aQuickTimeMovie.flv'.
        manager := SophieResourceManagerFolderPackage new.
        resource := SophieResource basedOn: manager.
        resource fileURI: aURI.
        movie := QuickTimeMovie new.
        movie resource: resource.
        movie openMovieFromURI: aURI.
        movie play.
        "movie stopMovieQT."

you will have a QuickTime movie playing for you in Squeak.

Todo:
* Have it be a morph - It's temporarily drawing the form directly to the display
* run the loop in another thread - it currently locks up the UI (a user interrupt will get you out)

It seems on the verve of being useable very soon!!

Sean

You'll note that there are a *lot* of SophieResourceManager subclasses.

It's goal in life was to provide a storage framework for resources and components of Sophie books.
The concrete class are all about, is the book stored on the internet, on the local hard drive, is it a gzip file, a directory of files, a gzip file WITHOUT compression (aka OLPC support).  Oh lots of choices and different trials were made over the years  resulted in lots of subclasses...

Ya, so make the manager, then stuff a resource into it.  You'll note it's not a MovieResource rather the SophieResource was really generic...

Sophie NEVER NEVER NEVER thought of a resource item being a file path. Great pains to ensure the proper UTF8 URI was used and at the point where we need the actual data bits, it gets translated into something the read/write underpinnings could digest.  Fortunately Quicktime was very happy to digest URI for us. Mind there was *some* problems with quicktime on windows but I recall that Apple over time shipped some fixes.

As earlier mentioned the SophieResource is there to contain the meta-data for the movie so we avoid digging in the movie every time we need an answer, also it provided a translation layer as the different decoders had to be invoked to drag the data out to be cached within the resource's meta-data.

On 2010-12-29, at 1:32 PM, Sean P. DeNigris wrote:

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================

That's amazing!
I mean, that's a player right there. Wow.

It worked with a QT movie and an MPEG one.
Very cool. You certainly found the way through.

Chris

So you want to draw bits, yes squeak draw bits from the bitmap of a form.
So you want an external library to place bits into that form's bitmap.

Oops problem, the squeak garbage collector wants to move the memory address of the bitmap, that won't work.

The surface plugin puts an integer into the slot where the oops reference to the bits would.
The surface plugin logic then interacts with the surface plugin client to say yes surface plugin form #72 actually points to that
chunk of memory you allocated for me and when I'm asked to draw the form we take the data from over there, not from the
Squeak image space.

Over in the third party library we allocate the chunk of memory for our bits, tell the surfaceplugin logic who we are and where the memory address is and
of course agree what the bits and shape is, we draw into our *shared* bits.. Everyone is happy.  


MMM I'm not sure I use the surface plugin for quicktime, rather I thought a Gworld was used and we copy the bytes. Been years, someone might look.

Cough I will point out there is a bug in the surface plugin logic.  It seems the drawing loop for Form drawing actually reads one word over the end of the bitmap.
Disaster lurks when the form is a multiple of the operating system page size, and memory is allocated such that page 1-N are the form, but page N+1 is tagged as executable, or non-readable, etc... I *think* we reverted from using the surface plugin in Sophie to using a gworld and doing the copy just because we occasionally could get a crash  on a read page protection error.  

Fixing the draw loop was non-trival, you are welcome to do that...


On 2010-12-29, at 1:54 PM, Chris Cunnington wrote:
> Yea... I'm not sure how that works: "splash onto the Tweak UI". A Form is full of bits. In an MPEGMoviePlayerMorph we have Forms full of
> bits for each frame, I suppose. But with the SurfacePlugin, we just have a Form defined location that contains nothing. I thought the SurfacePlugin
> created a surface that floated above Squeak controlled by the OS.

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================

If you have Gofer and Metacello installed, you can install the current version with:

Gofer new
    squeaksource: 'MetacelloRepository';
    package: 'ConfigurationOfFFI';
    load.

((Smalltalk at: #ConfigurationOfFFI) project latestVersion) load.
       
Gofer new squeaksource: 'SophieGoldDig'; package: 'SpdQT'; load.

I got it to work with a wmv file, which is cool.
I think the whole thing can uncoil to cover more and more ground now
that you've made it work, made a prototype.

Chris

Thanks for that rundown, John.
It definitely uses GWorld. For sure:

SophieQuickTimeMovieViaPlugin>>gWorldPtr
SophieQuickTimeInterface>>setGWorldPtrOntoSurface:width:height:rowBytes:depth:movie:

I guess you're saying that's somewhat different than the SurfacePlugin.
Cool stuff.

Chris

GWorld is a Apple 1986? technology for defining a bitmap and characteristics. The bytes can exist either in RAM or VRAM (depends).
Technically it's been replaced by a BitMap Graphics Context, but every mcintosh app written since the 1984 worked with gworlds,

Mmm ok well mm it HAS been nearly 5 years since I wrote all that, in checking I see that I *cheat* and take the GWorld pointer from QuickTime and
register it via stQuicktimeSetSurfacewidthheightrowBytesdepthmovie  as the bitmap for the Surface Plugin api. Therefore for macintosh we have quicktime actually write bits into the bitmap that is registered  with the surface plugin.

The downside is that for the windows platform (anyone want to build the quicktimeplugin?) it somewhere must copy the bits out of the gworld bitmap
back into a Squeak Form bitmap oops via a memory copy all of some sort. This means moving MB of data every frame that we don't do on the macintosh.


On 2010-12-29, at 3:52 PM, Chris Cunnington wrote:

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================

John M McIntosh wrote

Ya, so make the manager, then stuff a resource into it.  You'll note it's not a MovieResource rather the SophieResource was really generic...
<snip>
As earlier mentioned the SophieResource is there to contain the meta-data for the movie so we avoid digging in the movie every time we need an answer, also it provided a translation layer as the different decoders had to be invoked to drag the data out to be cached within the resource's meta-data.

Thanks for all the info John.  I wasn't sure I got the intention of your post.  Based on your earlier message, I was thinking that when I started cleaning up the port, I would replace SophieResource and the manager with a class specifically-designed to hold the meta-data you mentioned.  Was this reiterating that or did I miss something?

Sean

On 2010-12-30, at 7:38 AM, Sean P. DeNigris wrote:

> Thanks for all the info John.  I wasn't sure I got the intention of your
> post.  Based on your earlier message, I was thinking that when I started
> cleaning up the port, I would replace SophieResource and the manager with a
> class specifically-designed to hold the meta-data you mentioned.  Was this
> reiterating that or did I miss something?
>
> Sean

No that is correct, in the past people got hung up looking at SophieResourceManager and thinking they had to migrate that monster over.

Since you are in there, and dragging the FFI stuff about you might consider grabbing the clipboard stuff
so that we can copy/paste graphics on windows/mac to/from squeak?

I took a run at it a year back but I'm afraid the Windows community just couldn't care less so I couldn't get any
testers or confirmation  that it worked there.

Today I'd suggest we just abandon them and let them sort it out later.

I put the image & changes at

ftp://ftp.smalltalkconsulting.com/experimental

pharo.extendedClipboard.2.zip


For image read/writing it uses  SophieImageReadWriter which uses Quicktime's api to translate any supported media types into bitmaps.
If that fails or is not available it just like SophieMovie grinds down a decsion tree to find a supported Squeak plugin or smalltalk code to
decode the supplied mime-type.


PS ya, I see currentMIMETypes := SophieBookEditor clipboardMimeTypes.  but SophieBookEditor is not there and
ExtendedClipboardMacInterface>>copyImageDataFromClipboard   but is that used? or did I replace it with ExternalClipboard


Mmmm don't forget the file get/put dialogs for native file interface. We did both the Windows and the OS-X ones with proper UTF8 255 byte file name support.

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================

John, i put that in my todo list.



--
Best regards,
Igor Stasenko AKA sig.

On 30.12.2010, at 18:31, John M McIntosh wrote:

We use the extended clipboard plugin in Etoys. Copying/pasting bitmaps or rich text between apps works nicely (on Mac and Unix anyway, Windows remains to be implemented).

However, there is one major issue: When pasting, we have not found a way to determine which clipboard should take precedence - Squeak's text clipboard, Morphic's object clipboard, the system's string clipboard, or the extended Text/Bitmap/Etc. clipboard. In Sophie I guess you used the extended clipboard exclusively, but for Squeak it's not quite as clear-cut ...

- Bert -