Smalltalk › Pharo › Pharo Smalltalk Users

How to render video frames?

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

3 messages Options

Rafael Luque

How to render video frames?

Hello,

I'm experimenting with an integration between OpenCV and Pharo based on uFFI. My native C library opens a video camera and captures frames for detecting certain points (aka blobs) of interest for my use case. The data for these frames are available to Pharo as uchar* inside a C struct.

My issues came when I tried to render the video camera frames inside Pharo. The C examples I developed for testing the shared library show frames around 50-60 fps. However, the best I have got in Pharo is around 15-20 fps.

My first try consisted of writing an specific ImageReadWriter for the OpenCV image format, so I can build a Form for each frame and render it with Morphic. This way I can render the video camera in GTInspector, but at a terrible 5-10 fps.

The second try was to use Bloc to render the Forms. I though the Moz2D backend will provider better performance. Indeed, I got around 15-20 fps, but it is still insufficient for playing video smoothly.

My current try consist of using BlExternalForm to avoid all the frame data processing inside Pharo, but I'm finding problems to render this kind of "shallow" Forms using Bloc.

I've also find some old mails (2017) about Clément Bera working in games based on Cairo + SDL. Should I explore this approach for video rendering?

In summary, I don't know what is the recommended approach (if any) to show a sequence of video frames with decent bit-rate.

Thank you!

Aliaksei Syrel

Re: How to render video frames?

Hi Rafael,

Assuming you have your OpenCV camera frame object named image in BGR format (default for OpenCV)

Here is how you can render it at high fps in Bloc (override BlElement>>#drawOnSpartaCanvas: aCanvas in a subclass)

surface := aSpartaCanvas bitmap

fromBGR: image imageData

extent: image width @ image height
stride: image widthStep.

aSpartaCanvas fill

paint: surface;
path: surface bounds;
draw

Cheers,

Alex

On Wed, 12 Jun 2019 at 22:39, Rafael Luque <[hidden email]> wrote:

Hello,

I'm experimenting with an integration between OpenCV and Pharo based on uFFI. My native C library opens a video camera and captures frames for detecting certain points (aka blobs) of interest for my use case. The data for these frames are available to Pharo as uchar* inside a C struct.

My issues came when I tried to render the video camera frames inside Pharo. The C examples I developed for testing the shared library show frames around 50-60 fps. However, the best I have got in Pharo is around 15-20 fps.

My first try consisted of writing an specific ImageReadWriter for the OpenCV image format, so I can build a Form for each frame and render it with Morphic. This way I can render the video camera in GTInspector, but at a terrible 5-10 fps.

The second try was to use Bloc to render the Forms. I though the Moz2D backend will provider better performance. Indeed, I got around 15-20 fps, but it is still insufficient for playing video smoothly.

My current try consist of using BlExternalForm to avoid all the frame data processing inside Pharo, but I'm finding problems to render this kind of "shallow" Forms using Bloc.

I've also find some old mails (2017) about Clément Bera working in games based on Cairo + SDL. Should I explore this approach for video rendering?

In summary, I don't know what is the recommended approach (if any) to show a sequence of video frames with decent bit-rate.

Thank you!

Rafael Luque

Re: How to render video frames?

Thank you Alex!

You made my day. Using your suggestion the video is playing more smoothly.

Better than that, I've managed the C library to get the frame data from OpenCV in BGRA format in order to be able to use the BlExternalForm, but now I can use directly BGR and save all those alpha bytes.

However, I've noticed that if I try to update my custom Bloc element each time a new detected frame is available I get an empty window. Only when I force a small delay (like 5 milliseconds) between frames I can see the video playing in the window.

An example of the code I'm running on a Playground would be like the following:

detector := OCVBlobDetector createOn: 0.
aSpace := BlSpace new
extent: 740@580;
title: 'OpenCV Frame'.
aSpace root background: Color gray asBlBackground.
frameElement := (OCVBlocDetectionElement on: detector detect)
relocate: 50@50; yourself.
aSpace root addChild: frameElement.
aSpace show.

detector live: [ :detection |
frameElement detection: detection.
frameElement invalidate.
5 milliSeconds wait.
].

Thank you!

El mié., 12 jun. 2019 a las 20:52, Aliaksei Syrel (<[hidden email]>) escribió:

Hi Rafael,

Assuming you have your OpenCV camera frame object named image in BGR format (default for OpenCV)
Here is how you can render it at high fps in Bloc (override BlElement>>#drawOnSpartaCanvas: aCanvas in a subclass)

surface := aSpartaCanvas bitmap
fromBGR: image imageData
extent: image width @ image height
stride: image widthStep.

aSpartaCanvas fill
paint: surface;
path: surface bounds;
draw

Cheers,
Alex

On Wed, 12 Jun 2019 at 22:39, Rafael Luque <[hidden email]> wrote:
Hello,

I'm experimenting with an integration between OpenCV and Pharo based on uFFI. My native C library opens a video camera and captures frames for detecting certain points (aka blobs) of interest for my use case. The data for these frames are available to Pharo as uchar* inside a C struct.

My issues came when I tried to render the video camera frames inside Pharo. The C examples I developed for testing the shared library show frames around 50-60 fps. However, the best I have got in Pharo is around 15-20 fps.

My first try consisted of writing an specific ImageReadWriter for the OpenCV image format, so I can build a Form for each frame and render it with Morphic. This way I can render the video camera in GTInspector, but at a terrible 5-10 fps.

The second try was to use Bloc to render the Forms. I though the Moz2D backend will provider better performance. Indeed, I got around 15-20 fps, but it is still insufficient for playing video smoothly.

My current try consist of using BlExternalForm to avoid all the frame data processing inside Pharo, but I'm finding problems to render this kind of "shallow" Forms using Bloc.

I've also find some old mails (2017) about Clément Bera working in games based on Cairo + SDL. Should I explore this approach for video rendering?

In summary, I don't know what is the recommended approach (if any) to show a sequence of video frames with decent bit-rate.

Thank you!