IMAGE PROCESSING AND DISPLAYING

I'm working on a image class for some image processing operations and
i'm having some speed and performance problems.

I tried two options based on the bitmap support of Dolphin. I only allow
1, 8 and 24 bit bitmaps.

1- Create a Matrix of Array of ByteArrays for each bitmap. Then for
display operations I create a corresponding Bitmap.

2- Work directly over the Bitmap bytes mapping each pixel (x,y) to his
respectively location in the buffer memory address space. No extra
operation is needed for displaying since i already have the Bitmap.

The problem is that the first option is faster than the second one for
image processing operations, but it takes to much time for, i think,
garbage collecting objects, taking the development environment in a idle
state for a while. So I changed the representation of the Matrix to only
one ByteArray but nothing changed.

I need to know before make any change which it's better:

1 - Use Dolphin classes or allocate memory externally and access it like
an ExternallAddress object.
2 - Use Bitmap for display or write my own image display code.
3 - Use anything than Dolphin.

Thanks.

Adrián,

> I need to know before make any change which it's better:
>
> 1 - Use Dolphin classes or allocate memory externally and access it like
> an ExternallAddress object.
> 2 - Use Bitmap for display or write my own image display code.
> 3 - Use anything than Dolphin.

I wouldn't go as far as 3, because there is a lot to be gained from
developing in Smalltalk.  However, Smalltalk is not known for speed in
fixed-size arithmetic, so it wouldn't be surprising to need a C/C++ DLL for
some of the number crunching in order to get reasonable performance.  If you
go that way, keep in mind that extern "C" allows you to create C-callable
functions with non-mangled names (so they are easy to call) which have full
use of C++ extensions in the function body.

Squeak offers the ability to write in a Smalltalk subset called Slang which
allows testing and debugging in the image followed by a generation step to
write C code that can be compiled to create what Squeakers call a plugin.
There is also some image processing code for Squeak, but, the last copy I
saw of it (not to say it hasn't been updated since) was limited to 2.7 or
maybe 2.8.

Back to Dolphin, you are wise to think about garbage collection, but, since
you are already using ByteArray, it might not be a huge factor.
StructureArray can be very convenient, but, it can also be very slow because
(IIRC) it creates a new structure instance on every array access.

A good way to proceed would be to use Ian Bartholomew's profiler to see
where your code is spending time.  If it points to arithmetic, then a DLL is
probably a good answer.  Generally I let Dolphin allocate/deallocate memory
and pass pointers into C-callable C++ functions.  There is some good info on
the Wiki about external interfacing and avoiding trouble.  In particular,
watch out for errors in garbage collection.  External interfacing gone wrong
allows you to damage an image beyond repair; it's typically fine as long as
you don't save an image after it happens.  Also see the Wiki for backup and
recovery procedures; reading it before rather than after a disaster can save
you a lot of trouble.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Adrián,

> 2- Work directly over the Bitmap bytes mapping each pixel (x,y) to his
> respectively location in the buffer memory address space. No extra
> operation is needed for displaying since i already have the Bitmap.

How are you doing this?. If you are using the #pixelAt: and #pixelAt:put:
methods in Canvas then there might be a faster way that involves writing
directly to the bits contained within the DIBSection (I don't think you can
use the Bitmap class). NB This is a non standard, to say the least, and
requires a bit of knowledge about the way that Bitmaps are constructed and
stored. It's usefulness also depends on what exactly is involved in the
"Image Processing"

As an example, the following creates a monochrome DIBSection and draws
vertical bars on it. Each byte represents 8 columns so altering the byte
used in the code, 1 = black and 0 = white, changes the columns created.

dib := DIBSection width: 100 height: 100 depth: 1.
lineSize := dib getDIBSECTION dsBm bmWidthBytes.
bitStart := ExternalAddress fromInteger: dib imageBits.
1 to: dib extent y do: [:row |
    1 to: lineSize do: [:col |
        bitStart at: row - 1 * lineSize + col put: 2r00011000]].
ImagePresenter showOn: dib

It requires a bit more work on your part but, I imagine as I've never used
it in anger, can be quicker than using the API functions.

Mail me if you want more details

Regards
    Ian

Ian Bartholomew wrote:
 > Adrián,
 >
 >
 >>2- Work directly over the Bitmap bytes mapping each pixel (x,y) to his
 >>respectively location in the buffer memory address space. No extra
 >>operation is needed for displaying since i already have the Bitmap.
 >
 >
 > How are you doing this?. If you are using the #pixelAt: and

How are I doing this?

I create a subclass of DIBSection, MyBitmap class, and get the Bitmap
Address space with

imageBytes
        ^ByteArray fromAddress: self imageBits
                length: (self height * self rowBytes)

rowBytes
        "Each row must be multiple of 4"
        | row |
        row := self widthBytes.
        (row rem: 4) = 0 ifFalse: [^row + 4 - (row rem: 4)].
        ^row

widthBytes
        self depth > 1
                ifTrue: [^self width * (self depth / 8)] "8 and 24 bits"
                ifFalse: [^(self width / 8) ceiling] "1 bit"

Actually I'm not using this method for bitmap data access, I explained
below.

Then I create a hierarchy BMPMatrixAbstract with

Superclass subclass: #BMPMartixAbstract
        instanceVariableNames: 'image width height rowBytes rows length'
        classVariableNames: ''
        poolDictionaries: ''
        classInstanceVariableNames: ''

Where <image> handles the MyBitmap object; <width> and <height> values
for faster access; <rowBytes> the image rowBytes value for faster
access, <rows> a Collection of ExternalAddress objects, one for each
image row, or the MyBitmap imageBytes ByteArray (not using it now) ;
<length> = height * rowBytes (image buffer length) for faster access.

rowsInit
        | baseAddress |
        rows := OrderedCollection new.
        baseAddress := self image imageBits yourAddress.
        1 to: self height
                do: [:each |
        rows add: (ExternalAddress fromInteger: baseAddress +
                                        (self idxForRow: each))].

idxForRow: row
        ^self length - (self rowBytes * row)


Then I create the next 3 BMPMatrixAbstract subclasses only for buffer
access. Define only #at: and #at:put: methods for:

-BMPMatrix1 (1 bit/pixel)
-BMPMatrix8 (1 byte/pixel)
-BMPMatrix24 (3 bytes/pixel)

I created this because I already have a Pixmap class for images handling
the data with a simple Matrix class. So I only replaced the Matrix
representation with the BMPMatrix representation.

I think the problem is for BMPMatrix1 and BMPMatrix24 when I need to
write a pixel (#at:put:). For 24 bits images I write the r, the g, and
the b one byte a time. For 1 bit images I need to read the byte for the
corresponding bit, modify the bit and write the byte again to the buffer.

I hope you understand my explanation, and thanks for your answer.

Regards,

ADRIAN

Adrián,

Thanks for the explanation. As you are already getting quite close to the
internal implementation of the Bitmaps there's probably not a lot I can
suggest.

Bill's suggestion of writing the really time critical bits in a dll created
using your favourite low level language is one option that might solve the
speed problem.

You could avoid the overhead of creating ByteArrays by directly accessing
the bits in the bitmap and using your overidden #at: and #at:put: methods to
access the bitmap directly.

On the other hand, for 24 bit Bitmaps you might gain by using a
StructureArray for each row, each element in this array being an instance of
RGB. This should improve the speed at which you can change values for a
specified pixel (accessing the inst vars of an RGB should be faster than
accessing a ByteArray offset). You might also gain from using a flat
StructureArray rather than having one for each row?

All these depends on trade-offs though, how much you gain by "wasting" time
preparing a structure beforehand as opposed to the time taken to using it in
it's native format, and it's impossible to judge that from afar.

Regards
    Ian

Thanks for the advice. I planned before to use StructureArray but I need
   a RGB structure and write all the RGB existing behavior again; maybe
I'll do it. Also I planned to use a "Bit" structure to access each bit
directly, for binary images, but I didn't figure how.

Regards,
ADRIAN



Ian Bartholomew wrote:
 > Adrián,
 >
 > Thanks for the explanation. As you are already getting quite close to the
 > internal implementation of the Bitmaps there's probably not a lot I can
 > suggest.
 >
 > Bill's suggestion of writing the really time critical bits in a dll
created
 > using your favourite low level language is one option that might
solve the
 > speed problem.
 >
 > You could avoid the overhead of creating ByteArrays by directly accessing
 > the bits in the bitmap and using your overidden #at: and #at:put:
methods to
 > access the bitmap directly.
 >
 > On the other hand, for 24 bit Bitmaps you might gain by using a
 > StructureArray for each row, each element in this array being an
instance of
 > RGB. This should improve the speed at which you can change values for a
 > specified pixel (accessing the inst vars of an RGB should be faster than
 > accessing a ByteArray offset). You might also gain from using a flat
 > StructureArray rather than having one for each row?
 >
 > All these depends on trade-offs though, how much you gain by
"wasting" time
 > preparing a structure beforehand as opposed to the time taken to
using it in
 > it's native format, and it's impossible to judge that from afar.
 >
 > Regards
 >     Ian
 >
 >