CameraPlugin segmentation fault on 64-bit Linux

On Sat, Apr 29, 2017 at 10:08:23AM -0600, Daniel Drake wrote:
> Hi Dave,
>
> On Sat, Apr 29, 2017 at 9:58 AM, David T. Lewis <[hidden email]> wrote:
> > I don't think that the camera plugin was ever made 64-bit clean. A bug report
> > is at http://bugs.squeak.org/view.php?id=7816
> >
> > This would be a good project for an interested volunteer :-)
>
> I'm an interested volunteer, knowledgeable of 32 vs 64 bit challenges
> but with no squeak-vm experience. Can you briefly provide some
> direction of where to start here?
>
> Are there any other plugins that have previously been made 64-bit
> clean? (Maybe there is a similar example in svn history and I can find
> some direction from that.)


Hi Daniel,

Great, thanks for looking into this!

SoundPlugin and SqueakSLL are examples of plugins that required updates
for 64-bit.

The majority of issues involve storage of 64-bit pointer values in 32-bit
integers. Usually this is fixed by providing correct type declarations
throughout the support code, and by making sure that the translated code
from the plugin class in VMMaker produces code with type declarations
that match the support code.

In the case of CameraPlugin, I would be inclined to start by looking at
the support code in platforms/unix/plugins/CameraPlugin/ to get an idea
of what the function declarations are (or should be). Look through it
for any type casts, usually these are a red flag for something you are
going to need to fix.

If everything looks good in platforms/[unix|win32|Moc OS|iOS] then the
next thing to check will be the declarations in
platforms/Cross/plugins/CameraPlugin/CameraPlugin.h. These will probably
be ok, but make sure that anything declared as int is really a 32-bit
integer value.

If you are statisfied that the support code is all in good order for
compiling on 64-bits, then take a look at CameraPlugin in your VMMaker
image, and at it sgenerated source code in src/plugins/CameraPlugin/CameraPlugin.c.

Look for places in the generated code where the support code functions
are being called incorrectly, or where methods in the plugin itself
use default sqInt variable declarations that should be explicitly
declared as something else.

I do this sort of work using an interpreter VM on a 64-bit Linux
system because it lets me easily compile 64-bit VMs that run on
32-bit sqInt object memories, which is the combination that most
reliably finds type declaration problems.

You will probably want to run your VMs under gdb to locate problems,
but don't be afraid to hack some printf's into your generated plugin
source in order to find problems. Do a fflush after each printf
so you can see the output right before your VM crashes.

Thanks and HTH,
Dave