linux build stability

> Hi All,
>
>       you may already know that there have been strange stability
> problems with the Cog VM on linux.  Problems with the heartbeat appear
> to derive from specific compilations, one compilation of the same source
> producing an executable that will crash, another producing one that
> won't.  recent testing at Teleplace showed that an effect due to what
> was presumed to be a compiler bug (specifically the optimization level
> used to compile the heartbeat, high causing a crash) was not repeatable.
>   So today in building new production VMs for Teleplace I decided to do
> three parallel linux builds and see if all produced the same results.
>   While there are macros used in the source that are date dependent (use
> of __DATE__) AFAIA there are none apart from version.c/version.o that
> depend on time, and no timestamps or current directory paths in linux
> objects, and so, provided different compilations of the same source are
> done on the same day, the results should be bit-identical.  In my
> experiment this turns out not to be the case, which is more than a
> little alarming.
>
> What I'm seeing is different results duplicating unixbuild/bld to
> unixbuild/bldb and unixbuild/bldc, doing identical configures and makes
> in each of the three directories and then comparing resulting objects.
>   I see this in a bare metal laptop with local sources running CERN SLC5
> and on a Parallels VM running CentOS 5.3 (both derived from RHEL).  I'm
> using gcc 4.1.2.  Here's a script that shows example differences:
>
> bld$ for f in *.o vm/*.o; do echo $f;cmp $f ../bldb/$f; cmp $f
> ../bldc/$f; done
> disabledPlugins.o
> disabledPlugins.o ../bldb/disabledPlugins.o differ: byte 200, line 4
> disabledPlugins.o ../bldc/disabledPlugins.o differ: byte 200, line 4
> version.o
> version.o ../bldb/version.o differ: byte 166, line 3
> version.o ../bldc/version.o differ: byte 166, line 3
> vm/aio.o
> vm/cogit.o
> vm/debug.o
> vm/gcc3x-cointerp.o
> vm/osExports.o
> vm/sqExternalSemaphores.o
> vm/sqHeapMap.o
> vm/sqLinuxHeartbeat.o
> vm/sqLinuxWatchdog.o
> vm/sqLinuxWatchdog.o ../bldb/vm/sqLinuxWatchdog.o differ: byte 33, line 1
> vm/sqLinuxWatchdog.o ../bldc/vm/sqLinuxWatchdog.o differ: byte 33, line 1
> vm/sqNamedPrims.o
> vm/sqNamedPrims.o ../bldb/vm/sqNamedPrims.o differ: byte 6346, line 30
> vm/sqNamedPrims.o ../bldc/vm/sqNamedPrims.o differ: byte 6346, line 30
> vm/sqTicker.o
> vm/sqUnixCharConv.o
> vm/sqUnixExternalPrims.o
> vm/sqUnixMain.o
> vm/sqUnixMain.o ../bldb/vm/sqUnixMain.o differ: byte 31415, line 170
> vm/sqUnixMain.o ../bldc/vm/sqUnixMain.o differ: byte 31414, line 170
> vm/sqUnixMemory.o
> vm/sqUnixThreads.o
> vm/sqUnixVMProfile.o
> vm/sqVirtualMachine.o
>
> Using objdump --disassemble I can see for example that sqLinuxWatchdog.o
> and sqUnixMain.o differ only in the symbol table, not the executable
> code.  So perhaps this is not meaningful, and merely noise.  But with
> simple files like disabledPlugins.c that different objects are produced
> at all in different runs is rather worrying:
>
> bld$ cat disabledPlugins.c
> /* this should be in a header file, but it isn't.  ho hum. */
> typedef struct {
>    char *pluginName;
>    char *primitiveName;
>    void *primitiveAddress;
> } sqExport;
> sqExport vm_display_Quartz_exports[] = { 0, 0, 0 };
> sqExport vm_display_custom_exports[] = { 0, 0, 0 };
> sqExport vm_display_fbdev_exports[] = { 0, 0, 0 };
> sqExport vm_sound_MacOSX_exports[] = { 0, 0, 0 };
> sqExport vm_sound_NAS_exports[] = { 0, 0, 0 };
> sqExport vm_sound_OSS_exports[] = { 0, 0, 0 };
> sqExport vm_sound_Sun_exports[] = { 0, 0, 0 };
> sqExport vm_sound_custom_exports[] = { 0, 0, 0 };
>
>
> I wonder
> - do you see the same effect?
> - does this happen with gcc versions other than 4.1.2?
> - does it happen on non-RHEL-derived distros?
> - is this a meaningful signal or just harmless noise?
> - what am I doing wrong?
>
> Clearly I need to look more carefully but I thought I'd ask y'all in
> order to understand and hopefully solve the build instabilities as
> swiftly as possible.
>
> If you do want to try and reproduce this simply duplicate the build
> directory (unixbuild/bld in the Cog VM source) twice and do three
> separate configures and makes, one in each of the build directories,
> each from the same source code.  Then run some variation fo the script
> above to compare the object files so produced.
>
> best
> Eliot