How to generate identically image file after snapshots

> Hi All,
>
> We have tried to produce the same (bit identically) image file after two
> consecutive snapshots. We start from a base image then, fileIn several
> files into it and finally, we just SmalltalkImage current snapshot: true
> andQuit: true. We need this to verify the image file generated by a
> third-party with a checksum by executing a script.
> After trying several ways to get it (even by scripting the fileIn process
> and the snapshot), we found that the image files have, beside the
> timestamp differences, thousands of other differences and sometimes the
> snapshots have also size differeces.
> We supose that this kind of issues may occur due to the GC activity.
> Are this issues from the way GC process is changing dynamically the  
> memory
> bytes? There is a way to inhibite this activity?
> Attached are the scripts we use to produce the image files.
>
> Many thanks in advance,
> Martin Troielli

> Hi Martin,
>
> there are a lot of objects (like, for example subinstances of  
> ContextPart) allocated and deallocated on which you do not have much  
> control.
>
> One corner from which this could be started is to consider enumerating  
> (in two sister .images) all the objects you want to deploy. If that  
> fails to produce comparable objects (for any reason, for example if you  
> cannot order/compare object identities other than by hash identity and  
> the latter is assigned by the VM and not by you) then, hrm, it fails.
>
> But if not then you could trace out all the objects you want (thereby  
> disacrding all the unwanted) and the resulting (two sister) .image files  
> then have the same contents byte by byte, because you fix the object's  
> position in the files. I've done that with other images and  
> non-Smalltalk interpreters.
>
> Having said that, your project doesn't look to be easy.
>
> /Klaus
>
> On Mon, 30 Jul 2007 17:31:43 +0200, Martin wrote:
>
>> Hi All,
>>
>> We have tried to produce the same (bit identically) image file after two
>> consecutive snapshots. We start from a base image then, fileIn several
>> files into it and finally, we just SmalltalkImage current snapshot: true
>> andQuit: true. We need this to verify the image file generated by a
>> third-party with a checksum by executing a script.
>> After trying several ways to get it (even by scripting the fileIn  
>> process
>> and the snapshot), we found that the image files have, beside the
>> timestamp differences, thousands of other differences and sometimes the
>> snapshots have also size differeces.
>> We supose that this kind of issues may occur due to the GC activity.
>> Are this issues from the way GC process is changing dynamically the  
>> memory
>> bytes? There is a way to inhibite this activity?
>> Attached are the scripts we use to produce the image files.
>>
>> Many thanks in advance,
>> Martin Troielli

> Hi Klaus,
>
> Thanks for the information. We have modified the VM in order to reduce the
> GC activity inhibiting it until the fileIn processes are done, but had no
> luck. The produced files were different with less differences.
> We think we have to follow an approach similar to yours. We thought to
> generate a serialized file with all the CompiledMethods we use, without
> change the base image, merging them only when squeak starts up. We hope
> that this process does not demand too much time, since we have also a lot
> of resources to bring up at that time :S

> On Tuesday 31 July 2007 7:32 am, John M McIntosh wrote:
>> If you have some desire to make duplicate images look at
>>
>> primitiveSnapshot
>>
>> and consider cloning that to perform the writeImageFile() twice using
>> different image names.
> It would be simpler to copy image files after they are written. But  
> I don't
> this the issue was to copy image files locally. The original poster  
> wanted to
> update third-party images by shipping fileIns to a reference image  
> instead of
> the whole image itself. The poser, then, is how to verify that the  
> resulting
> image is same as intended.
>
> I would simply use xdelta (see xdelta.org) for situations like  
> this. E.g.
>
>  xdelta delta ref.image thirdparty.image thirdparty.xd
> and ship thirdparty.xd
>  xdelta patch thirdparty.xd ref.image thirdparty.image
>
> The downside is xdelta is a memory hungry utility. How big is the  
> image?
> Regards .. Subbu
>

> On 7/30/07, Martin Troielli <[hidden email]> wrote:
>> Hi Klaus,
>>
>> Thanks for the information. We have modified the VM in order to reduce  
>> the
>> GC activity inhibiting it until the fileIn processes are done, but had  
>> no
>> luck. The produced files were different with less differences.
>> We think we have to follow an approach similar to yours. We thought to
>> generate a serialized file with all the CompiledMethods we use, without
>> change the base image, merging them only when squeak starts up. We hope
>> that this process does not demand too much time, since we have also a  
>> lot
>> of resources to bring up at that time :S
>
> Maybe I'm alone in being unclear on this, but what is the root goal
> here?  Maybe there's a simpler way to achieve it.
>
> Avi
>

> Hi Avi,
>
> The main goal is to certify a software development. The certifier  
> must to check that a set of source files produce a binary output.
>
> We give them:
> 1 - The final image and VM
> 2 - The Smalltalk source files (fileOuts of our development)
> 3 - The VM C source files
> 4 - The base image
> 5 - A make script that compiles the VM, filesIn the smalltalk  
> source files on the base image and produces a final images and VM.
>
> They need to check that the two images, the one we give (1) and the  
> generated by our script (5) are the same. They check the  
> differences by doing a binary diff plus a hash over the files. They  
> only could allow changes refered to timestamps. They don't know  
> anything about Smalltalk...
>
> Best regards,
> Martin

>
> On Jul 31, 2007, at 6:52 AM, Martin Troielli wrote:
>
>> Hi Avi,
>>
>> The main goal is to certify a software development. The certifier must  
>> to check that a set of source files produce a binary output.
>>
>> We give them:
>> 1 - The final image and VM
>> 2 - The Smalltalk source files (fileOuts of our development)
>> 3 - The VM C source files
>> 4 - The base image
>> 5 - A make script that compiles the VM, filesIn the smalltalk source  
>> files on the base image and produces a final images and VM.
>>
>> They need to check that the two images, the one we give (1) and the  
>> generated by our script (5) are the same. They check the differences by  
>> doing a binary diff plus a hash over the files. They only could allow  
>> changes refered to timestamps. They don't know anything about  
>> Smalltalk...
>>
>> Best regards,
>> Martin
>
> 12 years back I had a client like this. Let's see if I remember...
>
> you could try doing
>
> | m |
> m := OrderedCollection new.
> SystemNavigation default allBehaviorsDo: [ :behavior |
> behavior selectors do: [ :sel |
> decompiled := Decompiler new decompile: sel in: behavior.
> m add: decompiled]].
> ^m
>
> where you sort the behaviors by the class name, then sort the selectors  
> and instead of collecting the decompiled value you
> stream the print string out to a stream. This should give you all the  
> source code for the image in a sorted order which you then can
> compare as text files.
>
> Think of it as decompiling the binary to see if the assembly  
> instructions are the same.
>
> What's missing is the globals and class variable values, but you might  
> not need those... ?
>
>
> Perhaps even a file out of all the methods in the image after the build  
> might help?
>
> --
> ========================================================================
> ===
> John M. McIntosh <[hidden email]>
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ========================================================================
> ===
>
>
>