image segments

Squeak 5.0 still has ImageSegments (class comment below). Saving and
loading of  projects currently is blocked in Squeak because of an
unfinished name space (environments implementation).

http://wiki.squeak.org/squeak/6218

And there is Fuel
http://rmod.inria.fr/web/software/Fuel

----------

ImageSegment

I represent a segment of Squeak address space.  I am created from an
array of root objects.  After storing, my segment contains a binary
encoding of every object accessible from my roots but not otherwise
accessible from anywhere else in the system.  My segment contains
outward pointers that are indices into my table of outPointers.
On load my segment is converted back into objects and becommed
into an Array of the loaded objects, so they can be enumerated.

        The main use of ImageSegments is to store Projects.  A dummy
version of SmartRefStream traverses the Project.  Everything it finds
is classified as either an object that is owned by the project (only
pointed to inside the project), or an object outside the project that
is pointed to from inside the project.  The objects that are
completely owned by the project are compressed into pure binary form
in an ImageSegment.  The outside objects are put in the 'outPointers'
array.  The entire ImageSegment (binary part plus outPointers) is
encoded in a SmartRefStream, and saved on the disk.  (aProject
exportSegmentWithChangeSet:fileName:directory:) calls (anImageSegment
writeForExportWithSources:inDirectory:changeSet:).

        Note that every object inside the project is put into the
segment's arrayOfRoots.  This is because a dummy SmartRefStream to
scan the project, in order to make intelligent decisions about what
belongs in the project.
        See Project's class comment for what messages are sent to
objects as they are unpacked in a new image.

---- Older Details ------

        The primary kind of image segment is an Export Segment.  It
can be saved on a server and read into a completely different Squeak
image.
Old way to create one:
(ImageSegment new copyFromRootsForExport: (Array with: Baz with: Baz class))
                writeForExport: 'myFile.extSeg'.
Old way to create one for a project:
        (Project named: 'Play With Me - 3') exportSegment.
To read it into another image:  Select 'myFile.extSeg' in a FileList,
Menu 'load as project'.  It will install its classes automatically.
If you need to see the roots array, it is temporarily stored in
(SmartRefStream scannedObject).

Most of 'states' of an ImageSegment are not used to export a project,
and have been abandoned.

        When a segment is written out onto a file, it goes in a
folder called <image name>_segs.  If your image is called
"Squeak2.6.image", the folder "Squeak2.6_segs" must accompany the
image whenever your move, copy, or rename it.
        Whenever a Class is in arrayOfRoots, its class (aClass class)
must also be in the arrayOfRoots.
        There are two kinds of image segments.  Normal image segments
are a piece of a specific Squeak image, and can only be read back
into that image.  The image holds the array of outPointers that are
necessary to turn the bits in the file into objects.
        To put out a normal segment that holds a Project (not the
current project), execute (Project named: 'xxx') storeSegment.


arrayOfRoots    The objects that head the tree we will trace.
segment                 The WordArray of raw bits of all objects in the tree.
outPointers             Oops of all objects outside the segment
pointed to from inside.
state                   (see below)
segmentName     Its basic name.  Often the name of a Project.
fileName                The local name of the file.  'Foo-23.seg'
userRootCnt             number of roots submitted by caller.  Extras
are added in preparation for saving.

state that an ImageSegment may exist in...

#activeCopy                     (has been copied, with the intent to
become active)
arrayOfRoots, segment, and outPointers have been created by
copyFromRoots:.  The tree of objects has been encoded in the segment,
but those objects are still present in the Squeak system.

#active                         (segment is actively holding objects)
The segment is now the only holder of tree of objects.  Each of the
original roots has been transmuted into an ImageSegmentRootStub that
refers back to this image segment.  The original objects in the
segment will all be garbageCollected.

#onFile
The segment has been written out to a file and replaced by a file
pointer.  Only ImageSegmentRootStubs and the array of outPointers
remains in the image.  To get this far:
(ImageSegment new copyFromRoots: (Array with: Baz with: Baz class))
                writeToFile: 'myFile.seg'.

#inactive
The segment has been brought back into memory and turned back into
objects.  rootsArray is set, but the segment is invalid.

#onFileWithSymbols
The segment has been written out to a file, along with the text of
all the symbols in the outPointers array, and replaced by a file
pointer.  This reduces the size of the outPointers array, and also
allows the system to reclaim any symbols that are not referred to
from elsewhere in the image.  The specific format used is that of a
literal array as follows:
        #(symbol1 symbol2 # symbol3 symbol4 'symbolWithSpaces' # symbol5).
In this case, the original outPointers array was 8 long, but the
compacted table of outPointers retains only two entries.  These get
inserted in place of the #'s in the array of symbols after it is read
back in.  Symbols with embedded spaces or other strange characters
are written as strings, and converted back to symbols when read back
in.  The symbol # is never written out.
        NOTE: All IdentitySets or dictionaries must be rehashed when
being read back from this format.  The symbols are effectively
internal.  (No, not if read back into same image.  If a different
image, then use #imported.  -tk)

#imported
The segment is on an external file or just read in from one.  The
segment and outPointers are meant to be read into a foreign image.
In this form, the image segment can be read from a URL, and
installed.  A copy of the original array of root objects is
constructed, with former outPointers bound to existing objects in the
host system.
        (Any Class inside the segment MUST be in the arrayOfRoots.
This is so its association can be inserted into Smalltalk.  The
class's metaclass must be in roots also.  Methods that are in
outPointers because blocks point at them, were found and added to the
roots.
        All IdentitySets and dictionaries are rehashed when being
read back from exported segments.)


To discover why only some of the objects in a project are being
written out, try this (***Destructive Test***).  This breaks lots of
backpointers in the target project, and puts up an array of
suspicious objects, a list of the classes of the outPointers, and a
debugger.
"Close any transcripts in the target project"
World currentHand objectToPaste ifNotNil: [
        self inform: 'Hand is holding a Morph in its paste buffer:\' withCRs,
                World currentHand objectToPaste printString].
PV := Project named: 'xxxx'.
(IS := ImageSegment new) findRogueRootsImSeg:
        (Array with: PV world presenter with: PV world).
IS findOwnersOutPtrs.   "Optionally: write a file with owner chains"
"Quit and DO NOT save"

When an export image segment is brought into an image, it is like an
image starting up.  Certain startUp messages need to be run.  These
are byte and word reversals for nonPointer data that comes from a
machine of the opposite endianness.  #startUpProc passes over all
objects in the segment, and:
        The first time an instance of class X is encountered, (msg _
X startUpFrom: anImageSegment) is sent.  If msg is nil, the usual
case, it means that instances of X do not need special work.  X is
included in the IdentitySet, noStartUpNeeded.  If msg is not nil,
store it in the dictionary, startUps (aClass -> aMessage).
        When a later instance of X is encountered, if X is in
noStartUpNeeded, do nothing.  If X is in startUps, send the message
to the instance.  Typically this is a message like #swapShortObjects.
        Every class that implements #startUp, should see if it needs
a parallel implementation of #startUpFrom:.

On 11/26/15, Luciano Notarfrancesco <[hidden email]> wrote:
> I don't see image segments in Cuis... I don't even know if they still exist
> in Squeak... Anyone is still using it?
>
> It would be cool to have it at least as a package. If I remember correctly
> it was very simple, like two primitives to load and save, but I think it
> depended on something like mark and sweep or some particular kind of
> garbage collector and it might not work in new VMs.
>

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

And this seems to cover what you want to do

How to use an ImageSegment to determine memory usage
http://wiki.squeak.org/squeak/1260



On 11/26/15, H. Hirzel <[hidden email]> wrote:

> That's interesting,
>
> see also http://wiki.squeak.org/squeak/2316 with a reference to an
> article evaluation ImageSegments at the end.
>
>
>
> On 11/26/15, Luciano Notarfrancesco <[hidden email]> wrote:
>> Cool, maybe it's possible to make it into a package, I'll take a look.
>>
>> I've never used it to store Projects, tho..  I used before to store
>> things
>> that must be loaded very fast on demand and can be forgotten once there
>> are
>> no more references to them.. like image fragments that compose a map of
>> the
>> world, and small sound fragments that are used to synthesize speech
>> (diphones). Yesterday I just wanted to know how big are my objects, and I
>> wanted to make some experiments to see how the bit-size of my objects
>> grow
>> as the problems I'm modeling grow by some other measure (degree of
>> polynomials, size of matrices are the most basic) and to know how much
>> memory are my algorithms actually using at every step... image segments
>> are
>> ideal for this, you can use them to know exactly how much memory would be
>> freed if a reference to a given object were forgotten.
>>
>>
>> On Thu, Nov 26, 2015 at 7:36 AM, H. Hirzel <[hidden email]>
>> wrote:
>>
>>> Squeak 5.0 still has ImageSegments (class comment below). Saving and
>>> loading of  projects currently is blocked in Squeak because of an
>>> unfinished name space (environments implementation).
>>>
>>> http://wiki.squeak.org/squeak/6218
>>>
>>> And there is Fuel
>>> http://rmod.inria.fr/web/software/Fuel
>>>
>>> ----------
>>>
>>> ImageSegment
>>>
>>> I represent a segment of Squeak address space.  I am created from an
>>> array of root objects.  After storing, my segment contains a binary
>>> encoding of every object accessible from my roots but not otherwise
>>> accessible from anywhere else in the system.  My segment contains
>>> outward pointers that are indices into my table of outPointers.
>>> On load my segment is converted back into objects and becommed
>>> into an Array of the loaded objects, so they can be enumerated.
>>>
>>>         The main use of ImageSegments is to store Projects.  A dummy
>>> version of SmartRefStream traverses the Project.  Everything it finds
>>> is classified as either an object that is owned by the project (only
>>> pointed to inside the project), or an object outside the project that
>>> is pointed to from inside the project.  The objects that are
>>> completely owned by the project are compressed into pure binary form
>>> in an ImageSegment.  The outside objects are put in the 'outPointers'
>>> array.  The entire ImageSegment (binary part plus outPointers) is
>>> encoded in a SmartRefStream, and saved on the disk.  (aProject
>>> exportSegmentWithChangeSet:fileName:directory:) calls (anImageSegment
>>> writeForExportWithSources:inDirectory:changeSet:).
>>>
>>>         Note that every object inside the project is put into the
>>> segment's arrayOfRoots.  This is because a dummy SmartRefStream to
>>> scan the project, in order to make intelligent decisions about what
>>> belongs in the project.
>>>         See Project's class comment for what messages are sent to
>>> objects as they are unpacked in a new image.
>>>
>>> ---- Older Details ------
>>>
>>>         The primary kind of image segment is an Export Segment.  It
>>> can be saved on a server and read into a completely different Squeak
>>> image.
>>> Old way to create one:
>>> (ImageSegment new copyFromRootsForExport: (Array with: Baz with: Baz
>>> class))
>>>                 writeForExport: 'myFile.extSeg'.
>>> Old way to create one for a project:
>>>         (Project named: 'Play With Me - 3') exportSegment.
>>> To read it into another image:  Select 'myFile.extSeg' in a FileList,
>>> Menu 'load as project'.  It will install its classes automatically.
>>> If you need to see the roots array, it is temporarily stored in
>>> (SmartRefStream scannedObject).
>>>
>>> Most of 'states' of an ImageSegment are not used to export a project,
>>> and have been abandoned.
>>>
>>>         When a segment is written out onto a file, it goes in a
>>> folder called <image name>_segs.  If your image is called
>>> "Squeak2.6.image", the folder "Squeak2.6_segs" must accompany the
>>> image whenever your move, copy, or rename it.
>>>         Whenever a Class is in arrayOfRoots, its class (aClass class)
>>> must also be in the arrayOfRoots.
>>>         There are two kinds of image segments.  Normal image segments
>>> are a piece of a specific Squeak image, and can only be read back
>>> into that image.  The image holds the array of outPointers that are
>>> necessary to turn the bits in the file into objects.
>>>         To put out a normal segment that holds a Project (not the
>>> current project), execute (Project named: 'xxx') storeSegment.
>>>
>>>
>>> arrayOfRoots    The objects that head the tree we will trace.
>>> segment                 The WordArray of raw bits of all objects in the
>>> tree.
>>> outPointers             Oops of all objects outside the segment
>>> pointed to from inside.
>>> state                   (see below)
>>> segmentName     Its basic name.  Often the name of a Project.
>>> fileName                The local name of the file.  'Foo-23.seg'
>>> userRootCnt             number of roots submitted by caller.  Extras
>>> are added in preparation for saving.
>>>
>>> state that an ImageSegment may exist in...
>>>
>>> #activeCopy                     (has been copied, with the intent to
>>> become active)
>>> arrayOfRoots, segment, and outPointers have been created by
>>> copyFromRoots:.  The tree of objects has been encoded in the segment,
>>> but those objects are still present in the Squeak system.
>>>
>>> #active                         (segment is actively holding objects)
>>> The segment is now the only holder of tree of objects.  Each of the
>>> original roots has been transmuted into an ImageSegmentRootStub that
>>> refers back to this image segment.  The original objects in the
>>> segment will all be garbageCollected.
>>>
>>> #onFile
>>> The segment has been written out to a file and replaced by a file
>>> pointer.  Only ImageSegmentRootStubs and the array of outPointers
>>> remains in the image.  To get this far:
>>> (ImageSegment new copyFromRoots: (Array with: Baz with: Baz class))
>>>                 writeToFile: 'myFile.seg'.
>>>
>>> #inactive
>>> The segment has been brought back into memory and turned back into
>>> objects.  rootsArray is set, but the segment is invalid.
>>>
>>> #onFileWithSymbols
>>> The segment has been written out to a file, along with the text of
>>> all the symbols in the outPointers array, and replaced by a file
>>> pointer.  This reduces the size of the outPointers array, and also
>>> allows the system to reclaim any symbols that are not referred to
>>> from elsewhere in the image.  The specific format used is that of a
>>> literal array as follows:
>>>         #(symbol1 symbol2 # symbol3 symbol4 'symbolWithSpaces' #
>>> symbol5).
>>> In this case, the original outPointers array was 8 long, but the
>>> compacted table of outPointers retains only two entries.  These get
>>> inserted in place of the #'s in the array of symbols after it is read
>>> back in.  Symbols with embedded spaces or other strange characters
>>> are written as strings, and converted back to symbols when read back
>>> in.  The symbol # is never written out.
>>>         NOTE: All IdentitySets or dictionaries must be rehashed when
>>> being read back from this format.  The symbols are effectively
>>> internal.  (No, not if read back into same image.  If a different
>>> image, then use #imported.  -tk)
>>>
>>> #imported
>>> The segment is on an external file or just read in from one.  The
>>> segment and outPointers are meant to be read into a foreign image.
>>> In this form, the image segment can be read from a URL, and
>>> installed.  A copy of the original array of root objects is
>>> constructed, with former outPointers bound to existing objects in the
>>> host system.
>>>         (Any Class inside the segment MUST be in the arrayOfRoots.
>>> This is so its association can be inserted into Smalltalk.  The
>>> class's metaclass must be in roots also.  Methods that are in
>>> outPointers because blocks point at them, were found and added to the
>>> roots.
>>>         All IdentitySets and dictionaries are rehashed when being
>>> read back from exported segments.)
>>>
>>>
>>> To discover why only some of the objects in a project are being
>>> written out, try this (***Destructive Test***).  This breaks lots of
>>> backpointers in the target project, and puts up an array of
>>> suspicious objects, a list of the classes of the outPointers, and a
>>> debugger.
>>> "Close any transcripts in the target project"
>>> World currentHand objectToPaste ifNotNil: [
>>>         self inform: 'Hand is holding a Morph in its paste buffer:\'
>>> withCRs,
>>>                 World currentHand objectToPaste printString].
>>> PV := Project named: 'xxxx'.
>>> (IS := ImageSegment new) findRogueRootsImSeg:
>>>         (Array with: PV world presenter with: PV world).
>>> IS findOwnersOutPtrs.   "Optionally: write a file with owner chains"
>>> "Quit and DO NOT save"
>>>
>>> When an export image segment is brought into an image, it is like an
>>> image starting up.  Certain startUp messages need to be run.  These
>>> are byte and word reversals for nonPointer data that comes from a
>>> machine of the opposite endianness.  #startUpProc passes over all
>>> objects in the segment, and:
>>>         The first time an instance of class X is encountered, (msg _
>>> X startUpFrom: anImageSegment) is sent.  If msg is nil, the usual
>>> case, it means that instances of X do not need special work.  X is
>>> included in the IdentitySet, noStartUpNeeded.  If msg is not nil,
>>> store it in the dictionary, startUps (aClass -> aMessage).
>>>         When a later instance of X is encountered, if X is in
>>> noStartUpNeeded, do nothing.  If X is in startUps, send the message
>>> to the instance.  Typically this is a message like #swapShortObjects.
>>>         Every class that implements #startUp, should see if it needs
>>> a parallel implementation of #startUpFrom:.
>>>
>>> On 11/26/15, Luciano Notarfrancesco <[hidden email]> wrote:
>>> > I don't see image segments in Cuis... I don't even know if they still
>>> exist
>>> > in Squeak... Anyone is still using it?
>>> >
>>> > It would be cool to have it at least as a package. If I remember
>>> correctly
>>> > it was very simple, like two primitives to load and save, but I think
>>> > it
>>> > depended on something like mark and sweep or some particular kind of
>>> > garbage collector and it might not work in new VMs.
>>> >
>>>
>>> _______________________________________________
>>> Cuis mailing list
>>> [hidden email]
>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>
>>
>

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org