Large difference in image size when loading configuration

Phil, what happens if you evaluate (and save after) to the big image:

MCFileBasedRepository flushAllCaches.

3 timesRepeat: [Smalltalk garbageCollect].

On Mon, Nov 11, 2013 at 8:30 AM, [hidden email] <[hidden email]> wrote:

[User.Techlab] → du -hs package-cache

5.3M    package-cache

Phil

On Mon, Nov 11, 2013 at 11:41 AM, Sven Van Caekenberghe <[hidden email]> wrote:

What is the total size of the package-cache, like du -hs ?
Could it be the same size of the difference in image size ?
That would mean that the contents of the packages themselves is cached in the image...

On 11 Nov 2013, at 10:39, [hidden email] wrote:

> I am experiencing the following while loading my configuration.
>
> REPO=http://www.smalltalkhub.com/mc/philippeback/HOWebStack/main
> ./pharo Pharo.image config $REPO ConfigurationOfHOWebStack --install=0.4
>
> Everything loads fine.
>
> But:
>
> with a package-cache/ empty, the final image is: 44.452.060 with a changes file of: 10.831.877
>
> with a primed package-cache (meaning, letting the mczs in place and starting with a fresh image), the final image is: 29.480.912 with a changes file of: 10.830.899
>
> That's quite a huge difference.
>
> I tried again to be sure (fresh image and empty package-cache, then fresh image only) and, weirdly enough, even if the difference in size was the same, the sizes themselves weren't.the same...
>
> 44.446.152 - 10.830.899
> 29.986.284 - 10.831.543
>
> Maybe that's due to a GC occurring differently between the two.
>
> But this gives the impression that one cannot load a base image, apply a configuration, and end up with the same image twice. Weird.
>
> Why is this difference so large in the first place ?
>
> Phil
>
>
>
>

--
Mariano
http://marianopeck.wordpress.com

On 11 Nov 2013, at 15:51, [hidden email] wrote:

> Image size goes back to:
>
> 29.068.612
>
> Looks like all package contents are cached in the image…

But by a factor 3 ??

> Phil
>
>
>
> On Mon, Nov 11, 2013 at 1:32 PM, Mariano Martinez Peck <[hidden email]> wrote:
> Phil, what happens if you evaluate (and save after) to the big image:
>
> MCFileBasedRepository flushAllCaches.
> 3 timesRepeat: [Smalltalk garbageCollect].
>
>
> On Mon, Nov 11, 2013 at 8:30 AM, [hidden email] <[hidden email]> wrote:
> [User.Techlab] → du -hs package-cache
> 5.3M    package-cache
>
> Phil
>
> On Mon, Nov 11, 2013 at 11:41 AM, Sven Van Caekenberghe <[hidden email]> wrote:
> What is the total size of the package-cache, like du -hs ?
> Could it be the same size of the difference in image size ?
> That would mean that the contents of the packages themselves is cached in the image...
>
> On 11 Nov 2013, at 10:39, [hidden email] wrote:
>
> > I am experiencing the following while loading my configuration.
> >
> > REPO=http://www.smalltalkhub.com/mc/philippeback/HOWebStack/main
> > ./pharo Pharo.image config $REPO ConfigurationOfHOWebStack --install=0.4
> >
> > Everything loads fine.
> >
> > But:
> >
> > with a package-cache/ empty, the final image is: 44.452.060 with a changes file of: 10.831.877
> >
> > with a primed package-cache (meaning, letting the mczs in place and starting with a fresh image), the final image is: 29.480.912 with a changes file of: 10.830.899
> >
> > That's quite a huge difference.
> >
> > I tried again to be sure (fresh image and empty package-cache, then fresh image only) and, weirdly enough, even if the difference in size was the same, the sizes themselves weren't.the same...
> >
> > 44.446.152 - 10.830.899
> > 29.986.284 - 10.831.543
> >
> > Maybe that's due to a GC occurring differently between the two.
> >
> > But this gives the impression that one cannot load a base image, apply a configuration, and end up with the same image twice. Weird.
> >
> > Why is this difference so large in the first place ?
> >
> > Phil
> >
> >
> >
> >
>
>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>

I had a look at the process browser and found this strange set of processes with semaphores and style in background.

How comes?

There is no browser open even.

See screenshot attached.

Phil

On Mon, Nov 11, 2013 at 5:10 PM, [hidden email] <[hidden email]> wrote:

Yeah, that's a tad weird.

I am quite concerned about that image ballooning effect.

I've got a dev image and it is now 185.515.948

I did the flush thing 

MCFileBasedRepository flushAllCaches.

3 timesRepeat: [Smalltalk garbageCollect].

and now, it is 158.740.484

SpaceTally new printSpaceAnalysis

shows:

Class                                          code space # instances  inst space     percent   inst average size

Array                                                3712      817192      31395576       19.30               38.42

Float                                               13047     1313593      15763116        9.70               12.00

ByteString                                           2785      378125      13078821        8.00               34.59

MorphExtension                                       3097      157557      10713876        6.60               68.00

Bitmap                                               3653        1975      10198648        6.30             5163.87

TextMethodLink                                        419      392396       9417504        5.80               24.00

Point                                                7105      711444       8537328        5.30               12.00

CompiledMethod                                      22467       82802       5215856        3.20               62.99

SHRange                                              1919      217316       4346320        2.70               20.00

Rectangle                                            8795      325385       3904620        2.40               12.00

TableLayoutProperties                                1169       49629       3573288        2.20               72.00

Semaphore                                             949      149195       2983900        1.80               20.00

A ton of MorphExtensions (which is a know problem).

but also quite some TextMethodLinks, which I do not understand.

SHRange, from styling things I guess, lots of remnants.

And quite a bunch of Semaphores, too much I think.

Tons of Arrays and floats. I do a lot of NeoCSV loads in the image.

Phil

On Mon, Nov 11, 2013 at 4:03 PM, Sven Van Caekenberghe <[hidden email]> wrote:

On 11 Nov 2013, at 15:51, [hidden email] wrote:

> Image size goes back to:
>
> 29.068.612
>
> Looks like all package contents are cached in the image…

But by a factor 3 ??

> Phil
>
>
>
> On Mon, Nov 11, 2013 at 1:32 PM, Mariano Martinez Peck <[hidden email]> wrote:
> Phil, what happens if you evaluate (and save after) to the big image:
>
> MCFileBasedRepository flushAllCaches.
> 3 timesRepeat: [Smalltalk garbageCollect].
>
>
> On Mon, Nov 11, 2013 at 8:30 AM, [hidden email] <[hidden email]> wrote:
> [User.Techlab] → du -hs package-cache
> 5.3M    package-cache
>
> Phil
>
> On Mon, Nov 11, 2013 at 11:41 AM, Sven Van Caekenberghe <[hidden email]> wrote:
> What is the total size of the package-cache, like du -hs ?
> Could it be the same size of the difference in image size ?
> That would mean that the contents of the packages themselves is cached in the image...
>
> On 11 Nov 2013, at 10:39, [hidden email] wrote:
>
> > I am experiencing the following while loading my configuration.
> >
> > REPO=http://www.smalltalkhub.com/mc/philippeback/HOWebStack/main
> > ./pharo Pharo.image config $REPO ConfigurationOfHOWebStack --install=0.4
> >
> > Everything loads fine.
> >
> > But:
> >
> > with a package-cache/ empty, the final image is: 44.452.060 with a changes file of: 10.831.877
> >
> > with a primed package-cache (meaning, letting the mczs in place and starting with a fresh image), the final image is: 29.480.912 with a changes file of: 10.830.899
> >
> > That's quite a huge difference.
> >
> > I tried again to be sure (fresh image and empty package-cache, then fresh image only) and, weirdly enough, even if the difference in size was the same, the sizes themselves weren't.the same...
> >
> > 44.446.152 - 10.830.899
> > 29.986.284 - 10.831.543
> >
> > Maybe that's due to a GC occurring differently between the two.
> >
> > But this gives the impression that one cannot load a base image, apply a configuration, and end up with the same image twice. Weird.
> >
> > Why is this difference so large in the first place ?
> >
> > Phil
> >
> >
> >
> >
>
>
>
>
>
>
> --
> Mariano
> http://marianopeck.wordpress.com
>

>>
>
> It seems that for each repository instance up to 512 (!) versions can be cached in memory (instance variable cache), including all their version info, full ancestry until the middle ages, and definitions, including source code and meta data, in the snapshot. This is really massive.

marcus is complaining about it since years :)

>
> The package-cache already contains this data, non-parsed, I know.
>
> Still, the in memory cache should be managed way better, using LRU or something like that.