Smalltalk › Pharo › Pharo Smalltalk Users

Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

7 messages Options

Mariano Martinez Peck

Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

OK, this was was my experiment....

Image fresh with all my app and dependencies loades: 30MB

After using it for some days/weeks: 160MB.

SpaceTally new printSpaceAnalysis showed:

Class code space # instances inst space percent inst average size

ByteString 2785 413144 116244078 69.90 281.36

Array 3712 181772 8466668 5.10 46.58

ByteArray 8574 1319 8186802 4.90 6206.82

Bitmap 3653 303 6656340 4.00 21968.12

CompiledMethod 22467 90554 5685374 3.40 62.78

After executing ImageCleaner cleanUpForRelease: 36MB

Then...I searched which part of #cleanUpForRelease: was making the difference, and it was:

Smalltalk cleanUp: true except: #() confirming: false.

So now it was time to know WHICH class did the diference, so I modified

#cleanUp: aggressive except: exclusions confirming: aBool

in these lines:

"Run the cleanup code"

classes

do:[:aClass|

Transcript show: 'Image size before cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString.

aClass cleanUp: aggressive.

3 timesRepeat: [Smalltalk garbageCollect].

Smalltalk snapshot: true andQuit: false.

Transcript show: 'Image size after cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString.

]

displayingProgress: [:aClass| 'Cleaning up in ', aClass name].

I then opened a Transcript, and evaluated

Smalltalk cleanUp: true except: #() confirming: false.

I went to prepare Mate, and when I come back, the result was, of course:

"Image size after cleaning MCFileBasedRepository : 39744008"

That clean up ends up doing:

flushAllCaches

self allSubInstancesDo: [:ea | ea flushCache]

So it sends #flushCache to all instances of MCHttpRepository and MCFileBasedRepository.

Now what I wanted to see if it there was a particular repo that could take most of the space (like package-cache).

And indeed, it was...I modified #flushCaches to:

flushAllCaches

| file |

file := 'repos.txt' asFileReference writeStream text.

self allSubInstancesDo: [:each |

file nextPutAll: 'Image size before cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString; cr.

each flushCache.

3 timesRepeat: [Smalltalk garbageCollect].

Smalltalk snapshot: true andQuit: false.

file nextPutAll: 'Image size after cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString;cr.

file flush; close.

And then I looked in the 'repos.txt' file. My package cache repo cleaned 60 MB. Glorp cleaned 35MB. Seaside30 cleaned 10MB.

So...cleaning cache of just 3 repos frees approx 100MB.

The question is....can we flush the cache safely? If they are called "cache", then I guess yes, we can.

Thoughts?

Thanks,

On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck <[hidden email]> wrote:

Guys, I have images also with seaside, magritte, glorp, postgresV2, etc and it is also around 200MB.
I will try to do some research today and let you know.

Cheers,

On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker <[hidden email]> wrote:

On Jul 30, 2013, at 1:49 PM, "[hidden email]" <[hidden email]> wrote:

> the changes file contained passwords and I replaced the text. So offsets may be wrong due to that.
>

Yes, the first thing I wanted to do is to recompile everything. Does not work.

> Memorymonitor is not doing fanct stuff. It just counts instances.
>

Yes, but maybe it holds on to these instances?

Marcus

--
Mariano
http://marianopeck.wordpress.com

--
Mariano
http://marianopeck.wordpress.com

Stéphane Ducasse

Re: [Pharo-dev] Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.

Stef

On Jul 31, 2013, at 8:53 PM, Mariano Martinez Peck <[hidden email]> wrote:

OK, this was was my experiment....

Image fresh with all my app and dependencies loades: 30MB

After using it for some days/weeks: 160MB.

SpaceTally new printSpaceAnalysis showed:

Class code space # instances inst space percent inst average size
ByteString 2785 413144 116244078 69.90 281.36

Array 3712 181772 8466668 5.10 46.58
ByteArray 8574 1319 8186802 4.90 6206.82

Bitmap 3653 303 6656340 4.00 21968.12
CompiledMethod 22467 90554 5685374 3.40 62.78

After executing ImageCleaner cleanUpForRelease: 36MB

Then...I searched which part of #cleanUpForRelease: was making the difference, and it was:

Smalltalk cleanUp: true except: #() confirming: false.

So now it was time to know WHICH class did the diference, so I modified
#cleanUp: aggressive except: exclusions confirming: aBool

in these lines:

"Run the cleanup code"
classes
do:[:aClass|

Transcript show: 'Image size before cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString.
aClass cleanUp: aggressive.

3 timesRepeat: [Smalltalk garbageCollect].
Smalltalk snapshot: true andQuit: false.
Transcript show: 'Image size after cleaning ', aClass name, ' : ', Smalltalk imagePath asFileReference size asString.

]
displayingProgress: [:aClass| 'Cleaning up in ', aClass name].

I then opened a Transcript, and evaluated

Smalltalk cleanUp: true except: #() confirming: false.

I went to prepare Mate, and when I come back, the result was, of course:

"Image size after cleaning MCFileBasedRepository : 39744008"

That clean up ends up doing:

flushAllCaches

self allSubInstancesDo: [:ea | ea flushCache]

So it sends #flushCache to all instances of MCHttpRepository and MCFileBasedRepository.

Now what I wanted to see if it there was a particular repo that could take most of the space (like package-cache).
And indeed, it was...I modified #flushCaches to:

flushAllCaches

| file |
file := 'repos.txt' asFileReference writeStream text.
self allSubInstancesDo: [:each |

file nextPutAll: 'Image size before cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString; cr.

each flushCache.
3 timesRepeat: [Smalltalk garbageCollect].
Smalltalk snapshot: true andQuit: false.

file nextPutAll: 'Image size after cleaning ', each printString, ' : ', Smalltalk imagePath asFileReference size asString;cr.

].
file flush; close.

And then I looked in the 'repos.txt' file. My package cache repo cleaned 60 MB. Glorp cleaned 35MB. Seaside30 cleaned 10MB.

So...cleaning cache of just 3 repos frees approx 100MB.

The question is....can we flush the cache safely? If they are called "cache", then I guess yes, we can.

Thoughts?

Thanks,

On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck <[hidden email]> wrote:

Guys, I have images also with seaside, magritte, glorp, postgresV2, etc and it is also around 200MB.
I will try to do some research today and let you know.

Cheers,

On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker <[hidden email]> wrote:

On Jul 30, 2013, at 1:49 PM, "[hidden email]" <[hidden email]> wrote:

> the changes file contained passwords and I replaced the text. So offsets may be wrong due to that.
>

Yes, the first thing I wanted to do is to recompile everything. Does not work.

> Memorymonitor is not doing fanct stuff. It just counts instances.
>

Yes, but maybe it holds on to these instances?

Marcus

--
Mariano
http://marianopeck.wordpress.com

--
Mariano
http://marianopeck.wordpress.com

Marcus Denker-4

Re: [Pharo-dev] Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <[hidden email]> wrote:

> since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
>

This is yet another thing.

Monticello keep all history data of all package in image, this is not a cache. When you delete it, your package loses
it's history and you can not merge anymore.

MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].

We did that on Pharo2 release and it saved 8MB (nearly a third of the image).

Marcus

signature.asc (210 bytes) Download Attachment

Goubier Thierry

Re: [Pharo-dev] Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

In reply to this post by Stéphane Ducasse

Hi Stef,

With Dale, we're working on a FileTree / GitFileTree without ancestors
info in the repository [1] :) ...

... but recreated on the fly by GitFileTree for the Monticello history
browsing tools [2] :(

[1] is underway and ...
[2] is already working in
https://github.com/dalehenrich/filetree/tree/pharo2.0/repository/MonticelloFileTree-Git.package

Thierry

Le 01/08/2013 08:56, Stéphane Ducasse a écrit :
> since years marcus is telling that MC storing ancestor information is
> doomed but we do not have something to really replace it.
>
> Stef

> On Jul 31, 2013, at 8:53 PM, Mariano Martinez Peck
> <[hidden email] <mailto:[hidden email]>> wrote:
>
>> OK, this was was my experiment....
>>
>> Image fresh with all my app and dependencies loades: 30MB
>>
>> After using it for some days/weeks: 160MB.
>>
>> SpaceTally new printSpaceAnalysis showed:
>>
>> Class code space # instances
>> inst space percent inst average size
>> ByteString 2785 413144
>> 116244078 69.90 281.36
>> Array 3712 181772
>> 8466668 5.10 46.58
>> ByteArray 8574 1319
>> 8186802 4.90 6206.82
>> Bitmap 3653 303
>> 6656340 4.00 21968.12
>> CompiledMethod 22467 90554
>> 5685374 3.40 62.78
>>
>>
>> After executing ImageCleaner cleanUpForRelease: 36MB
>>
>> Then...I searched which part of #cleanUpForRelease: was making the
>> difference, and it was:
>>
>> Smalltalk cleanUp: true except: #() confirming: false.
>>
>> So now it was time to know WHICH class did the diference, so I modified
>> #cleanUp: aggressive except: exclusions confirming: aBool
>>
>> in these lines:
>>
>> "Run the cleanup code"
>> classes
>> do:[:aClass|
>> Transcript show: 'Image size before cleaning ', aClass name, ' : ',
>> Smalltalk imagePath asFileReference size asString.
>> aClass cleanUp: aggressive.
>> 3 timesRepeat: [Smalltalk garbageCollect].
>> Smalltalk snapshot: true andQuit: false.
>> Transcript show: 'Image size after cleaning ', aClass name, ' : ',
>> Smalltalk imagePath asFileReference size asString.
>> ]
>> displayingProgress: [:aClass| 'Cleaning up in ', aClass name].
>> I then opened a Transcript, and evaluated
>>
>> Smalltalk cleanUp: true except: #() confirming: false.
>>
>> I went to prepare Mate, and when I come back, the result was, of course:
>>
>> "Image size after cleaning MCFileBasedRepository : 39744008"
>>
>> That clean up ends up doing:
>>
>> flushAllCaches
>> self allSubInstancesDo: [:ea | ea flushCache]
>> So it sends #flushCache to all instances of MCHttpRepository and
>> MCFileBasedRepository.
>>
>> Now what I wanted to see if it there was a particular repo that could
>> take most of the space (like package-cache).
>> And indeed, it was...I modified #flushCaches to:
>>
>> flushAllCaches
>> | file |
>> file := 'repos.txt' asFileReference writeStream text.
>> self allSubInstancesDo: [:each |
>> file nextPutAll: 'Image size before cleaning ', each printString, ' :
>> ', Smalltalk imagePath asFileReference size asString; cr.
>> each flushCache.
>> 3 timesRepeat: [Smalltalk garbageCollect].
>> Smalltalk snapshot: true andQuit: false.
>> file nextPutAll: 'Image size after cleaning ', each printString, ' :
>> ', Smalltalk imagePath asFileReference size asString;cr.
>> ].
>> file flush; close.
>> And then I looked in the 'repos.txt' file. My package cache repo
>> cleaned 60 MB. Glorp cleaned 35MB. Seaside30 cleaned 10MB.
>> So...cleaning cache of just 3 repos frees approx 100MB.
>>
>> The question is....can we flush the cache safely? If they are called
>> "cache", then I guess yes, we can.
>>
>> Thoughts?
>>
>> Thanks,
>>
>>
>>
>> On Wed, Jul 31, 2013 at 10:48 AM, Mariano Martinez Peck
>> <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> Guys, I have images also with seaside, magritte, glorp,
>> postgresV2, etc and it is also around 200MB.
>> I will try to do some research today and let you know.
>>
>> Cheers,
>>
>>
>> On Tue, Jul 30, 2013 at 8:55 AM, Marcus Denker
>> <[hidden email] <mailto:[hidden email]>> wrote:
>>
>>
>> On Jul 30, 2013, at 1:49 PM, "[hidden email]
>> <mailto:[hidden email]>" <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> > the changes file contained passwords and I replaced the
>> text. So offsets may be wrong due to that.
>> >
>> Yes, the first thing I wanted to do is to recompile
>> everything. Does not work.
>>
>> > Memorymonitor is not doing fanct stuff. It just counts
>> instances.
>> >
>> Yes, but maybe it holds on to these instances?
>>
>> Marcus
>>
>>
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
>>
>>
>>
>>
>> --
>> Mariano
>> http://marianopeck.wordpress.com <http://marianopeck.wordpress.com/>
>

--
Thierry Goubier
CEA list
Laboratoire des Fondations des Systèmes Temps Réel Embarqués
91191 Gif sur Yvette Cedex
France
Phone/Fax: +33 (0) 1 69 08 32 92 / 83 95

philippeback

Re: [Pharo-dev] Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

In reply to this post by Marcus Denker-4

This trimmed a couple of megs indeed.

Then I did an ImageCleaner cleanUpForRelease.

and saved.

But nothing terrific as the size continues to grow.

Need to see where those MorphExtensions are coming from...

Phil

On Thu, Aug 1, 2013 at 9:33 AM, Marcus Denker <[hidden email]> wrote:

On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <[hidden email]> wrote:

> since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
>

This is yet another thing.

Monticello keep all history data of all package in image, this is not a cache. When you delete it, your package loses
it's history and you can not merge anymore.

MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].

We did that on Pharo2 release and it saved 8MB (nearly a third of the image).

Marcus

Mariano Martinez Peck

Re: [Pharo-dev] Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

In reply to this post by Marcus Denker-4

On Thu, Aug 1, 2013 at 4:33 AM, Marcus Denker <[hidden email]> wrote:

On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <[hidden email]> wrote:

> since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
>

This is yet another thing.

Yes, I was talking about the cache, not the ancestors.

I think that removing the cache is indeed safe, isn't it?

Monticello keep all history data of all package in image, this is not a cache. When you delete it, your package loses
it's history and you can not merge anymore.

MCVersionInfo allInstances do: [ :each | each instVarNamed: 'ancestors' put: nil ].

We did that on Pharo2 release and it saved 8MB (nearly a third of the image).

Marcus

--
Mariano
http://marianopeck.wordpress.com

Marcus Denker-4

Re: [Pharo-dev] Large images reasons [WAS] Re: Pharo 2.0 with Seaside + DBXTalk + GlorpDBX + Magritte 3 + TWBS is getting slower and slower

On Aug 1, 2013, at 2:12 PM, Mariano Martinez Peck <[hidden email]> wrote:

On Thu, Aug 1, 2013 at 4:33 AM, Marcus Denker <[hidden email]> wrote:

On Aug 1, 2013, at 8:56 AM, Stéphane Ducasse <[hidden email]> wrote:

> since years marcus is telling that MC storing ancestor information is doomed but we do not have something to really replace it.
>

This is yet another thing.

Yes, I was talking about the cache, not the ancestors.
I think that removing the cache is indeed safe, isn't it?

Yes!

Marcus

signature.asc (210 bytes) Download Attachment