Smalltalk › Pharo › Pharo Smalltalk Users

Changes file size

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

18 messages Options

Esteban A. Maringolo

Changes file size

I know we're on our way to get rid of changes, but meanwhile after installing my Metacello config I end up with an image of 40megs and a changes file of 140 megs!!!

It doesn't load anything too esoteric: Glorp, Magritte, Seaside, Postgres, NeoCSV and my own packages.

I can run PharoChangesCondenser condense, but it takes forever! (I started it 20 minutes ago and it's still running).

Is there a way I can improve this situation?

Esteban A. Maringolo

Re: Changes file size

Well... after finishing the condensing, the changes file passed form 140 mb... to 280 mb :)

Esteban A. Maringolo

2014-10-28 3:08 GMT-03:00 Esteban A. Maringolo <[hidden email]>:

I know we're on our way to get rid of changes, but meanwhile after installing my Metacello config I end up with an image of 40megs and a changes file of 140 megs!!!

It doesn't load anything too esoteric: Glorp, Magritte, Seaside, Postgres, NeoCSV and my own packages.

I can run PharoChangesCondenser condense, but it takes forever! (I started it 20 minutes ago and it's still running).

Is there a way I can improve this situation?

Esteban A. Maringolo

Sven Van Caekenberghe-2

Re: Changes file size

Esteban,

The Reddit example's CI Job (https://ci.inria.fr/pharo-contribution/job/Reddit/) also loads Seaside and Glorp (two big packages) in Pharo 3 and it too results in a 110 Mb image and 142 Mb changes file. After condensing, that goes to 276 Mb !

I would say something is wrong here ;-)

Note that condensing changes on newly loaded code should not make much difference (in essence, all multiple versions of methods are reduced to 1).

I think we should create some issues out of this.

Sven

> On 28 Oct 2014, at 07:25, Esteban A. Maringolo <[hidden email]> wrote:
>
> Well... after finishing the condensing, the changes file passed form 140 mb... to 280 mb :)
>
>
>
> Esteban A. Maringolo
>
> 2014-10-28 3:08 GMT-03:00 Esteban A. Maringolo <[hidden email]>:
> I know we're on our way to get rid of changes, but meanwhile after installing my Metacello config I end up with an image of 40megs and a changes file of 140 megs!!!
>
> It doesn't load anything too esoteric: Glorp, Magritte, Seaside, Postgres, NeoCSV and my own packages.
>
> I can run PharoChangesCondenser condense, but it takes forever! (I started it 20 minutes ago and it's still running).
>
> Is there a way I can improve this situation?
>
> Esteban A. Maringolo
>

philippeback

Re: Changes file size

In reply to this post by Esteban A. Maringolo

ls -l ~jenkins/jobs/BaseWorker/workspace/*.changes

-rw-r--r-- 1 jenkins jenkins 244869 Oct 25 09:30 /var/lib/jenkins/jobs/BaseWorker/workspace/Pharo.changes

becoming

-rw-r--r-- 1 jenkins jenkins 265779 Oct 25 09:30 /var/lib/jenkins/jobs/BaseWorker/workspace/X-Builder.changes

becoming (couple packages like Seaside & friends)

-rw-r--r-- 1 jenkins jenkins 30127371 Oct 26 10:11 /var/lib/jenkins/jobs/BaseWorker/workspace/X-BaseWorker.changes

becoming (user code and customized packages)

-rw-r--r-- 1 jenkins jenkins 31774426 Oct 26 10:12 /var/lib/jenkins/jobs/BaseWorker/workspace/X-frontend.changes

Condensing that takes quite a while... I see a "Swapping source pointers" progress bar on the top left.

Looks like there are a lot of "become:" going on.

[ PharoChangesCondenser condense ] timeToRun 0:00:45:12.88 - in a VM under VMWare, i7 3.85Ghz host.

-rw-rw-r-- 1 philippeback philippeback 30008723 Oct 28 09:52 X-frontend-20141028_0848-tocondense-work-condensed.changes

Not that much growth on the changes file.

Working image has

-rw-rw-r-- 1 philippeback philippeback 85099190 Oct 28 08:48 X-frontend-20141028_0848-tocondense.changes

So, nice reduction and not growing.

Phil

Sven Van Caekenberghe-2

Re: Changes file size

Hmm, we need more tests and data points, I will try on Linux command line later on.

> On 28 Oct 2014, at 09:54, [hidden email] wrote:
>
>
> ls -l ~jenkins/jobs/BaseWorker/workspace/*.changes
>
> -rw-r--r-- 1 jenkins jenkins 244869 Oct 25 09:30 /var/lib/jenkins/jobs/BaseWorker/workspace/Pharo.changes
>
> becoming
>
> -rw-r--r-- 1 jenkins jenkins 265779 Oct 25 09:30 /var/lib/jenkins/jobs/BaseWorker/workspace/X-Builder.changes
>
> becoming (couple packages like Seaside & friends)
>
> -rw-r--r-- 1 jenkins jenkins 30127371 Oct 26 10:11 /var/lib/jenkins/jobs/BaseWorker/workspace/X-BaseWorker.changes
>
> becoming (user code and customized packages)
>
> -rw-r--r-- 1 jenkins jenkins 31774426 Oct 26 10:12 /var/lib/jenkins/jobs/BaseWorker/workspace/X-frontend.changes
>
> Condensing that takes quite a while... I see a "Swapping source pointers" progress bar on the top left.
> Looks like there are a lot of "become:" going on.
>
> [ PharoChangesCondenser condense ] timeToRun 0:00:45:12.88 - in a VM under VMWare, i7 3.85Ghz host.
>
> -rw-rw-r-- 1 philippeback philippeback 30008723 Oct 28 09:52 X-frontend-20141028_0848-tocondense-work-condensed.changes
>
> Not that much growth on the changes file.
>
> Working image has
>
> -rw-rw-r-- 1 philippeback philippeback 85099190 Oct 28 08:48 X-frontend-20141028_0848-tocondense.changes
>
> So, nice reduction and not growing.
>
> Phil
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

stepharo

Re: Changes file size

In reply to this post by Esteban A. Maringolo

Hi esteban

Which image are you using?
Do you get similar behavior with Pharo30?

Stef

Well... after finishing the condensing, the changes file passed form 140 mb... to 280 mb :)

Esteban A. Maringolo

2014-10-28 3:08 GMT-03:00 Esteban A. Maringolo <[hidden email]>:

I know we're on our way to get rid of changes, but meanwhile after installing my Metacello config I end up with an image of 40megs and a changes file of 140 megs!!!

It doesn't load anything too esoteric: Glorp, Magritte, Seaside, Postgres, NeoCSV and my own packages.

I can run PharoChangesCondenser condense, but it takes forever! (I started it 20 minutes ago and it's still running).

Is there a way I can improve this situation?

Esteban A. Maringolo

Esteban A. Maringolo

Re: Changes file size

Hi Steph,

I am using Pharo 3. I didn't try in it any other release.

I'm using the latest stable version and VM from the zeroconf scripts, runing the command from the command line via "eval" (though it doesn't seem to make any difference)

Regards!

El Tue Oct 28 2014 at 9:13:52 AM, stepharo <[hidden email]> escribió:

Hi esteban

Which image are you using?
Do you get similar behavior with Pharo30?

Stef

Well... after finishing the condensing, the changes file passed form 140 mb... to 280 mb :)

Esteban A. Maringolo

2014-10-28 3:08 GMT-03:00 Esteban A. Maringolo <[hidden email]>:

I know we're on our way to get rid of changes, but meanwhile after installing my Metacello config I end up with an image of 40megs and a changes file of 140 megs!!!

It doesn't load anything too esoteric: Glorp, Magritte, Seaside, Postgres, NeoCSV and my own packages.

I can run PharoChangesCondenser condense, but it takes forever! (I started it 20 minutes ago and it's still running).

Is there a way I can improve this situation?

Esteban A. Maringolo

Esteban A. Maringolo

Re: Changes file size

In reply to this post by Sven Van Caekenberghe-2

2014-10-28 5:41 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
> Esteban,
>
> The Reddit example's CI Job (https://ci.inria.fr/pharo-contribution/job/Reddit/) also loads Seaside and Glorp (two big packages) in Pharo 3 and it too results in a 110 Mb image and 142 Mb changes file. After condensing, that goes to 276 Mb !

> I would say something is wrong here ;-)

Either something is wrong, or we feature it and rename
PharoChangesCondenser>>#condense to PharoChangesExpander>>#expand :)

> Note that condensing changes on newly loaded code should not make much difference (in essence, all multiple versions of methods are reduced to 1).

> I think we should create some issues out of this.

This is true, however I find a changes of 140 megs to be MASSIVE.
I remember loading ~4000 classes in the order of 10^6 LOC in Dolphin, and
changes never got half that size. Sizes were ~28/55MB image/changes.
The same in VAST, any image beyond the 20MB was an alert of something
being leaked.

> Hmm, we need more tests and data points, I will try on Linux command line later on.

I ran in it in Linux (Ubuntu) through the command line.

Regards!

Sven Van Caekenberghe-2

Re: Changes file size

> On 28 Oct 2014, at 14:58, Esteban A. Maringolo <[hidden email]> wrote:
>
> 2014-10-28 5:41 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>> Esteban,
>>
>> The Reddit example's CI Job (https://ci.inria.fr/pharo-contribution/job/Reddit/) also loads Seaside and Glorp (two big packages) in Pharo 3 and it too results in a 110 Mb image and 142 Mb changes file. After condensing, that goes to 276 Mb !
>
>> I would say something is wrong here ;-)
>
> Either something is wrong, or we feature it and rename
> PharoChangesCondenser>>#condense to PharoChangesExpander>>#expand :)
>
>> Note that condensing changes on newly loaded code should not make much difference (in essence, all multiple versions of methods are reduced to 1).
>
>> I think we should create some issues out of this.
>
> This is true, however I find a changes of 140 megs to be MASSIVE.

I totally agree, this is unacceptable. However, I did some tests, and the problem is with Glorp (or any of its sub packages). I tried building both my Reddit and HP35 examples from scratch in Pharo 3 on Linux.

$ ./pharo reddit.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/Reddit/main ConfigurationOfReddit --install=stable

$ ./pharo hp35.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/HP35/main ConfigurationOfHP35 --install=stable --group=Web-UI

Both load Seaside and some other stuff, but only Reddit loads Glorp and PostgresV2.

I did a condense changes on the HP35 image, the one on the Reddit image ran out of physical memory (granted is was a small machine) !

Here are the resulting sizes:

$ ls -lah
total 337M
drwxr-xr-x 4 root root 4.0K Oct 28 15:09 .
drwx------ 12 root root 4.0K Oct 28 12:28 ..
-rw-r--r-- 1 root root 5.8M Oct 28 13:17 hp35.changes
-rw-r--r-- 1 root root 5.6M Oct 28 14:38 hp35-condense-test.changes
-rw-r--r-- 1 root root 5.8M Oct 28 14:27 hp35-condense-test.changes.bak
-rw-r--r-- 1 root root 29M Oct 28 14:38 hp35-condense-test.image
-rw-r--r-- 1 root root 29M Oct 28 13:17 hp35.image
drwxr-xr-x 2 root root 4.0K Oct 28 13:12 package-cache
-rwxr-xr-x 1 root root 367 Oct 28 12:29 pharo
-rw-rw-r-- 1 root root 265K Oct 24 14:42 Pharo.changes
-rw-rw-r-- 1 root root 21M Oct 24 14:42 Pharo.image
-rwxr-xr-x 1 root root 354 Oct 28 12:29 pharo-ui
drwxr-xr-x 3 root root 4.0K Oct 28 12:29 pharo-vm
-rw-r--r-- 1 root root 136M Oct 28 12:48 reddit.changes
-rw-r--r-- 1 root root 105M Oct 28 12:48 reddit.image

Note how the HP35 image and changes sizes are totally acceptable/normal, while the Reddit one explodes, with the difference being Glorp.

But I have no idea what causes this.

There is no way Glorp can generate 100 Mb changes, when Seaside is OK with 6 Mb.

> I remember loading ~4000 classes in the order of 10^6 LOC in Dolphin, and
> changes never got half that size. Sizes were ~28/55MB image/changes.
> The same in VAST, any image beyond the 20MB was an alert of something
> being leaked.
>
>> Hmm, we need more tests and data points, I will try on Linux command line later on.
>
> I ran in it in Linux (Ubuntu) through the command line.
>
>
> Regards!
>

Esteban A. Maringolo

Re: Changes file size

Just when I found a way to reduce the total size of the changes file, I
noticed the latest versions in the repository doesn't have this issue,
what a good way of wasting one's time learning about the guts of
Monticello :)

But just for the record:
The problem happened after installing into GlorpSession a
MCMethodDefinition (#dropTables:), I logged every single method
installation, and the point where things went bananas was after
installing that culprit method:

After a MCMethodDefinition(dropTables): 1201686 bytes (~1.2Mb)
After a MCMethodDefinition(dropTables:): 135419895 bytes (~135Mb)

I did a fileOut, deleted the method, and then file it in again, and
after versioning (and reloading) the changes increased ~1Mb after
installing Glorp.

I didn't find anything particularly suspicious about it, not in the
source code nor in the bytecodes.

Now after running PharoChangesCondenser condense the changes file
effectively shrinks.

Regards!

Esteban A. Maringolo

2014-10-28 12:31 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:

>
>> On 28 Oct 2014, at 14:58, Esteban A. Maringolo <[hidden email]> wrote:
>>
>> 2014-10-28 5:41 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>>> Esteban,
>>>
>>> The Reddit example's CI Job (https://ci.inria.fr/pharo-contribution/job/Reddit/) also loads Seaside and Glorp (two big packages) in Pharo 3 and it too results in a 110 Mb image and 142 Mb changes file. After condensing, that goes to 276 Mb !
>>
>>> I would say something is wrong here ;-)
>>
>> Either something is wrong, or we feature it and rename
>> PharoChangesCondenser>>#condense to PharoChangesExpander>>#expand :)
>>
>>> Note that condensing changes on newly loaded code should not make much difference (in essence, all multiple versions of methods are reduced to 1).
>>
>>> I think we should create some issues out of this.
>>
>> This is true, however I find a changes of 140 megs to be MASSIVE.
>
> I totally agree, this is unacceptable. However, I did some tests, and the problem is with Glorp (or any of its sub packages). I tried building both my Reddit and HP35 examples from scratch in Pharo 3 on Linux.
>
> $ ./pharo reddit.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/Reddit/main ConfigurationOfReddit --install=stable
>
> $ ./pharo hp35.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/HP35/main ConfigurationOfHP35 --install=stable --group=Web-UI
>
> Both load Seaside and some other stuff, but only Reddit loads Glorp and PostgresV2.
>
> I did a condense changes on the HP35 image, the one on the Reddit image ran out of physical memory (granted is was a small machine) !
>
> Here are the resulting sizes:
>
> $ ls -lah
> total 337M
> drwxr-xr-x 4 root root 4.0K Oct 28 15:09 .
> drwx------ 12 root root 4.0K Oct 28 12:28 ..
> -rw-r--r-- 1 root root 5.8M Oct 28 13:17 hp35.changes
> -rw-r--r-- 1 root root 5.6M Oct 28 14:38 hp35-condense-test.changes
> -rw-r--r-- 1 root root 5.8M Oct 28 14:27 hp35-condense-test.changes.bak
> -rw-r--r-- 1 root root 29M Oct 28 14:38 hp35-condense-test.image
> -rw-r--r-- 1 root root 29M Oct 28 13:17 hp35.image
> drwxr-xr-x 2 root root 4.0K Oct 28 13:12 package-cache
> -rwxr-xr-x 1 root root 367 Oct 28 12:29 pharo
> -rw-rw-r-- 1 root root 265K Oct 24 14:42 Pharo.changes
> -rw-rw-r-- 1 root root 21M Oct 24 14:42 Pharo.image
> -rwxr-xr-x 1 root root 354 Oct 28 12:29 pharo-ui
> drwxr-xr-x 3 root root 4.0K Oct 28 12:29 pharo-vm
> -rw-r--r-- 1 root root 136M Oct 28 12:48 reddit.changes
> -rw-r--r-- 1 root root 105M Oct 28 12:48 reddit.image
>
> Note how the HP35 image and changes sizes are totally acceptable/normal, while the Reddit one explodes, with the difference being Glorp.
>
> But I have no idea what causes this.
>
> There is no way Glorp can generate 100 Mb changes, when Seaside is OK with 6 Mb.
>
>> I remember loading ~4000 classes in the order of 10^6 LOC in Dolphin, and
>> changes never got half that size. Sizes were ~28/55MB image/changes.
>> The same in VAST, any image beyond the 20MB was an alert of something
>> being leaked.
>>
>>> Hmm, we need more tests and data points, I will try on Linux command line later on.
>>
>> I ran in it in Linux (Ubuntu) through the command line.
>>
>>
>> Regards!
>>
>
>

Sven Van Caekenberghe-2

Re: Changes file size

Esteban,

You might be on something important there.

I always found it strange what effect loading Glorp had on an image, blowing it up.

I can't say I understand though, best make it into an issue.

Sven

> On 10 Feb 2015, at 00:22, Esteban A. Maringolo <[hidden email]> wrote:
>
> Just when I found a way to reduce the total size of the changes file, I
> noticed the latest versions in the repository doesn't have this issue,
> what a good way of wasting one's time learning about the guts of
> Monticello :)
>
> But just for the record:
> The problem happened after installing into GlorpSession a
> MCMethodDefinition (#dropTables:), I logged every single method
> installation, and the point where things went bananas was after
> installing that culprit method:
>
> After a MCMethodDefinition(dropTables): 1201686 bytes (~1.2Mb)
> After a MCMethodDefinition(dropTables:): 135419895 bytes (~135Mb)
>
> I did a fileOut, deleted the method, and then file it in again, and
> after versioning (and reloading) the changes increased ~1Mb after
> installing Glorp.
>
> I didn't find anything particularly suspicious about it, not in the
> source code nor in the bytecodes.
>
> Now after running PharoChangesCondenser condense the changes file
> effectively shrinks.
>
> Regards!
>
>
> Esteban A. Maringolo
>
>
> 2014-10-28 12:31 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>>
>>> On 28 Oct 2014, at 14:58, Esteban A. Maringolo <[hidden email]> wrote:
>>>
>>> 2014-10-28 5:41 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>>>> Esteban,
>>>>
>>>> The Reddit example's CI Job (https://ci.inria.fr/pharo-contribution/job/Reddit/) also loads Seaside and Glorp (two big packages) in Pharo 3 and it too results in a 110 Mb image and 142 Mb changes file. After condensing, that goes to 276 Mb !
>>>
>>>> I would say something is wrong here ;-)
>>>
>>> Either something is wrong, or we feature it and rename
>>> PharoChangesCondenser>>#condense to PharoChangesExpander>>#expand :)
>>>
>>>> Note that condensing changes on newly loaded code should not make much difference (in essence, all multiple versions of methods are reduced to 1).
>>>
>>>> I think we should create some issues out of this.
>>>
>>> This is true, however I find a changes of 140 megs to be MASSIVE.
>>
>> I totally agree, this is unacceptable. However, I did some tests, and the problem is with Glorp (or any of its sub packages). I tried building both my Reddit and HP35 examples from scratch in Pharo 3 on Linux.
>>
>> $ ./pharo reddit.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/Reddit/main ConfigurationOfReddit --install=stable
>>
>> $ ./pharo hp35.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/HP35/main ConfigurationOfHP35 --install=stable --group=Web-UI
>>
>> Both load Seaside and some other stuff, but only Reddit loads Glorp and PostgresV2.
>>
>> I did a condense changes on the HP35 image, the one on the Reddit image ran out of physical memory (granted is was a small machine) !
>>
>> Here are the resulting sizes:
>>
>> $ ls -lah
>> total 337M
>> drwxr-xr-x 4 root root 4.0K Oct 28 15:09 .
>> drwx------ 12 root root 4.0K Oct 28 12:28 ..
>> -rw-r--r-- 1 root root 5.8M Oct 28 13:17 hp35.changes
>> -rw-r--r-- 1 root root 5.6M Oct 28 14:38 hp35-condense-test.changes
>> -rw-r--r-- 1 root root 5.8M Oct 28 14:27 hp35-condense-test.changes.bak
>> -rw-r--r-- 1 root root 29M Oct 28 14:38 hp35-condense-test.image
>> -rw-r--r-- 1 root root 29M Oct 28 13:17 hp35.image
>> drwxr-xr-x 2 root root 4.0K Oct 28 13:12 package-cache
>> -rwxr-xr-x 1 root root 367 Oct 28 12:29 pharo
>> -rw-rw-r-- 1 root root 265K Oct 24 14:42 Pharo.changes
>> -rw-rw-r-- 1 root root 21M Oct 24 14:42 Pharo.image
>> -rwxr-xr-x 1 root root 354 Oct 28 12:29 pharo-ui
>> drwxr-xr-x 3 root root 4.0K Oct 28 12:29 pharo-vm
>> -rw-r--r-- 1 root root 136M Oct 28 12:48 reddit.changes
>> -rw-r--r-- 1 root root 105M Oct 28 12:48 reddit.image
>>
>> Note how the HP35 image and changes sizes are totally acceptable/normal, while the Reddit one explodes, with the difference being Glorp.
>>
>> But I have no idea what causes this.
>>
>> There is no way Glorp can generate 100 Mb changes, when Seaside is OK with 6 Mb.
>>
>>> I remember loading ~4000 classes in the order of 10^6 LOC in Dolphin, and
>>> changes never got half that size. Sizes were ~28/55MB image/changes.
>>> The same in VAST, any image beyond the 20MB was an alert of something
>>> being leaked.
>>>
>>>> Hmm, we need more tests and data points, I will try on Linux command line later on.
>>>
>>> I ran in it in Linux (Ubuntu) through the command line.
>>>
>>>
>>> Regards!
>>>
>>
>>
>

Sven Van Caekenberghe-2

Re: Changes file size

On 10 Feb 2015, at 09:13, Sven Van Caekenberghe <[hidden email]> wrote:

Esteban,

You might be on something important there.

I always found it strange what effect loading Glorp had on an image, blowing it up.

I can't say I understand though, best make it into an issue.

I did some further digging. Basically, the GlorpSession>>#dropTables:'s MCMethodDefinitions are totally broken. I loaded Glorp in a clean 4.0 image (and did a merge) and I have two such definition instances, each with a #timeStamp of MEGABYTES, one ~ 60 Mb, one ~ 130 Mb, the contents is 'am From VisualWorks' and then lots of garbage:

No wonder the image grows by 200 Mb !

I will try to do what you did and commit a version of that method with a normal time stamp.

Sven

Sven

On 10 Feb 2015, at 00:22, Esteban A. Maringolo <[hidden email]> wrote:

Just when I found a way to reduce the total size of the changes file, I
noticed the latest versions in the repository doesn't have this issue,
what a good way of wasting one's time learning about the guts of
Monticello :)

But just for the record:
The problem happened after installing into GlorpSession a
MCMethodDefinition (#dropTables:), I logged every single method
installation, and the point where things went bananas was after
installing that culprit method:

After a MCMethodDefinition(dropTables): 1201686 bytes (~1.2Mb)
After a MCMethodDefinition(dropTables:): 135419895 bytes (~135Mb)

I did a fileOut, deleted the method, and then file it in again, and
after versioning (and reloading) the changes increased ~1Mb after
installing Glorp.

I didn't find anything particularly suspicious about it, not in the
source code nor in the bytecodes.

Now after running PharoChangesCondenser condense the changes file
effectively shrinks.

Regards!

Esteban A. Maringolo

2014-10-28 12:31 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:

On 28 Oct 2014, at 14:58, Esteban A. Maringolo <[hidden email]> wrote:

2014-10-28 5:41 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
Esteban,

The Reddit example's CI Job (https://ci.inria.fr/pharo-contribution/job/Reddit/) also loads Seaside and Glorp (two big packages) in Pharo 3 and it too results in a 110 Mb image and 142 Mb changes file. After condensing, that goes to 276 Mb !

I would say something is wrong here ;-)

Either something is wrong, or we feature it and rename
PharoChangesCondenser>>#condense to PharoChangesExpander>>#expand :)

Note that condensing changes on newly loaded code should not make much difference (in essence, all multiple versions of methods are reduced to 1).

I think we should create some issues out of this.

This is true, however I find a changes of 140 megs to be MASSIVE.

I totally agree, this is unacceptable. However, I did some tests, and the problem is with Glorp (or any of its sub packages). I tried building both my Reddit and HP35 examples from scratch in Pharo 3 on Linux.

$ ./pharo reddit.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/Reddit/main ConfigurationOfReddit --install=stable

$ ./pharo hp35.image config http://smalltalkhub.com/mc/SvenVanCaekenberghe/HP35/main ConfigurationOfHP35 --install=stable --group=Web-UI

Both load Seaside and some other stuff, but only Reddit loads Glorp and PostgresV2.

I did a condense changes on the HP35 image, the one on the Reddit image ran out of physical memory (granted is was a small machine) !

Here are the resulting sizes:

$ ls -lah
total 337M
drwxr-xr-x 4 root root 4.0K Oct 28 15:09 .
drwx------ 12 root root 4.0K Oct 28 12:28 ..
-rw-r--r-- 1 root root 5.8M Oct 28 13:17 hp35.changes
-rw-r--r-- 1 root root 5.6M Oct 28 14:38 hp35-condense-test.changes
-rw-r--r-- 1 root root 5.8M Oct 28 14:27 hp35-condense-test.changes.bak
-rw-r--r-- 1 root root 29M Oct 28 14:38 hp35-condense-test.image
-rw-r--r-- 1 root root 29M Oct 28 13:17 hp35.image
drwxr-xr-x 2 root root 4.0K Oct 28 13:12 package-cache
-rwxr-xr-x 1 root root 367 Oct 28 12:29 pharo
-rw-rw-r-- 1 root root 265K Oct 24 14:42 Pharo.changes
-rw-rw-r-- 1 root root 21M Oct 24 14:42 Pharo.image
-rwxr-xr-x 1 root root 354 Oct 28 12:29 pharo-ui
drwxr-xr-x 3 root root 4.0K Oct 28 12:29 pharo-vm
-rw-r--r-- 1 root root 136M Oct 28 12:48 reddit.changes
-rw-r--r-- 1 root root 105M Oct 28 12:48 reddit.image

Note how the HP35 image and changes sizes are totally acceptable/normal, while the Reddit one explodes, with the difference being Glorp.

But I have no idea what causes this.

There is no way Glorp can generate 100 Mb changes, when Seaside is OK with 6 Mb.

I remember loading ~4000 classes in the order of 10^6 LOC in Dolphin, and
changes never got half that size. Sizes were ~28/55MB image/changes.
The same in VAST, any image beyond the 20MB was an alert of something
being leaked.

Hmm, we need more tests and data points, I will try on Linux command line later on.

I ran in it in Linux (Ubuntu) through the command line.

Regards!

Sven Van Caekenberghe-2

Re: Changes file size

> On 11 Feb 2015, at 09:35, Sven Van Caekenberghe <[hidden email]> wrote:
>
> I will try to do what you did and commit a version of that method with a normal time stamp.

I committed:

===
Name: Glorp-SvenVanCaekenberghe.96
Author: SvenVanCaekenberghe
Time: 11 February 2015, 9:59:31.16347 am
UUID: 1e1961df-8d15-44a7-bcef-9ec7dd17aab6
Ancestors: Glorp-SvenVanCaekenberghe.95

Try editing GlorpSession>>#dropTables: to get rid of the enormous timestamp
===

And then tried loading in a fresh 4.0 image:

ConfigurationOfGlorp project bleedingEdge load.

And image/changes sizes remain normal:

$ ls -lah current/
total 137336
drwxr-xr-x 8 sven staff 272B Feb 11 10:27 .
drwxr-xr-x 11 sven staff 374B Feb 11 10:23 ..
-rw-rw-r-- 1 sven staff 27M Feb 11 10:30 Pharo.changes
-rw-rw-r--@ 1 sven staff 40M Feb 11 10:30 Pharo.image
-rw-r--r--@ 1 sven staff 12K Feb 11 10:27 PharoDebug.log
drwxr-xr-x 6 sven staff 204B Feb 11 10:25 package-cache
drwxr-xr-x 4 sven staff 136B Feb 11 10:26 play-cache
-rw-r--r-- 1 sven staff 29B Feb 11 10:23 updates40.staged

$ ls -lah current/package-cache/
total 1640
drwxr-xr-x 6 sven staff 204B Feb 11 10:25 .
drwxr-xr-x 8 sven staff 272B Feb 11 10:27 ..
-rw-r--r--@ 1 sven staff 7.5K Feb 11 10:24 ConfigurationOfGlorp-TorstenBergmann.42.mcz
-rw-r--r--@ 1 sven staff 541K Feb 11 10:25 Glorp-SvenVanCaekenberghe.96.mcz
-rw-r--r--@ 1 sven staff 2.8K Feb 11 10:25 GlorpPharoCompatibility-GuillermoPolito.6.mcz
-rw-r--r--@ 1 sven staff 261K Feb 11 10:25 GlorpTests-FranckWarlouzet.41.mcz

Let's hope it now stays like that ;-)

Sven

Esteban A. Maringolo

Re: Changes file size

In reply to this post by Sven Van Caekenberghe-2

2015-02-11 5:35 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:

>> On 10 Feb 2015, at 09:13, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> Esteban,
>>
>> You might be on something important there.
>> I always found it strange what effect loading Glorp had on an image, blowing it up.
>> I can't say I understand though, best make it into an issue.
>
> I did some further digging. Basically, the GlorpSession>>#dropTables:'s MCMethodDefinitions are totally broken. I loaded Glorp in a clean 4.0 image (and did a merge) and I have two such definition instances, each with a #timeStamp of MEGABYTES, one ~ 60 Mb, one ~ 130 Mb, the contents is 'am From VisualWorks' and then lots of garbage:
>
> No wonder the image grows by 200 Mb !
>
> I will try to do what you did and commit a version of that method with a normal time stamp.
> Sven

Wow, I missed looking into the MCMethodDefinition itself. I don't
understand how such a malformed timestamp can be there. But even 130MB
is several times the image size of VW.

I'll update my configurations.

Thank you Sven!

ps: As said in another thread, Torsten's version (descendant of mine)
didn't seem to have this issue.

Sven Van Caekenberghe-2

Re: Changes file size

> On 11 Feb 2015, at 13:54, Esteban A. Maringolo <[hidden email]> wrote:
>
> 2015-02-11 5:35 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>>> On 10 Feb 2015, at 09:13, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Esteban,
>>>
>>> You might be on something important there.
>>> I always found it strange what effect loading Glorp had on an image, blowing it up.
>>> I can't say I understand though, best make it into an issue.
>>
>> I did some further digging. Basically, the GlorpSession>>#dropTables:'s MCMethodDefinitions are totally broken. I loaded Glorp in a clean 4.0 image (and did a merge) and I have two such definition instances, each with a #timeStamp of MEGABYTES, one ~ 60 Mb, one ~ 130 Mb, the contents is 'am From VisualWorks' and then lots of garbage:
>>
>> No wonder the image grows by 200 Mb !
>>
>> I will try to do what you did and commit a version of that method with a normal time stamp.
>> Sven
>
> Wow, I missed looking into the MCMethodDefinition itself. I don't
> understand how such a malformed timestamp can be there. But even 130MB
> is several times the image size of VW.
>
>
> I'll update my configurations.
>
> Thank you Sven!
>
> ps: As said in another thread, Torsten's version (descendant of mine)
> didn't seem to have this issue.

Yes, we are making (some) progress.
Maybe we should promote the latest version to #stable ?

Esteban A. Maringolo

Re: Changes file size

2015-02-11 10:02 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>> ps: As said in another thread, Torsten's version (descendant of mine)
>> didn't seem to have this issue.

> Yes, we are making (some) progress.
> Maybe we should promote the latest version to #stable ?

I think we should, but there is something with that too. There is only
one version in the MetaRepo whilst the rest of Metacello configs are
in the DBXTalk/Configurations repo.

I don't know if that is intentional or accidental.

Regards.

Esteban A. Maringolo

Sven Van Caekenberghe-2

Re: Changes file size

> On 11 Feb 2015, at 16:07, Esteban A. Maringolo <[hidden email]> wrote:
>
> 2015-02-11 10:02 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>>> ps: As said in another thread, Torsten's version (descendant of mine)
>>> didn't seem to have this issue.
>
>> Yes, we are making (some) progress.
>> Maybe we should promote the latest version to #stable ?
>
> I think we should, but there is something with that too. There is only
> one version in the MetaRepo whilst the rest of Metacello configs are
> in the DBXTalk/Configurations repo.

Torsten, should I try to fix this, or do you have other plans ?

> I don't know if that is intentional or accidental.
>
> Regards.
>
> Esteban A. Maringolo

Guillermo Polito

Re: Changes file size

In reply to this post by Esteban A. Maringolo

El Wed Feb 11 2015 at 4:08:36 PM, Esteban A. Maringolo <[hidden email]> escribió:

2015-02-11 10:02 GMT-03:00 Sven Van Caekenberghe <[hidden email]>:
>> ps: As said in another thread, Torsten's version (descendant of mine)
>> didn't seem to have this issue.

> Yes, we are making (some) progress.
> Maybe we should promote the latest version to #stable ?

I think we should, but there is something with that too. There is only
one version in the MetaRepo whilst the rest of Metacello configs are
in the DBXTalk/Configurations repo.

Well, the configurations should inhabit close to their projects. MetaRepo is only to publish the configurations. So the usual way to release should be:

- commit the configuration to the repo of its project

- then copy this configuration to metarepo

Guille