Smalltalk › Gemtalk › GLASS

image grows unexpected

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

9 messages Options

NorbertHartl

image grows unexpected

I have an installation running for some time. For a long time the size of the repository was of the same size. In the last two weeks it doubled size and I don't have a glue what might have caused this. I didn't do anything special as far as i can remember.

What are the best analyzing tricks to get an idea where the space has gone?

thanks,

Norbert

Joseph Bacanskas-4

Re: image grows unexpected

HI Norbert:

Are you running Epoch GC? Did you make any changes to your GemStone model or code?

You can restore a current backup of your repository to another machine and run an FDC. That will tell you the live vs. dead objects. If your live objects are the overwhelming majority, you are persisting objects that you don't expect to be persistent. If you have a huge dead object count, you have several possible issues. This is solved via iterative analysis. Please post your findings.

BTW - If you decide to do an FDC, follow the recommendations for cache settings in the manual. The ideal FDC has, after a short initial period, the FDC gem using ~85-90% of a CPU with there being almost zero "cache misses". VSD is your friend. :-)

On Nov 13, 2010, at 5:41 PM, Norbert Hartl wrote:

> I have an installation running for some time. For a long time the size of the repository was of the same size. In the last two weeks it doubled size and I don't have a glue what might have caused this. I didn't do anything special as far as i can remember.
>
> What are the best analyzing tricks to get an idea where the space has gone?
>
> thanks,
>
> Norbert

Thanks!!
Joseph Bacanskas [|]
--- I use Smalltalk. My amp goes to eleven.

NorbertHartl

Re: image grows unexpected

Hi,

On 14.11.2010, at 08:20, Joseph Bacanskas wrote:

> HI Norbert:
>
> Are you running Epoch GC? Did you make any changes to your GemStone model or code?
>
I don't do anything special so I think I don't do epoch GC :) I did change some of my code the last two weeks.

> You can restore a current backup of your repository to another machine and run an FDC. That will tell you the live vs. dead objects. If your live objects are the overwhelming majority, you are persisting objects that you don't expect to be persistent. If you have a huge dead object count, you have several possible issues. This is solved via iterative analysis. Please post your findings.
>
> BTW - If you decide to do an FDC, follow the recommendations for cache settings in the manual. The ideal FDC has, after a short initial period, the FDC gem using ~85-90% of a CPU with there being almost zero "cache misses". VSD is your friend. :-)
>
After doing an MFC the live object count is what I described. I have a multi user setup. There are 5 users in my image. For every user I run the maintenance script once an hour and after all users have been clean I run a MFC. Yesterday evening I took the time to go through all users manually flushed some caches, expired handlers, did some cleanups and invoked MFC consequently. The live objects didn't decrease noticably. I did also do an allInstances scan on the most prominent class that might cause this. But the counts were low. So I need a rough overview where most of the objects live. It would even help if I would know which user is causing this. A classes amount of objects in image would be great.

Norbert

> On Nov 13, 2010, at 5:41 PM, Norbert Hartl wrote:
>
>> I have an installation running for some time. For a long time the size of the repository was of the same size. In the last two weeks it doubled size and I don't have a glue what might have caused this. I didn't do anything special as far as i can remember.
>>
>> What are the best analyzing tricks to get an idea where the space has gone?
>>
>> thanks,
>>
>> Norbert
>
> Thanks!!
> Joseph Bacanskas [|]
> --- I use Smalltalk. My amp goes to eleven.
>

James Foster-8

Re: image grows unexpected

Norbert,

Dale recently discovered that the maintenance gem (responsible for expiring old sessions) was keeping many objects from being included in the MFC. I don't remember the details, but as I recall stopping the maintenance gem and restarting it will allow the objects to be reclaimed.

If you want to investigate what objects are in your database, you might be interested in the ScanBackup tool described at http://programminggems.wordpress.com/2009/12/15/scanbackup-2/.

James

On Nov 14, 2010, at 8:25 AM, Norbert Hartl wrote:

> Hi,
>
> On 14.11.2010, at 08:20, Joseph Bacanskas wrote:
>
>> HI Norbert:
>>
>> Are you running Epoch GC? Did you make any changes to your GemStone model or code?
>>
> I don't do anything special so I think I don't do epoch GC :) I did change some of my code the last two weeks.
>
>> You can restore a current backup of your repository to another machine and run an FDC. That will tell you the live vs. dead objects. If your live objects are the overwhelming majority, you are persisting objects that you don't expect to be persistent. If you have a huge dead object count, you have several possible issues. This is solved via iterative analysis. Please post your findings.
>>
>> BTW - If you decide to do an FDC, follow the recommendations for cache settings in the manual. The ideal FDC has, after a short initial period, the FDC gem using ~85-90% of a CPU with there being almost zero "cache misses". VSD is your friend. :-)
>>
> After doing an MFC the live object count is what I described. I have a multi user setup. There are 5 users in my image. For every user I run the maintenance script once an hour and after all users have been clean I run a MFC. Yesterday evening I took the time to go through all users manually flushed some caches, expired handlers, did some cleanups and invoked MFC consequently. The live objects didn't decrease noticably. I did also do an allInstances scan on the most prominent class that might cause this. But the counts were low. So I need a rough overview where most of the objects live. It would even help if I would know which user is causing this. A classes amount of objects in image would be great.
>
> Norbert
>
>> On Nov 13, 2010, at 5:41 PM, Norbert Hartl wrote:
>>
>>> I have an installation running for some time. For a long time the size of the repository was of the same size. In the last two weeks it doubled size and I don't have a glue what might have caused this. I didn't do anything special as far as i can remember.
>>>
>>> What are the best analyzing tricks to get an idea where the space has gone?
>>>
>>> thanks,
>>>
>>> Norbert
>>
>> Thanks!!
>> Joseph Bacanskas [|]
>> --- I use Smalltalk. My amp goes to eleven.
>>
>

NorbertHartl

Re: image grows unexpected

James,

On 14.11.2010, at 12:57, James Foster wrote:

> Norbert,
>
> Dale recently discovered that the maintenance gem (responsible for expiring old sessions) was keeping many objects from being included in the MFC. I don't remember the details, but as I recall stopping the maintenance gem and restarting it will allow the objects to be reclaimed.
>
that can't happen in my scenario. I have 5 gemstone users. I've split the maintenance gem script into the cleaning part and the MFC part. Furthermore I removed the endless loop and let the gem exit. I start the gems by a cronjoub every hour. So I have 6 invocations an hour (5 x cleaning, 1 x MFC). After a few seconds the mainenance gems are down again. The only ones that keep running are the web server gems. I have 8 of these runnig (1 user has 3 gems, 1 has 2 and the others 1).

> If you want to investigate what objects are in your database, you might be interested in the ScanBackup tool described at http://programminggems.wordpress.com/2009/12/15/scanbackup-2/.
>
I looked at yesterday but it is a 32 bit version and I'm a GLASS user. I don't think it gives much information if you can't read the backup file, right?

Norbert

> James
>
> On Nov 14, 2010, at 8:25 AM, Norbert Hartl wrote:
>
>> Hi,
>>
>> On 14.11.2010, at 08:20, Joseph Bacanskas wrote:
>>
>>> HI Norbert:
>>>
>>> Are you running Epoch GC? Did you make any changes to your GemStone model or code?
>>>
>> I don't do anything special so I think I don't do epoch GC :) I did change some of my code the last two weeks.
>>
>>> You can restore a current backup of your repository to another machine and run an FDC. That will tell you the live vs. dead objects. If your live objects are the overwhelming majority, you are persisting objects that you don't expect to be persistent. If you have a huge dead object count, you have several possible issues. This is solved via iterative analysis. Please post your findings.
>>>
>>> BTW - If you decide to do an FDC, follow the recommendations for cache settings in the manual. The ideal FDC has, after a short initial period, the FDC gem using ~85-90% of a CPU with there being almost zero "cache misses". VSD is your friend. :-)
>>>
>> After doing an MFC the live object count is what I described. I have a multi user setup. There are 5 users in my image. For every user I run the maintenance script once an hour and after all users have been clean I run a MFC. Yesterday evening I took the time to go through all users manually flushed some caches, expired handlers, did some cleanups and invoked MFC consequently. The live objects didn't decrease noticably. I did also do an allInstances scan on the most prominent class that might cause this. But the counts were low. So I need a rough overview where most of the objects live. It would even help if I would know which user is causing this. A classes amount of objects in image would be great.
>>
>> Norbert
>>
>>> On Nov 13, 2010, at 5:41 PM, Norbert Hartl wrote:
>>>
>>>> I have an installation running for some time. For a long time the size of the repository was of the same size. In the last two weeks it doubled size and I don't have a glue what might have caused this. I didn't do anything special as far as i can remember.
>>>>
>>>> What are the best analyzing tricks to get an idea where the space has gone?
>>>>
>>>> thanks,
>>>>
>>>> Norbert
>>>
>>> Thanks!!
>>> Joseph Bacanskas [|]
>>> --- I use Smalltalk. My amp goes to eleven.
>>>
>>
>

James Foster-8

Re: image grows unexpected

On Nov 14, 2010, at 9:40 AM, Norbert Hartl wrote:

>> If you want to investigate what objects are in your database, you might be interested in the ScanBackup tool described at http://programminggems.wordpress.com/2009/12/15/scanbackup-2/.
>>
> I looked at yesterday but it is a 32 bit version and I'm a GLASS user. I don't think it gives much information if you can't read the backup file, right?

"I’ve updated the ScanBackup tool ... so that it supports ... 64-bit 2.3.x and 2.4.x and compressed as well as uncompressed backups." -- http://programminggems.wordpress.com/2009/12/15/scanbackup-2/

James

NorbertHartl

Re: image grows unexpected

On 14.11.2010, at 13:50, James Foster wrote:

> On Nov 14, 2010, at 9:40 AM, Norbert Hartl wrote:
>
>>> If you want to investigate what objects are in your database, you might be interested in the ScanBackup tool described at http://programminggems.wordpress.com/2009/12/15/scanbackup-2/.
>>>
>> I looked at yesterday but it is a 32 bit version and I'm a GLASS user. I don't think it gives much information if you can't read the backup file, right?
>
> "I’ve updated the ScanBackup tool ... so that it supports ... 64-bit 2.3.x and 2.4.x and compressed as well as uncompressed backups." -- http://programminggems.wordpress.com/2009/12/15/scanbackup-2/
>
Oh thanks. I was looking yesterday for the tool because I remembered somebody mentioning it. I found it over Dales blog but that seems to be the old one. However it doesn't work in my case. In ScanTool>>readFile there is a call to

self determineFileTypeAndVersion

which seems only be present in ScanLog64. As the ScanBackup64 does not seem to use the initialized instance variables from determineFileTypeAndVersion I commented out the statement. But then I get an error in LogRootRecord "Unrecognized swizzle transformer'". So I'm asking myself if the button in gemtools Admin->Repository->Backup produces the right backup format. I guess it does a fullBackupTo: but I'm not sure.

I'm also not sure about how to proceed in my case. I think I need to start to tool as DataCurator or even as System? The code base of user DataCurator is somewhat older because every active code is handled by another user. What is the best setup for this? Can I see still the class names of the findings in my scenario?

Thanks,

Norbert

NorbertHartl

Re: image grows unexpected [update]

In reply to this post by NorbertHartl

On 14.11.2010, at 02:41, Norbert Hartl wrote:

I have an installation running for some time. For a long time the size of the repository was of the same size. In the last two weeks it doubled size and I don't have a glue what might have caused this. I didn't do anything special as far as i can remember.

What are the best analyzing tricks to get an idea where the space has gone?

A short update. It turned out that I myself was the cause of the problem. With my multi user setup I needed to separate the maintenance script into the seaside session cleaning part and the MFC part. In the seaside script I changed to invocation of the new cleanup task to

"run maintenance tasks"

(WAGemStoneMaintenanceTask taskNamed: 'Seaside Session Expiration' ) performTask: count.

While doing that first I noticed that WAGemStoneMaintenanceTask>>taskNamed: is lacking the ^ to return the instance of the task. I thought I fixed it for every user but I forgot to fix it for my main site. So the cron triggered maintenance script just exits without doing anything. This way I collected a lot of old seaside sessions. What is still strage is that I'm quite sure I did a manual "WARegistry clearAllHandlers" yesterday evening without having any effect.

I had no glue where the space is lacking. But looking at the cherokee status page I could at least see that in the last two weeks my main site had the most requests of all. Furthermore I investigated the apache access logs. This way I could see that there is a coincidence in incoming requests to that particular site and the growth in gemstone. I confirmed it by issuing 1000 requests to the site and saw that the repository keeps growing a lot. Btw. with all my tests I made the repository grow 10 times bigger. At the boundary of 4GB I was really glad that there is the new key available. Thanks again for the extension of the license. The repository grew up to 8GB until I found the cause.

It takes some time until the effect is visible in repository size. At that point I was thankful for having vsd running. The graph made me half believe that it worked

And there is even a solution to confirm it finally that it worked. I made a backup and saw that file size dropped to 1/10 of its size before.

At least I used statmonitor for the first time and got a bit comfortable in using vsd. You can make the whole thing a permanent monitoring solution even over the network. On the remote machine I ran

statmonitor seaside -f /tmp/stat.mon -i 60

And on my local machine I did

while true; do date; scp biggy:/var/lib/vz/private/34/tmp/stat.* .; sleep 60 ; done

Opening vsd on my local machine, opening the stat.mon file and choosing in the vsd menu File->Auto update made it a live stat monitoring over the network. I like it.

Norbert

Joseph Bacanskas-4

Re: image grows unexpected [update]

Hi Norbert:

You might want to monitor PagesNeedingReclaimSize as well. VSD is a spectacular tool. It gets even more powerful as you start looking at groups of stats over an extended period of time. You'll find as you monitor your system live in VSD over a few months that you'll know instantly by looking at the charts that something is amiss.

BTW, if you are not running Epoch GC, possibleDead will only have non-zero values during the middle of the MFC.

Cheers!!

On Nov 14, 2010, at 8:23 AM, Norbert Hartl wrote:

>
> On 14.11.2010, at 02:41, Norbert Hartl wrote:
>
>> I have an installation running for some time. For a long time the size of the repository was of the same size. In the last two weeks it doubled size and I don't have a glue what might have caused this. I didn't do anything special as far as i can remember.
>>
>> What are the best analyzing tricks to get an idea where the space has gone?
>>
>>
> A short update. It turned out that I myself was the cause of the problem. With my multi user setup I needed to separate the maintenance script into the seaside session cleaning part and the MFC part. In the seaside script I changed to invocation of the new cleanup task to
>
> "run maintenance tasks"
> (WAGemStoneMaintenanceTask taskNamed: 'Seaside Session Expiration' ) performTask: count.
>
> While doing that first I noticed that WAGemStoneMaintenanceTask>>taskNamed: is lacking the ^ to return the instance of the task. I thought I fixed it for every user but I forgot to fix it for my main site. So the cron triggered maintenance script just exits without doing anything. This way I collected a lot of old seaside sessions. What is still strage is that I'm quite sure I did a manual "WARegistry clearAllHandlers" yesterday evening without having any effect.
>
> I had no glue where the space is lacking. But looking at the cherokee status page I could at least see that in the last two weeks my main site had the most requests of all. Furthermore I investigated the apache access logs. This way I could see that there is a coincidence in incoming requests to that particular site and the growth in gemstone. I confirmed it by issuing 1000 requests to the site and saw that the repository keeps growing a lot. Btw. with all my tests I made the repository grow 10 times bigger. At the boundary of 4GB I was really glad that there is the new key available. Thanks again for the extension of the license. The repository grew up to 8GB until I found the cause.
>
> It takes some time until the effect is visible in repository size. At that point I was thankful for having vsd running. The graph made me half believe that it worked
>
>
>
>
> And there is even a solution to confirm it finally that it worked. I made a backup and saw that file size dropped to 1/10 of its size before.
>
> At least I used statmonitor for the first time and got a bit comfortable in using vsd. You can make the whole thing a permanent monitoring solution even over the network. On the remote machine I ran
>
> statmonitor seaside -f /tmp/stat.mon -i 60
>
> And on my local machine I did
>
> while true; do date; scp biggy:/var/lib/vz/private/34/tmp/stat.* .; sleep 60 ; done
>
> Opening vsd on my local machine, opening the stat.mon file and choosing in the vsd menu File->Auto update made it a live stat monitoring over the network. I like it.
>
> Norbert

Thanks!!
Joseph Bacanskas [|]
--- I use Smalltalk. My amp goes to eleven.