[Glass] GLASS performance & cleanup scripts

Hi all!
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

James,

I'm back :)

I have made some test based on your post...

>Welcome to GemStone.
Thnks! I'm happy I can finally try it.

>First, as to configuring the Shared Page Cache (SCP), the rule is simple:
>get the most that you can afford (up to slightly more than the repository
>size). With the free license the maximum size is 2 GB, but that should be
>pretty good for your database.
OK, so I change (in testing) the /etc/sysctl.conf file like this according to 2GB free license limit
kernel.shmmax = 1610612736
kernel.shmall = 393216

I assume I must not change SHR_PAGE_CACHE_SIZE_KB parameter because is the extent page chache size, actually is 128000

>The free license does not have a limit on the database size, so you can
>keep growing.
Perfect.

>The #'fullBackupCompressedTo' method does an object-by-object backup that
>is not identical to the extent0.dbf file in layout (it has all the objects,
>but is typically more compact since the extents can have free space). The
>alternative is to suspend checkpoints, copy the extent(s), and resume
>checkpoints. That will give you an extent backup that can be used directly
>as a repository. Unless you have reason to prefer the extent-file-copy
>approach, I suggest you stick with the existing backup (and practice a
>restore occasionally).
Ok, I prefer the object-by-object backup approach.
One question: When I make a #fullBackupCompressedTo, I can restore with the Gemstools "restore w/o tranlog" option? Or how can I make a restore from a compress database (I prefer the command line, but any help is ok)

>Often one will choose to start a new transaction log just before taking a
>backup. Then you don't have to wonder which transaction logs are needed to
>go with a particular backup. I have seen scripts that delete transaction
>logs that are older than the oldest backup that you would use for disaster
>recovery (and you typically keep at least two backups).
Ok, thats great. If I make a good backup I have no worry about tranlogs.

>Strategies for making reports faster typically involve indexes and/or
>clustering. You can read about them in the programming guide.
Ok, I began reading about it. So next I'm going to check is this.

>Keep asking questions and let us help you get the system running smoothly.
>Of course, at some point I might suggest you pay for consulting and/or a
>license to have a larger Shared Page Cache. ;-)
Yes of course! I assume that's the reason I'm not gain more performance with more RAM (maybe the only thing is with a SSD disk).
I think is more convenient to pay for consulting here, because is more bourocratic to get money for licenses... I'm going to send you a private email for this.

Thnks!
Leo




On Fri, Nov 29, 2013 at 4:12 PM, <[hidden email]> wrote:

> Hi all!
>
> Recently I have began to work with GLASS, I have experience working with
> smalltalk but no experience working with Gemstone & Linux, so be patient
> with me :)
>
> My client already have a GLASS system working on production enviroment,
> but with some performance problems when they perform large reports, so I
> plan a strategy of 3 stages:
> 1) Review the hardware
> 2) Review the Server & Gemstone configuration
> 3) Review the code
>
> The first issue, the hardware, already check and I think its ok for a 15GB
> repository and only 10 concurrent users. Is composed of a
> -#HP ProLiant ML350 G5
> -#8 CPU Cores (Intel Xeon E5405 @ 2Ghz)
> -#8 GB RAM
> -#3 146GB disks in RAID 5 (for storage) (bay 1-2-3)
>
> The only thing here that I read in some post is changing the disk drives
> to SSD but is not a quick decision here where I am :)
>
> The second one, is where actually I am. I change the host SO, I mounted a
> VMWare ESXi and then mounted a GemStone/S 64 Bit in a VMware Appliance to
> it. Then I added a 2 more spindles for the repository and the tranlogs and
> also increase the swap partition (see image attached).
>
> Questions:
> -My repository size & concurrent users dont fit to pre-Selected
> configuration (repository size is for a Medium conf and users fit with
> Small one), so what is a recommended main variables for the config file
> according to my repository values?
>
> -*Repository size*: My repository is 15 GB, I read somewhere that 16 GB
> is a limit. Why is that? Is recommended in these case to split in 2
> repositories?
>
> -*Estimating Shared Page Cache issue*: I read here:
> http://programminggems.wordpress.com/2012/04/06/configuring-shared-memory/ and
> then read also the Gemstone installation guide documentation (
> http://community.gemstone.com/download/attachments/6816862/GS64-InstallGuide-Linux-3.1.pdf),
> but the calculation are different, so Im confused about it. What are the
> things I have to consider to properly configure shred parge cache?
>
> -*Cleanup scripts*: I read here to automate a cleanup scrip, but also I
> find differents approach with others scripts:
> a) The backup make with #fullBackupCompressedTo: make a gzip file that
> when decompress have no extension (is a extent.dbf copy no?). The other
> approach tell my to make a direct copy of the extent (cp command).
> b) The tranlog clean up script is cheking the files datetime but in other
> post (http://forum.world.st/Translog-space-problem-td2403379.html) I read
> about the #oldestLogFileIdForRecovery that tell me the oldest tranlog
> needed for a recovery, is more convenient to make a script with that info?
>
> Thnks in advance!
> Leo
>
>
>
>
> _______________________________________________
> Glass mailing list
> [hidden email]
> http://lists.gemtalksystems.com/mailman/listinfo/glass
>
>

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

On Dec 4, 2013, at 5:37 PM, leodm <[hidden email]> wrote:

> James,
>
> I'm back :)

Good!

These appear to total ~1.5GB, so you could push it a bit more if you wanted.

> I assume I must not change SHR_PAGE_CACHE_SIZE_KB parameter because is the
> extent page chache size, actually is 128000

That is 128MB, far short of the maximum. You could easily add a zero to the end and have 1.28 GB. (The SPC needs to be slightly smaller than the allowed shared memory since the system allocates other shared space besides the page cache.)

I think so, but I don’t use GemTools much. Are you using Linux exclusively, or Linux in a VM on Windows, or Mac, or …?

> Or how can I make a restore from a
> compress database (I prefer the command line, but any help is ok)

To use the command line you would use Topaz. There is an entire chapter in the System Administration Guide on backup/restore. I haven’t tested the following code, but from memory it would be something like the following (in Topaz comments begin with the ‘!’ character):

$ topaz -l << EOF
set user DataCurator pass swordfish gemstone gs64stone
login
run
SystemRepository restoreFromBackup: ‘/path/to/my/backup.bak.gz’
%
! this should give an “error” (logging you out) reporting the successful restore
login
send SystemRepository restoreStatus
! this should report that it is waiting for transaction logs
! if you don’t want to restore any logs, then commits as follows
send SystemRepository commitRestore
! this should give an “error” (logging you out) and report success
logout
exit

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Hi! me again :)

>> First, as to configuring the Shared Page Cache (SCP), the rule is simple:
>> get the most that you can afford (up to slightly more than the repository
>> size). With the free license the maximum size is 2 GB, but that should be
>> pretty good for your database.
> OK, so I change (in testing) the /etc/sysctl.conf file like this according
> to 2GB free license limit
> kernel.shmmax = 1610612736
> kernel.shmall = 393216

>These appear to total ~1.5GB, so you could push it a bit more if you wanted.

Yes, I finally changed to
kernel.shmmax=2147483648
kernel.shmall=524288

and SHR_PAGE_CACHE_SIZE_KB to 1280000

and reboot ok, but I get this error when I run the garbage collection manually. My gc script is like this:


ObjectLogEntry emptyLog.
 MCRepositoryGroup default repositoriesDo: [:rep | rep flushCache ].
 MCDefinition clearInstances.
 MCMethodDefinition cachedDefinitions  removeKeys:
    (MCMethodDefinition cachedDefinitions keys).
 MCMethodDefinition shutDown.
 MethodVersionHistory uniqueInstance cleanUp.
%
commit
begin
printit
SystemRepository markForCollection
%

and I get this error:


topaz 1> topaz 1> start MFC mark/sweep phase: 12/10/2013 02:28:32 PM ART
WARNING, unable to resolve either garMarkSweepPageBufSize or mfcGcPageBufSize, using default 320
MarkSweepArgsSType::(): buffer size= 320 pages

I think that my recently changes affect this process, for the error message I asume I have to change other gemstone config variables that are related to the changes I made, its ok?

>> The #'fullBackupCompressedTo' method does an object-by-object backup that
>> is not identical to the extent0.dbf file in layout (it has all the objects,
>> but is typically more compact since the extents can have free space). The
>> alternative is to suspend checkpoints, copy the extent(s), and resume
>> checkpoints. That will give you an extent backup that can be used directly
>> as a repository. Unless you have reason to prefer the extent-file-copy
>> approach, I suggest you stick with the existing backup (and practice a
>> restore occasionally).
> Ok, I prefer the object-by-object backup approach.
> One question: When I make a #fullBackupCompressedTo, I can restore with the
> Gemstools "restore w/o tranlog" option?

>I think so, but I don’t use GemTools much. Are you using Linux exclusively, or Linux in a VM on Windows, or >Mac, or …?

Im using a Linux VM (is a classic GLASS Apliance VM with some changes) running on a VMware ESXi

> Or how can I make a restore from a
> compress database (I prefer the command line, but any help is ok)

>To use the command line you would use Topaz. There is an entire chapter in the System Administration >Guide on backup/restore. I haven’t tested the following code, but from memory it would be something like >the following (in Topaz comments begin with the ‘!’ character):

>$ topaz -l << EOF
>set user DataCurator pass swordfish gemstone gs64stone
>login
>run
>SystemRepository restoreFromBackup: ‘/path/to/my/backup.bak.gz’
>%
<! this should give an “error” (logging you out) reporting the successful restore
>login
>send SystemRepository restoreStatus
>! this should report that it is waiting for transaction logs
>! if you don’t want to restore any logs, then commits as follows
>send SystemRepository commitRestore
>! this should give an “error” (logging you out) and report success
>logout
>exit

I try this out and I read all the chepter 9 of the Admin Guide, but I cant restore any backup yet. Always reports me probems with other sessions. The only way is to delete the repository, replace with the original one and then make the restore? is the only way? I cant just restore a backup on top of the current repository?

Thnks!
Leo

See below...

On Dec 10, 2013, at 9:53 AM, leodm <[hidden email]> wrote:

This is not an Error, but a Warning and may be safely ignored. If you want to avoid the warning you can set a value in UserGlobals for GcUser.


You can’t restore a backup when other users are logged in. Try adding ‘send System stopUserSessions’ to your Topaz scripts.

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass