Hello, Facing a fast growing extent, I am trying to understand the disk space taken by a tree of objects. I am guessing that you use physicalSize recursively through the object instance variables to get to total size, correct? Or is there a better way to do that?
Thanks, Thierry
|
James has a post on scanbackup[1] with code here[2] which one can effectively use to analyze the makeup of your entire data base and you might be able to extract the information you want from the scan backup report rather than try to analyze the size of a particular tree of objects ...
If you are only interested in the size of a particular object graph, then I think that your technique sounds about right... Dale [1] http://programminggems.wordpress.com/2009/05/14/scanbackup/ [2] http://seaside.gemstone.com/ss/ScanBackup.html ----- Original Message ----- | From: "Thierry Thelliez" <[hidden email]> | To: "GemStone Seaside beta discussion" <[hidden email]> | Sent: Thursday, February 9, 2012 9:42:00 AM | Subject: [GS/SS Beta] How to calculate the size of an object tree? | | | Hello, | | | | Facing a fast growing extent, I am trying to understand the disk | space taken by a tree of objects. I am guessing that you use | physicalSize recursively through the object instance variables to | get to total size, correct? Or is there a better way to do that? | | | | | Thanks, | Thierry |
Dale, Yes I know this tool, thanks. Actually I ran it last night and found a rather large number of Strings while garbage collection is reporting 0 dead objects. So my next step was trying to see if I have a bug somewhere and if I am holding to some un-needed objects.
BTW, the scanbackup tool does not seem to work with the backups generated from GemTools. It fails at LogRecordHdr > isBackupDataRecord
"LOG_BACKUP_DATA_RECORD defined in logdisk.ht"
(CompatibilityLevel = 830 and: [self recordKind = 17]) ifTrue: [^true].
^false. the recordKind is was not 17 (sorry I forgot to write the number down, I think it was 19). It worked if I execute the fullBackup from topaz. Thierry |
Thierry,
Thanks for the report on the ScanBackup tool. I'll work with Dale to try to figure out what is different when the backup is made from GemTools. -James On Feb 9, 2012, at 11:08 AM, Thierry Thelliez wrote:
|
In reply to this post by Thelliez
Thierry,
Here's the GemTools backup expression: System abortTransaction. result := SystemRepository fullBackupTo: backupPath. result == true ifFalse: [^ self error: 'Could not complete the backup: ' , result]. Not likely to be problem specific to GemTools backup ... Sorry for the dumb question, but are you sure that GemTools and topaz were connected to the same stone? A different data record type than expected sounds like a different version of GemStone is involved ... On another tack, are you able to backup and restore from GemTools? I would expect a similar error from restore, if indeed incorrect log records were being introduced into the backup file ... Dale ----- Original Message ----- | From: "Thierry Thelliez" <[hidden email]> | To: "GemStone Seaside beta discussion" <[hidden email]> | Sent: Thursday, February 9, 2012 11:08:37 AM | Subject: Re: [GS/SS Beta] How to calculate the size of an object tree? | | | Dale, | | | | Yes I know this tool, thanks. Actually I ran it last night and found | a rather large number of Strings while garbage collection is | reporting 0 dead objects. So my next step was trying to see if I | have a bug somewhere and if I am holding to some un-needed objects. | | | | | BTW, the scanbackup tool does not seem to work with the backups | generated from GemTools. It fails at LogRecordHdr > | isBackupDataRecord | "LOG_BACKUP_DATA_RECORD defined in logdisk.ht " (CompatibilityLevel = | 830 and: [self recordKind = 17]) ifTrue: [^true]. ^false. | | | the recordKind is was not 17 (sorry I forgot to write the number | down, I think it was 19). | | | It worked if I execute the fullBackup from topaz. | | | | | | | Thierry | | | | | | |
Thierry,
It is also possible (well, almost certain!) that backups taken at different times have different content. Of course, most of the content would be the same (data pages), but depending on the state of the system (particularly with respect to garbage collection), some more exotic things might be included. For example, if there was a list of shadow pages to be reclaimed then this might cause something else to be written to the backup so that the restore could finish the GC activity properly. The backup tool works fine for the backups that I've tested, but I didn't work from an exhaustive list of all possible record types. If you still have the backup that is a problem, we could investigate this further (either with a more detailed description of the error or with you sending me the backup). -James On Feb 9, 2012, at 12:35 PM, Dale Henrichs wrote: > Thierry, > > Here's the GemTools backup expression: > > System abortTransaction. > result := SystemRepository fullBackupTo: backupPath. > result == true > ifFalse: [^ self error: 'Could not complete the backup: ' , result]. > > Not likely to be problem specific to GemTools backup ... > > Sorry for the dumb question, but are you sure that GemTools and topaz were connected to the same stone? A different data record type than expected sounds like a different version of GemStone is involved ... > > On another tack, are you able to backup and restore from GemTools? I would expect a similar error from restore, if indeed incorrect log records were being introduced into the backup file ... > > Dale > > ----- Original Message ----- > | From: "Thierry Thelliez" <[hidden email]> > | To: "GemStone Seaside beta discussion" <[hidden email]> > | Sent: Thursday, February 9, 2012 11:08:37 AM > | Subject: Re: [GS/SS Beta] How to calculate the size of an object tree? > | > | > | Dale, > | > | > | > | Yes I know this tool, thanks. Actually I ran it last night and found > | a rather large number of Strings while garbage collection is > | reporting 0 dead objects. So my next step was trying to see if I > | have a bug somewhere and if I am holding to some un-needed objects. > | > | > | > | > | BTW, the scanbackup tool does not seem to work with the backups > | generated from GemTools. It fails at LogRecordHdr > > | isBackupDataRecord > | "LOG_BACKUP_DATA_RECORD defined in logdisk.ht " (CompatibilityLevel = > | 830 and: [self recordKind = 17]) ifTrue: [^true]. ^false. > | > | > | the recordKind is was not 17 (sorry I forgot to write the number > | down, I think it was 19). > | > | > | It worked if I execute the fullBackup from topaz. > | > | > | > | > | > | > | Thierry > | > | > | > | > | > | |
In reply to this post by Dale Henrichs
Dale, Just one stone... I have not tried restoring backups from GemTools... I do not know if this is related to the issue or not but:
a couple of weeks ago I attempted to do the backup/restore dance to shrink my extent. (backup from gemtools, restore from topaz). I have done these steps many times in the past, but I was surprised this time that I was prompted for a transaction log that did not exist in the file system. During the restore 2 small transaction logs had been created. The restore process asked me for a transaction file id that was not yet created. Sorry that I do not have much data about it.
Thierry
|
In reply to this post by James Foster-8
James, I still have the backup and I just tried again to scan it but it failed at the same place. In fact it fails reading the 'firstPhyicalRecord' (see attachment).
If I inspect the header (LogRecordHdr), I see: ByteArray withAll: #(0 0 0 0 0 0 0 0 0 0 19 0 152 62 0 0 ) You can see the value 19 instead of the expected 17 (in isBackupDataRecord). My backup file is too large to send, but here is a screen copy of the hex representation of the file start. The 19 (h13) is highlighted.
Thierry Screen Shot 2012-02-09 at 2.20.49 PM.png (21K) Download Attachment Screen Shot 2012-02-09 at 2.30.04 PM.png (76K) Download Attachment |
In reply to this post by James Foster-8
Hello,
This is not specifically a Gemstone/Seaside query but I'll throw it out there anyway. I have a domain which I have used a CNAME record to point to my Amazon EC2 instance running Seaside. The www version of my domain works great. Now, if I try just the non-www version of my domain, I get forwarded to an "It Works" page from an Apache server. I had played with Apache but didn't think it was running now and I don't see an Apache process when I do ps -A. I talked to the Register.com guys and they said I had to own an A record to point and that for $50 a year, they would point the non-www address but it didn't work. Is there some sort of configuration setting on the Amazon side? Is it an NGINX setting? What do I have to do in order to get the non-www name to point to the www. name. Regards, Larry |
In reply to this post by Thelliez
Thierry,
What version of ScanBackup are you running? I don't see #'readFile' in ScanBackup in the latest code. Also, it appears that record type 19 is recognized as LOG_BACKUP_POSS_DEAD_OBJS and thus included in #'isOkayToIgnore'. ~James On Feb 9, 2012, at 1:33 PM, Thierry Thelliez wrote:
|
James, Sorry, I did not see that you had new versions of the scanbackup tool. I think that I grab it last fall.... Upgrading right now. Thierry On Thu, Feb 9, 2012 at 2:54 PM, James Foster <[hidden email]> wrote:
|
In reply to this post by Thelliez
Thierry,
I backup and restore through gemtools on a regular basis and I have never seen issues ... If you run into odd results during a restore in the future it is worth bringing that to our attention so that we can figure out what might be going wrong ... sometimes it is explainable, and sometimes it turns out to be a bug... Dale ----- Original Message ----- | From: "Thierry Thelliez" <[hidden email]> | To: "GemStone Seaside beta discussion" <[hidden email]> | Sent: Thursday, February 9, 2012 1:18:27 PM | Subject: Re: [GS/SS Beta] How to calculate the size of an object tree? | | | Dale, | | | | Just one stone... | | | I have not tried restoring backups from GemTools... | | | I do not know if this is related to the issue or not but: | a couple of weeks ago I attempted to do the backup/restore dance to | shrink my extent. (backup from gemtools, restore from topaz). I have | done these steps many times in the past, but I was surprised this | time that I was prompted for a transaction log that did not exist in | the file system. During the restore 2 small transaction logs had | been created. The restore process asked me for a transaction file id | that was not yet created. Sorry that I do not have much data about | it. | | | | | | | Thierry |
In reply to this post by Larry Kellogg
Don't use a CNAME and instead point the DNS A record to the IP for the
Amazon EC2 instance. You don't need to forward it for $50 per yr. In Apache you'll probably want a rewrite rule to send requests to yourdomain.com to www.yourdomain.com or vice versa. On 12-02-09 01:41 PM, Lawrence Kellogg wrote: > Hello, > This is not specifically a Gemstone/Seaside query but I'll throw it out there anyway. > > I have a domain which I have used a CNAME record to point to my Amazon EC2 > instance running Seaside. The www version of my domain works great. > > Now, if I try just the non-www version of my domain, I get forwarded to an "It Works" > page from an Apache server. I had played with Apache but didn't think it was running > now and I don't see an Apache process when I do ps -A. > > I talked to the Register.com guys and they said I had to own an A record > to point and that for $50 a year, they would point the non-www address but it didn't work. > > Is there some sort of configuration setting on the Amazon side? Is it an NGINX setting? > What do I have to do in order to get the non-www name to point to the www. name. > > Regards, > > Larry |
On Feb 9, 2012, at 6:32 PM, Paul DeBruicker wrote: > Don't use a CNAME and instead point the DNS A record to the IP for the Amazon EC2 instance. You don't need to forward it for $50 per yr. In Apache you'll probably want a rewrite rule to send requests to yourdomain.com to www.yourdomain.com or vice versa. > Thanks for the help. The funny thing is that the guy at Register.com seemed to imply that I needed a dedicated server in order to get a DNS A record. Is that true or false? I don't have a dedicated server, I just have an instance on Amazon EC2. Furthermore, I realize I messed up by pointing the CNAME record at the current public DNS record because if I reboot the instance, I will get a new Public DNS and my CNAME forwarding will break. I need to get an Elastic IP and associate it with the instance and use that in the CNAME record. But, I have been told that the Elastic IP replaces the current IP immediately after applying it, and that CNAME records can take 24 hours to update. So, how do I accomplish the switcheroo without causing a service outage or having to hit the server every 5 minutes to figure out when the CNAME change takes effect? The funny thing about Apache is that I'm not running it. I'm running nginx. I guess I would need a rewrite rule for nginx. I can't figure out which apache is getting hit by the non-www domain when my instance replies that httpd is not running when I try apachectl -k stop. Larry > > > > On 12-02-09 01:41 PM, Lawrence Kellogg wrote: >> Hello, >> This is not specifically a Gemstone/Seaside query but I'll throw it out there anyway. >> >> I have a domain which I have used a CNAME record to point to my Amazon EC2 >> instance running Seaside. The www version of my domain works great. >> >> Now, if I try just the non-www version of my domain, I get forwarded to an "It Works" >> page from an Apache server. I had played with Apache but didn't think it was running >> now and I don't see an Apache process when I do ps -A. >> >> I talked to the Register.com guys and they said I had to own an A record >> to point and that for $50 a year, they would point the non-www address but it didn't work. >> >> Is there some sort of configuration setting on the Amazon side? Is it an NGINX setting? >> What do I have to do in order to get the non-www name to point to the www. name. >> >> Regards, >> >> Larry > |
In reply to this post by Paul DeBruicker
This is probably a stupid error on my part, but I'll share it. So, my wife told me I needed to link the logo on the top of all the pages back to the home page, which is a dynamically generated Seaside page. Without thinking too hard about it, I coded this: url: 'http://www.example.com'; with: [ html image url: 'example.jpg' ] and clicked on the image from within one of other Seaside pages. Boom, it went into an infinite recursion, (I think), and everything fell down. Eventually some Error 404 got spit out. It took be quite a while to get everything shut down, and restarted to bring the app back to life. So, I can see why this is a big no-no. I don't need to really get back to that login page, I just need to build some links on the other pages to some static content. Thoughts? Larry On Feb 9, 2012, at 6:32 PM, Paul DeBruicker wrote:
|
In reply to this post by Thelliez
James, I ran the updated ScanBackup tool (v15) and it worked fine. And I am getting the same numbers. Sorry for the confusion. The version I had had been working ok previously.
Now, why do I have so many Strings... ;-) Thanks! Thierry |
Thierry,
You can find the reference path for an object (i.e., the path (paths) back to a persistent root) using Repository>>findReferencePathToObject: The global SystemRepository is the sole instance of Repository, so you'd do this: SystemRepository findReferencePathToObject: 'why am i alive' There is a family of findReferencePath* methods and read the comments for clues on making sense of the results ... Occasionally, you'll end up with the result sets in your reference path results so it's a bit tedious but it should get the job done ... Dale ----- Original Message ----- | From: "Thierry Thelliez" <[hidden email]> | To: "GemStone Seaside beta discussion" <[hidden email]> | Sent: Friday, February 10, 2012 7:28:25 AM | Subject: Re: [GS/SS Beta] How to calculate the size of an object tree? | | | James, | | | I ran the updated ScanBackup tool (v15) and it worked fine. And I am | getting the same numbers. | | | Sorry for the confusion. The version I had had been working ok | previously. | | | | | Now, why do I have so many Strings... ;-) | | | Thanks! | Thierry | | | | | | | | | |
In reply to this post by Larry Kellogg
Am 09.02.2012 um 22:41 schrieb Lawrence Kellogg: > Hello, > This is not specifically a Gemstone/Seaside query but I'll throw it out there anyway. > > I have a domain which I have used a CNAME record to point to my Amazon EC2 > instance running Seaside. The www version of my domain works great. > > Now, if I try just the non-www version of my domain, I get forwarded to an "It Works" > page from an Apache server. I had played with Apache but didn't think it was running > now and I don't see an Apache process when I do ps -A. > > I talked to the Register.com guys and they said I had to own an A record > to point and that for $50 a year, they would point the non-www address but it didn't work. > > Is there some sort of configuration setting on the Amazon side? Is it an NGINX setting? > What do I have to do in order to get the non-www name to point to the www. name. Larry, you won't get it made without extra effort I guess. Basically a CNAME is the right thing to do. It is exact the case it was designed for which is a well-misunderstood fact. Unfortunately CNAME does not work on zones itself (the non www name). That is a shortcoming of DNS. On the other hand a CNAME would introduce problems when you reboot a machine. It will take up to 3 hours to have your CNAME point to the right address. Just look yourself > dig soa compute-1.amazonaws.com +short pdns1.ultradns.net. hostmaster.amazon.com. 2011110102 10800 3600 2592000 60 The number 10800 is the zone TTL meaning the number of seconds it should be cached. So for two reasons the CNAME does not seem what you are looking for. Amazon offers Route 53 DNS service where you could shorten the TTL. But that wouldn't solve the access for your domain ip. The easiest will be to buy an elastic IP or if you have a big installation an elastic load balancer. Then you just need to make nginx aware that there are www. and none www names it should react upon (you can just write the server names separated by space in the server name line. I don't know what is causing the "It works". But the "It works" is just a html file. Probably nginx points to the same document root as an apache installation. Usually to be found in /var/www/ Well...this is not an advize to do it but there is one option you can do if saving money is your top priority. You can use an dyndns provider to serve your domain. There are ready made scripts that are triggered when your machine is booted. The script reads the assigned IP address and updates the entry at your dyndns provider. The TTL for dyndns domains is usually 60 seconds. That could solve your troubles. But then you would need your domain to reside at the dyndns provider. Otherwise there is less chance to update the IP address for your non-www name. And maybe there are also costs involved hope this helps, Norbert |
In reply to this post by Dale Henrichs
Hello,After a long series of calls and answers in Seaside, I got an Object does not exist error that brought down the fast cgi gateway. My production error handler was not triggered, although it was installed, as far as I could tell. Strange. My supervise fastcgi process also did not pick up my dead fastcgi process. Not a good day.Here is the walkback: USER IDS: REAL=seasideuser (500) EFFECTIVE=seasideuser (500) | |_____________________________________________________________________________| topaz> topaz> topaz> topaz> topaz> topaz> topaz> [Info]: LNK client/gem GCI levels = 844/844 [Info]: User ID: DataCurator [Info]: Repository: seaside [Info]: Session ID: 5 [Info]: GCI Client Host: <Linked> [Info]: Page server PID: -1 [Info]: Login Time: 02/10/2012 09:27:56 PM.572 UTC [02/10/2012 09:27:56 PM.592 UTC] gci login: currSession 1 rpc gem processId -1 successful login topaz 1> topaz 1> [20 sz:0 cls: 76289 UndefinedObject] nil topaz 1> topaz 1> [76033 sz:19 cls: 801793 System class] System class superClass [72193 sz:19 cls: 206081 Object class] Object class format [258 sz:0 cls: 74241 SmallInteger] 32 instVars [2 sz:0 cls: 74241 SmallInteger] 0 instVarNames [422145 sz:0 cls: 66817 Array] an Array constraints [421889 sz:0 cls: 66817 Array] an Array classVars [421633 sz:26 cls: 111361 SymbolDictionary] a SymbolDictionary methodDict [711937 sz:112 cls: 99073 GsMethodDictionary] a GsMethodDictionary poolDictionaries [421377 sz:0 cls: 66817 Array] an Array categories [421121 sz:28 cls: 99073 GsMethodDictionary] a GsMethodDictionary secondarySuperclasses [20 sz:0 cls: 76289 UndefinedObject] nil name [801537 sz:6 cls: 110849 Symbol] System classHistory [420865 sz:3 cls: 82689 ClassHistory] a ClassHistory description [13498369 sz:10 cls: 95233 GsClassDocumentation] a GsClassDocumentation migrationDestination [20 sz:0 cls: 76289 UndefinedObject] nil timeStamp [14510593 sz:4 cls: 118785 DateTime] a DateTime userId [14508801 sz:10 cls: 74753 String] SystemUser extraDict [14524929 sz:14 cls: 111361 SymbolDictionary] a SymbolDictionary classCategory [20 sz:0 cls: 76289 UndefinedObject] nil subclasses [20 sz:0 cls: 76289 UndefinedObject] nil topaz 1> topaz 1> WAFastCGIAdaptor Server started on port 9001 ----------------------------------------------------- GemStone: Error Nonfatal The object with object ID 81065894683350528 does not exist. Error Category: 231169 [GemStone] Number: 2101 Arg Count: 1 Context : 526648321 Arg 1: [648527157466804226 sz:0 cls: 74241 SmallInteger] 81065894683350528 Now executing the following command saved from "iferr 1": where ==> 1 GsProcess class >> installPartialContinuation:atLevel:value: @2 line 1 [GsMethod 4487425] 2 WAPartialContinuation >> value: @13 line 11 [GsMethod 212791041] 3 WAPartialContinuation >> valueWithPossibleArguments: @2 line 2 [GsMethod 218326273] 4 ComplexBlock in WAComponent >> show:onAnswer:delegation: @7 line 7 [GsMethod 194749185] 5 ComplexBlock in ExecutableBlock >> valueWithPossibleArguments: @12 line 8 [GsMethod 116163585] 6 WAAnswerHandler >> handleAnswer:continueWith: @3 line 2 [GsMethod 194735873] 7 WADecoration >> handleAnswer: @6 line 3 [GsMethod 194731009] 8 WAComponent >> answer: @2 line 5 [GsMethod 194751745] I found this on the net: Object does not exist errorsGBS applications may encounter object does not exist errors in non-persistentobjects. This is due to a problem with the fix for bug #40703 in version 2.4.4, which resulted in non-persistent objects not properly protected from garbage collection. Workaround: none/not applicable from: So, should I worry about this? I implemented code that allowed a user to call an editor, switch out on the navigation menu, call another editor, switch out. With each switch, I send answer: false to the open editor to close it so that it is not hanging around open. My idea was to hide the section navigation when the editor came up but I'm getting push back from my user. What do you think? Larry |
In reply to this post by NorbertHartl
Hello Norbert,
Thanks for all the great information. I'm still puzzling my way through it. Some comments below. On Feb 10, 2012, at 2:08 PM, Norbert Hartl wrote:
Funny, the Register.com people said it could take up to 24 hours to propagate a CNAME change, unfortunately.
Well, I did a little more experimenting with my system. I have a staging server that has a DNA name that starts with EC2. So, I introduced this rewrite rule in nginx, if ($host !~* ^www\.) { rewrite ^(.*)$ http://www.$host$1 permanent; } killed nginx and restarted it and saw that the url was rewritten to www.EC2, not found... So, I tried to do the same thing on my production server, which does have a www. redirection from Register.com but the rewrite rule did not work. The only difference was that I used -HUP to restart nginx. not sure if killing and restarting nginx would make a difference. So, I'm kind of confused by what happened. # nginx -t -c /etc/nginx/nginx.conf 2007/10/18 20:55:07 [info] 3125#0: the configuration file /etc/nginx/nginx.conf syntax is ok 2007/10/18 20:55:07 [info] 3125#0: the configuration file /etc/nginx/nginx.conf was tested successfully Next, look for the process id of the master nginx process: # ps -ef|grep nginx
root 1911 1 0 18:00 ? 00:00:00 nginx: master process /usr/sbin/nginx
www-data 1912 1911 0 18:00 ? 00:00:00 nginx: worker process
Lastly, tell nginx to reload the configuration and restart the worker processes: # kill -HUP 1911
I was told by the Register guys that I would need an A record in order to do ip address forwarding. Is that true? I thought you could only get an A record if you had a physical server. I have no physical server, just aRegister.com domain pointing at Amazon EC2. I'm wondering if I have to worry about getting an elastic ip. How long will a large instance stay up? I am paying for two large instances? If the thing never goes down, maybe I'll just leave it alone. I would like to switch it over to a more stable configuration at some point, though... Regards, Larry
|
Free forum by Nabble | Edit this page |