[Glass] GemTools-1.0-beta.8.7.1-32x can't log in to GS 3.2.4

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
57 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry ...

Ah, I had missed the method that James mentioned ... Repository>>#'findReferencePathToObject:'...

That just might get us the answer we are looking for ...

Do an allInstances on WASession and get the oop of an old instance ...

then logout and to the following (using your oop):

  System abortTransaction.
  SystemRepository findReferencePathToObject: (Object _objectForOop: 119868673)

This will take a bit, but you will get back an array that looks something like this:

(class)@ -> Array
(oop)@   -> 365061121
(size)@  -> 11
1@       -> aWASession
2@       -> true
3@       -> WADispatcher
4@       -> aWADispatcher
5@       -> aDictionary( 'browse'->aWAApplication, 'javascript'->aWADispatcher, 'seaside'->aWALegacyRedirectionHandler, 'examples'->aWADispatcher, 'stat...
6@       -> 'browse'->aWAApplication
7@       -> aWAApplication
8@       -> aWACache
9@       -> aRcKeyValueDictionary( '_pIYr5fSpwCZ3W4b'->aWASession, '8t5reroNR8Typqki'->aWASession, 'eXX4KnQGvkUOFEN7'->aWASession, '1hV8xpORJw_5YSey'->a...
10@      -> aRcCollisionBucket
11@      -> aWASession

The last slot in the array is the target object and the 3rd slot in the array is the root object ... the case I've got here is the expected reference path for an unexpired session ... but if you run this in your data base, you should find an unexpected reference path and we'll be able to clean this up ...

This method only returns _a reference path_ (there could be others) but this should give us a useful path to start identifying the object leak...

Sorry about missing Repository>>#'findReferencePathToObject:' ...

Dale

On 04/03/2015 12:33 PM, James Foster wrote:
It appears that I mistakenly sent this as a private email rather than to the group as I intended. I apologize for the confusion.

On Apr 1, 2015, at 8:09 PM, James Foster <[hidden email]> wrote:

Larry,

I’m glad that the backup scan worked for you (it doesn’t get tested with every release). The object count list is quite interesting. Three hundred million instances of RcCounterElement is certainly a surprise.

What jumps out at me is that there are over a hundred thousand instances of WAPartialContinuation. This suggests that some Seaside cache is still holding things. Dale will likely have some good advice on how to find them but a couple ideas occur to me. 

First, grab one of the instances (Class>>#’allInstances’) and then look at one (or more) reference paths (Repository>>#'findReferencePathToObject:*’ or #’Repository>>#’findAllReferencePathsToObject:’ if you are in 2.4.6 or later). 

Another thing that would be interesting is to replace all the continuations with a new object (each become: Object new). After another MFC/reclaim cycle there should not be any continuation instances and it would be interesting to see what sort of space that frees (Repository>>#’fileSizeReport’). 

Of course, knowing who is holding on to them is the primary question.

James

On Apr 1, 2015, at 7:05 PM, Lawrence Kellogg via Glass <[hidden email]> wrote:


On Mar 31, 2015, at 12:41 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...

So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...

So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.


Yes, the collection at UserGlobals at: #ExpiryCleanup is empty. 

I ran the reapSeasideCache code again.



Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...

Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...


I cleared the caches. 

I ran another MFC

<Screen Shot 2015-04-01 at 10.02.25 PM.png>



At this point there's basically two directions that we can take:

  1. Top down. Start inspecting the data structures in your application and look
      for suspicious collections/objects that could be hanging onto objects above and
      beyond those absolutely needed.

  2. Bottom up. Scan your recent backup and get an instance count report[2] that
      will tell you what class of object is clogging up your data base .... Perhaps you'll
      recognize a big runner or two and know where to look to drop the references.
     If no, we'll have to pick a suspicious class, list the instances of that class, and then
     Repository>>listReferences:  to work our way back to a known root and then
     NUKE THE SUCKER:)


Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask  and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.

1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView


Larry



Dale

[1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches
[2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/
On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 6:24 PM, Dale Henrichs <[hidden email]> wrote:

The initial MFC gave you (pre-backup):

  390,801,691 live objects with 23,382,898 dead

The second MFC gave you (post-backup):

  391,007,811 live objects with 107 dead

Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...

So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...

I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...

I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …


Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has: 

<Mail Attachment.png>


Is it necessary to reinitialize the ObjectQueue?

Is there some report I can run that will tell me what is holding onto so much space?

Best,

Larry



Dale

On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results


Ok, I made it through another mark for collection and here is the result:

<Mail Attachment.png>




Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much. 





 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...


Here is the file size report before the mark for collection 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3478.58 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3478.58 Megabytes

and after 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3476.47 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3476.47 Megabytes



I await further instructions. 

Best,

Larry





By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Ok, I did what you said but I’m at a bit of a loss as to what to do next. Here are my results:

1 - aWASession
2 - true
3 - WADispatcher
4 - aWADispatcher
5 - aDictionary( 'CreateMusicalPieceRepertoireItemInterface'->aWAApplication, 'LessonTaskRecordingUpdatePrivacySettings'->aWAApplication, 'RetrieveRepertoireItemRequestHandler'->aRetrieveRepertoireItemRequestHandler, 'PracticeSessionUpdateTimePracticedInterface'->aWAApplication, 'CreateMethodBookInterface'->aWAApplication, 'CreatePracticeSessionRequestHandler'->aCreatePracticeSessionRequestHandler, 'LessonTaskRecordingsRequestHandler'->aLessonTaskRecordingsRequestHandler, 'UserActivityRequestHandler'->anUserActivityRequestHandler, 'EmailOptOutInterface'->aWAApplication, 'LessonTaskInformationServer'->aWAApplication, 'UserRegistrationRequestHandler'->anUserRegistrationRequestHandler, 'GetMethodBooks'->aWAApplication, …)
6 - 'UserInformationInterface'->aWAApplication
7 - aWAApplication
8 - aWACache
9 - aRcKeyValueDictionary( '5jTYOuLPzkOYLv4d'->aWASession, 'ePxPIpWmyhtOHA0U'->aWASession, 'zRyovUbd2YaYCEuC'->aWADocumentHandler, 'Qwhb4dokS54vxz03'->aWASession, 'bsXyv4UGt5-gbrEo'->aWASession, 'qTDuZk9Mfu9cllpR'->aWADocumentHandler, 'bSnQRSCCU6tLSuHy'->aWADocumentHandler, 'Dv8_EljXp4UeBCYm'->aWADocumentHandler, 'JRp685a4lHUKhDiT'->aWASession, 'JG6o8KnnCuSOtNjx'->aWASession, 'VqMHwuCbBLRFo40Y'->aWASession, '1pRW-D5PPlAhW37g'->aWASession, 'VyzeJ-GNB8GF-aMJ'->aWASession, 'YOVib-WowYRZNJaN'->aWADocumentHandler, 'I9PoXhB7gBlGMNgY'->aWADocumentHandler, '-lyxYEftM3kUE_Wg'->aWASession, 'ZVnusqwGHRr1Hb7_'->aWADocumentHandler, '9Vch6_hBThnx_eln'->aWASession, 'wNwBarOaES_2Uigs'->aWADocumentHandler, 'oSCks-wA3lHOqbPu'->aWADocumentHandler, ...)


10 - aRcCollisionBucket
11 - aWASession

What do I do now? 

Larry




On Apr 3, 2015, at 3:45 PM, Dale Henrichs <[hidden email]> wrote:

Larry ...

Ah, I had missed the method that James mentioned ... Repository>>#'findReferencePathToObject:'...

That just might get us the answer we are looking for ...

Do an allInstances on WASession and get the oop of an old instance ...

then logout and to the following (using your oop):

  System abortTransaction.
  SystemRepository findReferencePathToObject: (Object _objectForOop: 119868673)

This will take a bit, but you will get back an array that looks something like this:

(class)@ -> Array
(oop)@   -> 365061121
(size)@  -> 11
1@       -> aWASession
2@       -> true
3@       -> WADispatcher
4@       -> aWADispatcher
5@       -> aDictionary( 'browse'->aWAApplication, 'javascript'->aWADispatcher, 'seaside'->aWALegacyRedirectionHandler, 'examples'->aWADispatcher, 'stat...
6@       -> 'browse'->aWAApplication
7@       -> aWAApplication
8@       -> aWACache
9@       -> aRcKeyValueDictionary( '_pIYr5fSpwCZ3W4b'->aWASession, '8t5reroNR8Typqki'->aWASession, 'eXX4KnQGvkUOFEN7'->aWASession, '1hV8xpORJw_5YSey'->a...
10@      -> aRcCollisionBucket
11@      -> aWASession

The last slot in the array is the target object and the 3rd slot in the array is the root object ... the case I've got here is the expected reference path for an unexpired session ... but if you run this in your data base, you should find an unexpected reference path and we'll be able to clean this up ...

This method only returns _a reference path_ (there could be others) but this should give us a useful path to start identifying the object leak...

Sorry about missing Repository>>#'findReferencePathToObject:' ...

Dale

On 04/03/2015 12:33 PM, James Foster wrote:
It appears that I mistakenly sent this as a private email rather than to the group as I intended. I apologize for the confusion.

On Apr 1, 2015, at 8:09 PM, James Foster <[hidden email]> wrote:

Larry,

I’m glad that the backup scan worked for you (it doesn’t get tested with every release). The object count list is quite interesting. Three hundred million instances of RcCounterElement is certainly a surprise.

What jumps out at me is that there are over a hundred thousand instances of WAPartialContinuation. This suggests that some Seaside cache is still holding things. Dale will likely have some good advice on how to find them but a couple ideas occur to me. 

First, grab one of the instances (Class>>#’allInstances’) and then look at one (or more) reference paths (Repository>>#'findReferencePathToObject:*’ or #’Repository>>#’findAllReferencePathsToObject:’ if you are in 2.4.6 or later). 

Another thing that would be interesting is to replace all the continuations with a new object (each become: Object new). After another MFC/reclaim cycle there should not be any continuation instances and it would be interesting to see what sort of space that frees (Repository>>#’fileSizeReport’). 

Of course, knowing who is holding on to them is the primary question.

James

On Apr 1, 2015, at 7:05 PM, Lawrence Kellogg via Glass <[hidden email]> wrote:


On Mar 31, 2015, at 12:41 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...

So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...

So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.


Yes, the collection at UserGlobals at: #ExpiryCleanup is empty. 

I ran the reapSeasideCache code again.



Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...

Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...


I cleared the caches. 

I ran another MFC

<Screen Shot 2015-04-01 at 10.02.25 PM.png>



At this point there's basically two directions that we can take:

  1. Top down. Start inspecting the data structures in your application and look
      for suspicious collections/objects that could be hanging onto objects above and
      beyond those absolutely needed.

  2. Bottom up. Scan your recent backup and get an instance count report[2] that
      will tell you what class of object is clogging up your data base .... Perhaps you'll
      recognize a big runner or two and know where to look to drop the references.
     If no, we'll have to pick a suspicious class, list the instances of that class, and then
     Repository>>listReferences:  to work our way back to a known root and then
     NUKE THE SUCKER:)


Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask  and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.

1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView


Larry



Dale

[1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches
[2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/
On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 6:24 PM, Dale Henrichs <[hidden email]> wrote:

The initial MFC gave you (pre-backup):

  390,801,691 live objects with 23,382,898 dead

The second MFC gave you (post-backup):

  391,007,811 live objects with 107 dead

Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...

So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...

I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...

I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …


Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has: 

<Mail Attachment.png>


Is it necessary to reinitialize the ObjectQueue?

Is there some report I can run that will tell me what is holding onto so much space?

Best,

Larry



Dale

On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results


Ok, I made it through another mark for collection and here is the result:

<Mail Attachment.png>




Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much. 





 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...


Here is the file size report before the mark for collection 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3478.58 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3478.58 Megabytes

and after 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3476.47 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3476.47 Megabytes



I await further instructions. 

Best,

Larry





By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass





_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

Well, this looks like the session expiration code didn't expire the session state as expected:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransaction

So we'll need to understand why ... and the first thing I want to know is the result of this:

  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: #'objectsByKey'.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

This is basically the expression that gets evaluated when we are trying to remove the expired sessions.

If it returns true, then something is wrong with the reaping code and we'll want to step through that logic..

If it returns false, then we need to understand why this presumably ancient session is not expired ...

Dale


On 04/06/2015 02:56 PM, Lawrence Kellogg wrote:
Ok, I did what you said but I’m at a bit of a loss as to what to do next. Here are my results:

1 - aWASession
2 - true
3 - WADispatcher
4 - aWADispatcher
5 - aDictionary( 'CreateMusicalPieceRepertoireItemInterface'->aWAApplication, 'LessonTaskRecordingUpdatePrivacySettings'->aWAApplication, 'RetrieveRepertoireItemRequestHandler'->aRetrieveRepertoireItemRequestHandler, 'PracticeSessionUpdateTimePracticedInterface'->aWAApplication, 'CreateMethodBookInterface'->aWAApplication, 'CreatePracticeSessionRequestHandler'->aCreatePracticeSessionRequestHandler, 'LessonTaskRecordingsRequestHandler'->aLessonTaskRecordingsRequestHandler, 'UserActivityRequestHandler'->anUserActivityRequestHandler, 'EmailOptOutInterface'->aWAApplication, 'LessonTaskInformationServer'->aWAApplication, 'UserRegistrationRequestHandler'->anUserRegistrationRequestHandler, 'GetMethodBooks'->aWAApplication, …)
6 - 'UserInformationInterface'->aWAApplication
7 - aWAApplication
8 - aWACache
9 - aRcKeyValueDictionary( '5jTYOuLPzkOYLv4d'->aWASession, 'ePxPIpWmyhtOHA0U'->aWASession, 'zRyovUbd2YaYCEuC'->aWADocumentHandler, 'Qwhb4dokS54vxz03'->aWASession, 'bsXyv4UGt5-gbrEo'->aWASession, 'qTDuZk9Mfu9cllpR'->aWADocumentHandler, 'bSnQRSCCU6tLSuHy'->aWADocumentHandler, 'Dv8_EljXp4UeBCYm'->aWADocumentHandler, 'JRp685a4lHUKhDiT'->aWASession, 'JG6o8KnnCuSOtNjx'->aWASession, 'VqMHwuCbBLRFo40Y'->aWASession, '1pRW-D5PPlAhW37g'->aWASession, 'VyzeJ-GNB8GF-aMJ'->aWASession, 'YOVib-WowYRZNJaN'->aWADocumentHandler, 'I9PoXhB7gBlGMNgY'->aWADocumentHandler, '-lyxYEftM3kUE_Wg'->aWASession, 'ZVnusqwGHRr1Hb7_'->aWADocumentHandler, '9Vch6_hBThnx_eln'->aWASession, 'wNwBarOaES_2Uigs'->aWADocumentHandler, 'oSCks-wA3lHOqbPu'->aWADocumentHandler, ...)


10 - aRcCollisionBucket
11 - aWASession

What do I do now? 

Larry




On Apr 3, 2015, at 3:45 PM, Dale Henrichs <[hidden email]> wrote:

Larry ...

Ah, I had missed the method that James mentioned ... Repository>>#'findReferencePathToObject:'...

That just might get us the answer we are looking for ...

Do an allInstances on WASession and get the oop of an old instance ...

then logout and to the following (using your oop):

  System abortTransaction.
  SystemRepository findReferencePathToObject: (Object _objectForOop: 119868673)

This will take a bit, but you will get back an array that looks something like this:

(class)@ -> Array
(oop)@   -> 365061121
(size)@  -> 11
1@       -> aWASession
2@       -> true
3@       -> WADispatcher
4@       -> aWADispatcher
5@       -> aDictionary( 'browse'->aWAApplication, 'javascript'->aWADispatcher, 'seaside'->aWALegacyRedirectionHandler, 'examples'->aWADispatcher, 'stat...
6@       -> 'browse'->aWAApplication
7@       -> aWAApplication
8@       -> aWACache
9@       -> aRcKeyValueDictionary( '_pIYr5fSpwCZ3W4b'->aWASession, '8t5reroNR8Typqki'->aWASession, 'eXX4KnQGvkUOFEN7'->aWASession, '1hV8xpORJw_5YSey'->a...
10@      -> aRcCollisionBucket
11@      -> aWASession

The last slot in the array is the target object and the 3rd slot in the array is the root object ... the case I've got here is the expected reference path for an unexpired session ... but if you run this in your data base, you should find an unexpected reference path and we'll be able to clean this up ...

This method only returns _a reference path_ (there could be others) but this should give us a useful path to start identifying the object leak...

Sorry about missing Repository>>#'findReferencePathToObject:' ...

Dale

On 04/03/2015 12:33 PM, James Foster wrote:
It appears that I mistakenly sent this as a private email rather than to the group as I intended. I apologize for the confusion.

On Apr 1, 2015, at 8:09 PM, James Foster <[hidden email]> wrote:

Larry,

I’m glad that the backup scan worked for you (it doesn’t get tested with every release). The object count list is quite interesting. Three hundred million instances of RcCounterElement is certainly a surprise.

What jumps out at me is that there are over a hundred thousand instances of WAPartialContinuation. This suggests that some Seaside cache is still holding things. Dale will likely have some good advice on how to find them but a couple ideas occur to me. 

First, grab one of the instances (Class>>#’allInstances’) and then look at one (or more) reference paths (Repository>>#'findReferencePathToObject:*’ or #’Repository>>#’findAllReferencePathsToObject:’ if you are in 2.4.6 or later). 

Another thing that would be interesting is to replace all the continuations with a new object (each become: Object new). After another MFC/reclaim cycle there should not be any continuation instances and it would be interesting to see what sort of space that frees (Repository>>#’fileSizeReport’). 

Of course, knowing who is holding on to them is the primary question.

James

On Apr 1, 2015, at 7:05 PM, Lawrence Kellogg via Glass <[hidden email]> wrote:


On Mar 31, 2015, at 12:41 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...

So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...

So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.


Yes, the collection at UserGlobals at: #ExpiryCleanup is empty. 

I ran the reapSeasideCache code again.



Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...

Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...


I cleared the caches. 

I ran another MFC

<Screen Shot 2015-04-01 at 10.02.25 PM.png>



At this point there's basically two directions that we can take:

  1. Top down. Start inspecting the data structures in your application and look
      for suspicious collections/objects that could be hanging onto objects above and
      beyond those absolutely needed.

  2. Bottom up. Scan your recent backup and get an instance count report[2] that
      will tell you what class of object is clogging up your data base .... Perhaps you'll
      recognize a big runner or two and know where to look to drop the references.
     If no, we'll have to pick a suspicious class, list the instances of that class, and then
     Repository>>listReferences:  to work our way back to a known root and then
     NUKE THE SUCKER:)


Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask  and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.

1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView


Larry



Dale

[1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches
[2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/
On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 6:24 PM, Dale Henrichs <[hidden email]> wrote:

The initial MFC gave you (pre-backup):

  390,801,691 live objects with 23,382,898 dead

The second MFC gave you (post-backup):

  391,007,811 live objects with 107 dead

Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...

So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...

I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...

I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …


Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has: 

<Mail Attachment.png>


Is it necessary to reinitialize the ObjectQueue?

Is there some report I can run that will tell me what is holding onto so much space?

Best,

Larry



Dale

On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results


Ok, I made it through another mark for collection and here is the result:

<Mail Attachment.png>




Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much. 





 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...


Here is the file size report before the mark for collection 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3478.58 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3478.58 Megabytes

and after 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3476.47 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3476.47 Megabytes



I await further instructions. 

Best,

Larry





By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass






_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list

On Apr 6, 2015, at 6:20 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Well, this looks like the session expiration code didn't expire the session state as expected:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransaction

So we'll need to understand why ... and the first thing I want to know is the result of this:

  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: #'objectsByKey'.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

This is basically the expression that gets evaluated when we are trying to remove the expired sessions.

If it returns true, then something is wrong with the reaping code and we'll want to step through that logic..

If it returns false, then we need to understand why this presumably ancient session is not expired ...



Ok, I had to rewrite a few things in your code because it crashed, so, I did this, 


  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey at:'5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

and it returns false.

So, I guess we have to figure out why this old session is not expired.

Larry




Dale


On 04/06/2015 02:56 PM, Lawrence Kellogg wrote:
Ok, I did what you said but I’m at a bit of a loss as to what to do next. Here are my results:

1 - aWASession
2 - true
3 - WADispatcher
4 - aWADispatcher
5 - aDictionary( 'CreateMusicalPieceRepertoireItemInterface'->aWAApplication, 'LessonTaskRecordingUpdatePrivacySettings'->aWAApplication, 'RetrieveRepertoireItemRequestHandler'->aRetrieveRepertoireItemRequestHandler, 'PracticeSessionUpdateTimePracticedInterface'->aWAApplication, 'CreateMethodBookInterface'->aWAApplication, 'CreatePracticeSessionRequestHandler'->aCreatePracticeSessionRequestHandler, 'LessonTaskRecordingsRequestHandler'->aLessonTaskRecordingsRequestHandler, 'UserActivityRequestHandler'->anUserActivityRequestHandler, 'EmailOptOutInterface'->aWAApplication, 'LessonTaskInformationServer'->aWAApplication, 'UserRegistrationRequestHandler'->anUserRegistrationRequestHandler, 'GetMethodBooks'->aWAApplication, …)
6 - 'UserInformationInterface'->aWAApplication
7 - aWAApplication
8 - aWACache
9 - aRcKeyValueDictionary( '5jTYOuLPzkOYLv4d'->aWASession, 'ePxPIpWmyhtOHA0U'->aWASession, 'zRyovUbd2YaYCEuC'->aWADocumentHandler, 'Qwhb4dokS54vxz03'->aWASession, 'bsXyv4UGt5-gbrEo'->aWASession, 'qTDuZk9Mfu9cllpR'->aWADocumentHandler, 'bSnQRSCCU6tLSuHy'->aWADocumentHandler, 'Dv8_EljXp4UeBCYm'->aWADocumentHandler, 'JRp685a4lHUKhDiT'->aWASession, 'JG6o8KnnCuSOtNjx'->aWASession, 'VqMHwuCbBLRFo40Y'->aWASession, '1pRW-D5PPlAhW37g'->aWASession, 'VyzeJ-GNB8GF-aMJ'->aWASession, 'YOVib-WowYRZNJaN'->aWADocumentHandler, 'I9PoXhB7gBlGMNgY'->aWADocumentHandler, '-lyxYEftM3kUE_Wg'->aWASession, 'ZVnusqwGHRr1Hb7_'->aWADocumentHandler, '9Vch6_hBThnx_eln'->aWASession, 'wNwBarOaES_2Uigs'->aWADocumentHandler, 'oSCks-wA3lHOqbPu'->aWADocumentHandler, ...)


10 - aRcCollisionBucket
11 - aWASession

What do I do now? 

Larry




On Apr 3, 2015, at 3:45 PM, Dale Henrichs <[hidden email]> wrote:

Larry ...

Ah, I had missed the method that James mentioned ... Repository>>#'findReferencePathToObject:'...

That just might get us the answer we are looking for ...

Do an allInstances on WASession and get the oop of an old instance ...

then logout and to the following (using your oop):

  System abortTransaction.
  SystemRepository findReferencePathToObject: (Object _objectForOop: 119868673)

This will take a bit, but you will get back an array that looks something like this:

(class)@ -> Array
(oop)@   -> 365061121
(size)@  -> 11
1@       -> aWASession
2@       -> true
3@       -> WADispatcher
4@       -> aWADispatcher
5@       -> aDictionary( 'browse'->aWAApplication, 'javascript'->aWADispatcher, 'seaside'->aWALegacyRedirectionHandler, 'examples'->aWADispatcher, 'stat...
6@       -> 'browse'->aWAApplication
7@       -> aWAApplication
8@       -> aWACache
9@       -> aRcKeyValueDictionary( '_pIYr5fSpwCZ3W4b'->aWASession, '8t5reroNR8Typqki'->aWASession, 'eXX4KnQGvkUOFEN7'->aWASession, '1hV8xpORJw_5YSey'->a...
10@      -> aRcCollisionBucket
11@      -> aWASession

The last slot in the array is the target object and the 3rd slot in the array is the root object ... the case I've got here is the expected reference path for an unexpired session ... but if you run this in your data base, you should find an unexpected reference path and we'll be able to clean this up ...

This method only returns _a reference path_ (there could be others) but this should give us a useful path to start identifying the object leak...

Sorry about missing Repository>>#'findReferencePathToObject:' ...

Dale

On 04/03/2015 12:33 PM, James Foster wrote:
It appears that I mistakenly sent this as a private email rather than to the group as I intended. I apologize for the confusion.

On Apr 1, 2015, at 8:09 PM, James Foster <[hidden email]> wrote:

Larry,

I’m glad that the backup scan worked for you (it doesn’t get tested with every release). The object count list is quite interesting. Three hundred million instances of RcCounterElement is certainly a surprise.

What jumps out at me is that there are over a hundred thousand instances of WAPartialContinuation. This suggests that some Seaside cache is still holding things. Dale will likely have some good advice on how to find them but a couple ideas occur to me. 

First, grab one of the instances (Class>>#’allInstances’) and then look at one (or more) reference paths (Repository>>#'findReferencePathToObject:*’ or #’Repository>>#’findAllReferencePathsToObject:’ if you are in 2.4.6 or later). 

Another thing that would be interesting is to replace all the continuations with a new object (each become: Object new). After another MFC/reclaim cycle there should not be any continuation instances and it would be interesting to see what sort of space that frees (Repository>>#’fileSizeReport’). 

Of course, knowing who is holding on to them is the primary question.

James

On Apr 1, 2015, at 7:05 PM, Lawrence Kellogg via Glass <[hidden email]> wrote:


On Mar 31, 2015, at 12:41 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...

So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...

So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.


Yes, the collection at UserGlobals at: #ExpiryCleanup is empty. 

I ran the reapSeasideCache code again.



Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...

Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...


I cleared the caches. 

I ran another MFC

<Screen Shot 2015-04-01 at 10.02.25 PM.png>



At this point there's basically two directions that we can take:

  1. Top down. Start inspecting the data structures in your application and look
      for suspicious collections/objects that could be hanging onto objects above and
      beyond those absolutely needed.

  2. Bottom up. Scan your recent backup and get an instance count report[2] that
      will tell you what class of object is clogging up your data base .... Perhaps you'll
      recognize a big runner or two and know where to look to drop the references.
     If no, we'll have to pick a suspicious class, list the instances of that class, and then
     Repository>>listReferences:  to work our way back to a known root and then
     NUKE THE SUCKER:)


Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask  and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.

1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView


Larry



Dale

[1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches
[2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/
On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 6:24 PM, Dale Henrichs <[hidden email]> wrote:

The initial MFC gave you (pre-backup):

  390,801,691 live objects with 23,382,898 dead

The second MFC gave you (post-backup):

  391,007,811 live objects with 107 dead

Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...

So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...

I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...

I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …


Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has: 

<Mail Attachment.png>


Is it necessary to reinitialize the ObjectQueue?

Is there some report I can run that will tell me what is holding onto so much space?

Best,

Larry



Dale

On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results


Ok, I made it through another mark for collection and here is the result:

<Mail Attachment.png>




Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much. 





 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...


Here is the file size report before the mark for collection 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3478.58 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3478.58 Megabytes

and after 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3476.47 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3476.47 Megabytes



I await further instructions. 

Best,

Larry





By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass







_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

I've been reading code a bit, looking for some sort of path that would foil the gemstoneReap code and the most likely scenario I can find has to do with this code in WARcLastAccessExpiryPolicy>>isExpired:key:

  isExpired: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ WARcLastAccessEntry new ].
  ^ entry isExpired: self timeout

and WARcLastAccessExpiryPolicy>>isExpiredUpdating:key:

  isExpiredUpdating: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ WARcLastAccessEntry new ].
  ^ entry isExpiredUpdating: self timeout

So if somehow the lastAccessTable has lost a key for a session, then the session will survive forever ...

We can test this by the following code:

  | app cache lastAccessTable |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  lastAccessTable at: '5jTYOuLPzkOYLv4d' ifAbsent: [ 'missing' ]

I expect the result to be 'missing' ...

Based on this expectation, I'll start writing some repair code ...

Dale

On 04/06/2015 03:30 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 6:20 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Well, this looks like the session expiration code didn't expire the session state as expected:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransaction

So we'll need to understand why ... and the first thing I want to know is the result of this:

  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: #'objectsByKey'.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

This is basically the expression that gets evaluated when we are trying to remove the expired sessions.

If it returns true, then something is wrong with the reaping code and we'll want to step through that logic..

If it returns false, then we need to understand why this presumably ancient session is not expired ...



Ok, I had to rewrite a few things in your code because it crashed, so, I did this, 


  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey at:'5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

and it returns false.

So, I guess we have to figure out why this old session is not expired.

Larry




Dale


On 04/06/2015 02:56 PM, Lawrence Kellogg wrote:
Ok, I did what you said but I’m at a bit of a loss as to what to do next. Here are my results:

1 - aWASession
2 - true
3 - WADispatcher
4 - aWADispatcher
5 - aDictionary( 'CreateMusicalPieceRepertoireItemInterface'->aWAApplication, 'LessonTaskRecordingUpdatePrivacySettings'->aWAApplication, 'RetrieveRepertoireItemRequestHandler'->aRetrieveRepertoireItemRequestHandler, 'PracticeSessionUpdateTimePracticedInterface'->aWAApplication, 'CreateMethodBookInterface'->aWAApplication, 'CreatePracticeSessionRequestHandler'->aCreatePracticeSessionRequestHandler, 'LessonTaskRecordingsRequestHandler'->aLessonTaskRecordingsRequestHandler, 'UserActivityRequestHandler'->anUserActivityRequestHandler, 'EmailOptOutInterface'->aWAApplication, 'LessonTaskInformationServer'->aWAApplication, 'UserRegistrationRequestHandler'->anUserRegistrationRequestHandler, 'GetMethodBooks'->aWAApplication, …)
6 - 'UserInformationInterface'->aWAApplication
7 - aWAApplication
8 - aWACache
9 - aRcKeyValueDictionary( '5jTYOuLPzkOYLv4d'->aWASession, 'ePxPIpWmyhtOHA0U'->aWASession, 'zRyovUbd2YaYCEuC'->aWADocumentHandler, 'Qwhb4dokS54vxz03'->aWASession, 'bsXyv4UGt5-gbrEo'->aWASession, 'qTDuZk9Mfu9cllpR'->aWADocumentHandler, 'bSnQRSCCU6tLSuHy'->aWADocumentHandler, 'Dv8_EljXp4UeBCYm'->aWADocumentHandler, 'JRp685a4lHUKhDiT'->aWASession, 'JG6o8KnnCuSOtNjx'->aWASession, 'VqMHwuCbBLRFo40Y'->aWASession, '1pRW-D5PPlAhW37g'->aWASession, 'VyzeJ-GNB8GF-aMJ'->aWASession, 'YOVib-WowYRZNJaN'->aWADocumentHandler, 'I9PoXhB7gBlGMNgY'->aWADocumentHandler, '-lyxYEftM3kUE_Wg'->aWASession, 'ZVnusqwGHRr1Hb7_'->aWADocumentHandler, '9Vch6_hBThnx_eln'->aWASession, 'wNwBarOaES_2Uigs'->aWADocumentHandler, 'oSCks-wA3lHOqbPu'->aWADocumentHandler, ...)


10 - aRcCollisionBucket
11 - aWASession

What do I do now? 

Larry




On Apr 3, 2015, at 3:45 PM, Dale Henrichs <[hidden email]> wrote:

Larry ...

Ah, I had missed the method that James mentioned ... Repository>>#'findReferencePathToObject:'...

That just might get us the answer we are looking for ...

Do an allInstances on WASession and get the oop of an old instance ...

then logout and to the following (using your oop):

  System abortTransaction.
  SystemRepository findReferencePathToObject: (Object _objectForOop: 119868673)

This will take a bit, but you will get back an array that looks something like this:

(class)@ -> Array
(oop)@   -> 365061121
(size)@  -> 11
1@       -> aWASession
2@       -> true
3@       -> WADispatcher
4@       -> aWADispatcher
5@       -> aDictionary( 'browse'->aWAApplication, 'javascript'->aWADispatcher, 'seaside'->aWALegacyRedirectionHandler, 'examples'->aWADispatcher, 'stat...
6@       -> 'browse'->aWAApplication
7@       -> aWAApplication
8@       -> aWACache
9@       -> aRcKeyValueDictionary( '_pIYr5fSpwCZ3W4b'->aWASession, '8t5reroNR8Typqki'->aWASession, 'eXX4KnQGvkUOFEN7'->aWASession, '1hV8xpORJw_5YSey'->a...
10@      -> aRcCollisionBucket
11@      -> aWASession

The last slot in the array is the target object and the 3rd slot in the array is the root object ... the case I've got here is the expected reference path for an unexpired session ... but if you run this in your data base, you should find an unexpected reference path and we'll be able to clean this up ...

This method only returns _a reference path_ (there could be others) but this should give us a useful path to start identifying the object leak...

Sorry about missing Repository>>#'findReferencePathToObject:' ...

Dale

On 04/03/2015 12:33 PM, James Foster wrote:
It appears that I mistakenly sent this as a private email rather than to the group as I intended. I apologize for the confusion.

On Apr 1, 2015, at 8:09 PM, James Foster <[hidden email]> wrote:

Larry,

I’m glad that the backup scan worked for you (it doesn’t get tested with every release). The object count list is quite interesting. Three hundred million instances of RcCounterElement is certainly a surprise.

What jumps out at me is that there are over a hundred thousand instances of WAPartialContinuation. This suggests that some Seaside cache is still holding things. Dale will likely have some good advice on how to find them but a couple ideas occur to me. 

First, grab one of the instances (Class>>#’allInstances’) and then look at one (or more) reference paths (Repository>>#'findReferencePathToObject:*’ or #’Repository>>#’findAllReferencePathsToObject:’ if you are in 2.4.6 or later). 

Another thing that would be interesting is to replace all the continuations with a new object (each become: Object new). After another MFC/reclaim cycle there should not be any continuation instances and it would be interesting to see what sort of space that frees (Repository>>#’fileSizeReport’). 

Of course, knowing who is holding on to them is the primary question.

James

On Apr 1, 2015, at 7:05 PM, Lawrence Kellogg via Glass <[hidden email]> wrote:


On Mar 31, 2015, at 12:41 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...

So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...

So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.


Yes, the collection at UserGlobals at: #ExpiryCleanup is empty. 

I ran the reapSeasideCache code again.



Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...

Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...


I cleared the caches. 

I ran another MFC

<Screen Shot 2015-04-01 at 10.02.25 PM.png>



At this point there's basically two directions that we can take:

  1. Top down. Start inspecting the data structures in your application and look
      for suspicious collections/objects that could be hanging onto objects above and
      beyond those absolutely needed.

  2. Bottom up. Scan your recent backup and get an instance count report[2] that
      will tell you what class of object is clogging up your data base .... Perhaps you'll
      recognize a big runner or two and know where to look to drop the references.
     If no, we'll have to pick a suspicious class, list the instances of that class, and then
     Repository>>listReferences:  to work our way back to a known root and then
     NUKE THE SUCKER:)


Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask  and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.

1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView


Larry



Dale

[1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches
[2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/
On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 6:24 PM, Dale Henrichs <[hidden email]> wrote:

The initial MFC gave you (pre-backup):

  390,801,691 live objects with 23,382,898 dead

The second MFC gave you (post-backup):

  391,007,811 live objects with 107 dead

Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...

So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...

I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...

I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …


Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has: 

<Mail Attachment.png>


Is it necessary to reinitialize the ObjectQueue?

Is there some report I can run that will tell me what is holding onto so much space?

Best,

Larry



Dale

On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results


Ok, I made it through another mark for collection and here is the result:

<Mail Attachment.png>




Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much. 





 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...


Here is the file size report before the mark for collection 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3478.58 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3478.58 Megabytes

and after 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3476.47 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3476.47 Megabytes



I await further instructions. 

Best,

Larry





By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass








_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Okay, I think we'll try taking this route with WACacheExpiryPolicy>>isExpiredUpdating:key:

  isExpiredUpdating: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ ^true ].
  ^ entry isExpiredUpdating: self timeout

This code should force any sessions with a missing `lastAccessTable` to be expired ...

Looking at the WACache>>gemstoneReap code, I think I can tell how we load the entries in the `lastAccessTable`.

If you notice, in the first loop in WACache>>gemstoneReap, that WACache>>notifyRemoved:key: is called for each of the expired sessions and then when we are looping through `expired` sessions we start committing every 100 sessions ... now after we have processed the first 100 sessions, we commit the removal of ALL of the sessions from the `lastAccessTable` but we only commit the removal of the first 100 sessions from from the `objectsByKey` and `keysByObect` tables and if we crash before finishing the whole operation, we will have corrupted the `lastAccessTable` ... and I seem to recall that we did have such a crash ...

This one will take a while to run, but we should be able to get most of the repository cleaned up ...

I will submit a bug on the `lastAccessTable` problem ...

Dale

On 04/06/2015 03:47 PM, Dale Henrichs wrote:
Larry,

I've been reading code a bit, looking for some sort of path that would foil the gemstoneReap code and the most likely scenario I can find has to do with this code in WARcLastAccessExpiryPolicy>>isExpired:key:

  isExpired: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ WARcLastAccessEntry new ].
  ^ entry isExpired: self timeout

and WARcLastAccessExpiryPolicy>>isExpiredUpdating:key:

  isExpiredUpdating: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ WARcLastAccessEntry new ].
  ^ entry isExpiredUpdating: self timeout

So if somehow the lastAccessTable has lost a key for a session, then the session will survive forever ...

We can test this by the following code:

  | app cache lastAccessTable |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  lastAccessTable at: '5jTYOuLPzkOYLv4d' ifAbsent: [ 'missing' ]

I expect the result to be 'missing' ...

Based on this expectation, I'll start writing some repair code ...

Dale

On 04/06/2015 03:30 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 6:20 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Well, this looks like the session expiration code didn't expire the session state as expected:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransaction

So we'll need to understand why ... and the first thing I want to know is the result of this:

  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: #'objectsByKey'.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

This is basically the expression that gets evaluated when we are trying to remove the expired sessions.

If it returns true, then something is wrong with the reaping code and we'll want to step through that logic..

If it returns false, then we need to understand why this presumably ancient session is not expired ...



Ok, I had to rewrite a few things in your code because it crashed, so, I did this, 


  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey at:'5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

and it returns false.

So, I guess we have to figure out why this old session is not expired.

Larry




Dale


On 04/06/2015 02:56 PM, Lawrence Kellogg wrote:
Ok, I did what you said but I’m at a bit of a loss as to what to do next. Here are my results:

1 - aWASession
2 - true
3 - WADispatcher
4 - aWADispatcher
5 - aDictionary( 'CreateMusicalPieceRepertoireItemInterface'->aWAApplication, 'LessonTaskRecordingUpdatePrivacySettings'->aWAApplication, 'RetrieveRepertoireItemRequestHandler'->aRetrieveRepertoireItemRequestHandler, 'PracticeSessionUpdateTimePracticedInterface'->aWAApplication, 'CreateMethodBookInterface'->aWAApplication, 'CreatePracticeSessionRequestHandler'->aCreatePracticeSessionRequestHandler, 'LessonTaskRecordingsRequestHandler'->aLessonTaskRecordingsRequestHandler, 'UserActivityRequestHandler'->anUserActivityRequestHandler, 'EmailOptOutInterface'->aWAApplication, 'LessonTaskInformationServer'->aWAApplication, 'UserRegistrationRequestHandler'->anUserRegistrationRequestHandler, 'GetMethodBooks'->aWAApplication, …)
6 - 'UserInformationInterface'->aWAApplication
7 - aWAApplication
8 - aWACache
9 - aRcKeyValueDictionary( '5jTYOuLPzkOYLv4d'->aWASession, 'ePxPIpWmyhtOHA0U'->aWASession, 'zRyovUbd2YaYCEuC'->aWADocumentHandler, 'Qwhb4dokS54vxz03'->aWASession, 'bsXyv4UGt5-gbrEo'->aWASession, 'qTDuZk9Mfu9cllpR'->aWADocumentHandler, 'bSnQRSCCU6tLSuHy'->aWADocumentHandler, 'Dv8_EljXp4UeBCYm'->aWADocumentHandler, 'JRp685a4lHUKhDiT'->aWASession, 'JG6o8KnnCuSOtNjx'->aWASession, 'VqMHwuCbBLRFo40Y'->aWASession, '1pRW-D5PPlAhW37g'->aWASession, 'VyzeJ-GNB8GF-aMJ'->aWASession, 'YOVib-WowYRZNJaN'->aWADocumentHandler, 'I9PoXhB7gBlGMNgY'->aWADocumentHandler, '-lyxYEftM3kUE_Wg'->aWASession, 'ZVnusqwGHRr1Hb7_'->aWADocumentHandler, '9Vch6_hBThnx_eln'->aWASession, 'wNwBarOaES_2Uigs'->aWADocumentHandler, 'oSCks-wA3lHOqbPu'->aWADocumentHandler, ...)


10 - aRcCollisionBucket
11 - aWASession

What do I do now? 

Larry




On Apr 3, 2015, at 3:45 PM, Dale Henrichs <[hidden email]> wrote:

Larry ...

Ah, I had missed the method that James mentioned ... Repository>>#'findReferencePathToObject:'...

That just might get us the answer we are looking for ...

Do an allInstances on WASession and get the oop of an old instance ...

then logout and to the following (using your oop):

  System abortTransaction.
  SystemRepository findReferencePathToObject: (Object _objectForOop: 119868673)

This will take a bit, but you will get back an array that looks something like this:

(class)@ -> Array
(oop)@   -> 365061121
(size)@  -> 11
1@       -> aWASession
2@       -> true
3@       -> WADispatcher
4@       -> aWADispatcher
5@       -> aDictionary( 'browse'->aWAApplication, 'javascript'->aWADispatcher, 'seaside'->aWALegacyRedirectionHandler, 'examples'->aWADispatcher, 'stat...
6@       -> 'browse'->aWAApplication
7@       -> aWAApplication
8@       -> aWACache
9@       -> aRcKeyValueDictionary( '_pIYr5fSpwCZ3W4b'->aWASession, '8t5reroNR8Typqki'->aWASession, 'eXX4KnQGvkUOFEN7'->aWASession, '1hV8xpORJw_5YSey'->a...
10@      -> aRcCollisionBucket
11@      -> aWASession

The last slot in the array is the target object and the 3rd slot in the array is the root object ... the case I've got here is the expected reference path for an unexpired session ... but if you run this in your data base, you should find an unexpected reference path and we'll be able to clean this up ...

This method only returns _a reference path_ (there could be others) but this should give us a useful path to start identifying the object leak...

Sorry about missing Repository>>#'findReferencePathToObject:' ...

Dale

On 04/03/2015 12:33 PM, James Foster wrote:
It appears that I mistakenly sent this as a private email rather than to the group as I intended. I apologize for the confusion.

On Apr 1, 2015, at 8:09 PM, James Foster <[hidden email]> wrote:

Larry,

I’m glad that the backup scan worked for you (it doesn’t get tested with every release). The object count list is quite interesting. Three hundred million instances of RcCounterElement is certainly a surprise.

What jumps out at me is that there are over a hundred thousand instances of WAPartialContinuation. This suggests that some Seaside cache is still holding things. Dale will likely have some good advice on how to find them but a couple ideas occur to me. 

First, grab one of the instances (Class>>#’allInstances’) and then look at one (or more) reference paths (Repository>>#'findReferencePathToObject:*’ or #’Repository>>#’findAllReferencePathsToObject:’ if you are in 2.4.6 or later). 

Another thing that would be interesting is to replace all the continuations with a new object (each become: Object new). After another MFC/reclaim cycle there should not be any continuation instances and it would be interesting to see what sort of space that frees (Repository>>#’fileSizeReport’). 

Of course, knowing who is holding on to them is the primary question.

James

On Apr 1, 2015, at 7:05 PM, Lawrence Kellogg via Glass <[hidden email]> wrote:


On Mar 31, 2015, at 12:41 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...

So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...

So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.


Yes, the collection at UserGlobals at: #ExpiryCleanup is empty. 

I ran the reapSeasideCache code again.



Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...

Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...


I cleared the caches. 

I ran another MFC

<Screen Shot 2015-04-01 at 10.02.25 PM.png>



At this point there's basically two directions that we can take:

  1. Top down. Start inspecting the data structures in your application and look
      for suspicious collections/objects that could be hanging onto objects above and
      beyond those absolutely needed.

  2. Bottom up. Scan your recent backup and get an instance count report[2] that
      will tell you what class of object is clogging up your data base .... Perhaps you'll
      recognize a big runner or two and know where to look to drop the references.
     If no, we'll have to pick a suspicious class, list the instances of that class, and then
     Repository>>listReferences:  to work our way back to a known root and then
     NUKE THE SUCKER:)


Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask  and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.

1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView


Larry



Dale

[1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches
[2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/
On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 6:24 PM, Dale Henrichs <[hidden email]> wrote:

The initial MFC gave you (pre-backup):

  390,801,691 live objects with 23,382,898 dead

The second MFC gave you (post-backup):

  391,007,811 live objects with 107 dead

Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...

So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...

I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...

I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …


Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has: 

<Mail Attachment.png>


Is it necessary to reinitialize the ObjectQueue?

Is there some report I can run that will tell me what is holding onto so much space?

Best,

Larry



Dale

On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results


Ok, I made it through another mark for collection and here is the result:

<Mail Attachment.png>




Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much. 





 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...


Here is the file size report before the mark for collection 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3478.58 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3478.58 Megabytes

and after 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3476.47 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3476.47 Megabytes



I await further instructions. 

Best,

Larry





By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Here is the bug report[1] ... I know that this is a bit premature, because I haven't quite gotten confirmation from Larry, but I think the logic is "just plain wrong" based on code reading:)

Dale

[1] https://github.com/GsDevKit/Seaside31/issues/68

On 04/06/2015 03:57 PM, Dale Henrichs wrote:
Okay, I think we'll try taking this route with WACacheExpiryPolicy>>isExpiredUpdating:key:

  isExpiredUpdating: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ ^true ].
  ^ entry isExpiredUpdating: self timeout

This code should force any sessions with a missing `lastAccessTable` to be expired ...

Looking at the WACache>>gemstoneReap code, I think I can tell how we load the entries in the `lastAccessTable`.

If you notice, in the first loop in WACache>>gemstoneReap, that WACache>>notifyRemoved:key: is called for each of the expired sessions and then when we are looping through `expired` sessions we start committing every 100 sessions ... now after we have processed the first 100 sessions, we commit the removal of ALL of the sessions from the `lastAccessTable` but we only commit the removal of the first 100 sessions from from the `objectsByKey` and `keysByObect` tables and if we crash before finishing the whole operation, we will have corrupted the `lastAccessTable` ... and I seem to recall that we did have such a crash ...

This one will take a while to run, but we should be able to get most of the repository cleaned up ...

I will submit a bug on the `lastAccessTable` problem ...

Dale

On 04/06/2015 03:47 PM, Dale Henrichs wrote:
Larry,

I've been reading code a bit, looking for some sort of path that would foil the gemstoneReap code and the most likely scenario I can find has to do with this code in WARcLastAccessExpiryPolicy>>isExpired:key:

  isExpired: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ WARcLastAccessEntry new ].
  ^ entry isExpired: self timeout

and WARcLastAccessExpiryPolicy>>isExpiredUpdating:key:

  isExpiredUpdating: anObject key: aString
  | entry |
  entry := lastAccessTable at: aString ifAbsent: [ WARcLastAccessEntry new ].
  ^ entry isExpiredUpdating: self timeout

So if somehow the lastAccessTable has lost a key for a session, then the session will survive forever ...

We can test this by the following code:

  | app cache lastAccessTable |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  lastAccessTable at: '5jTYOuLPzkOYLv4d' ifAbsent: [ 'missing' ]

I expect the result to be 'missing' ...

Based on this expectation, I'll start writing some repair code ...

Dale

On 04/06/2015 03:30 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 6:20 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

Well, this looks like the session expiration code didn't expire the session state as expected:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransaction

So we'll need to understand why ... and the first thing I want to know is the result of this:

  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: #'objectsByKey'.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

This is basically the expression that gets evaluated when we are trying to remove the expired sessions.

If it returns true, then something is wrong with the reaping code and we'll want to step through that logic..

If it returns false, then we need to understand why this presumably ancient session is not expired ...



Ok, I had to rewrite a few things in your code because it crashed, so, I did this, 


  | app cache objectsByKey assoc |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey at:'5jTYOuLPzkOYLv4d'.
  cache expiryPolicy isExpired: assoc value key: assoc key

and it returns false.

So, I guess we have to figure out why this old session is not expired.

Larry




Dale


On 04/06/2015 02:56 PM, Lawrence Kellogg wrote:
Ok, I did what you said but I’m at a bit of a loss as to what to do next. Here are my results:

1 - aWASession
2 - true
3 - WADispatcher
4 - aWADispatcher
5 - aDictionary( 'CreateMusicalPieceRepertoireItemInterface'->aWAApplication, 'LessonTaskRecordingUpdatePrivacySettings'->aWAApplication, 'RetrieveRepertoireItemRequestHandler'->aRetrieveRepertoireItemRequestHandler, 'PracticeSessionUpdateTimePracticedInterface'->aWAApplication, 'CreateMethodBookInterface'->aWAApplication, 'CreatePracticeSessionRequestHandler'->aCreatePracticeSessionRequestHandler, 'LessonTaskRecordingsRequestHandler'->aLessonTaskRecordingsRequestHandler, 'UserActivityRequestHandler'->anUserActivityRequestHandler, 'EmailOptOutInterface'->aWAApplication, 'LessonTaskInformationServer'->aWAApplication, 'UserRegistrationRequestHandler'->anUserRegistrationRequestHandler, 'GetMethodBooks'->aWAApplication, …)
6 - 'UserInformationInterface'->aWAApplication
7 - aWAApplication
8 - aWACache
9 - aRcKeyValueDictionary( '5jTYOuLPzkOYLv4d'->aWASession, 'ePxPIpWmyhtOHA0U'->aWASession, 'zRyovUbd2YaYCEuC'->aWADocumentHandler, 'Qwhb4dokS54vxz03'->aWASession, 'bsXyv4UGt5-gbrEo'->aWASession, 'qTDuZk9Mfu9cllpR'->aWADocumentHandler, 'bSnQRSCCU6tLSuHy'->aWADocumentHandler, 'Dv8_EljXp4UeBCYm'->aWADocumentHandler, 'JRp685a4lHUKhDiT'->aWASession, 'JG6o8KnnCuSOtNjx'->aWASession, 'VqMHwuCbBLRFo40Y'->aWASession, '1pRW-D5PPlAhW37g'->aWASession, 'VyzeJ-GNB8GF-aMJ'->aWASession, 'YOVib-WowYRZNJaN'->aWADocumentHandler, 'I9PoXhB7gBlGMNgY'->aWADocumentHandler, '-lyxYEftM3kUE_Wg'->aWASession, 'ZVnusqwGHRr1Hb7_'->aWADocumentHandler, '9Vch6_hBThnx_eln'->aWASession, 'wNwBarOaES_2Uigs'->aWADocumentHandler, 'oSCks-wA3lHOqbPu'->aWADocumentHandler, ...)


10 - aRcCollisionBucket
11 - aWASession

What do I do now? 

Larry




On Apr 3, 2015, at 3:45 PM, Dale Henrichs <[hidden email]> wrote:

Larry ...

Ah, I had missed the method that James mentioned ... Repository>>#'findReferencePathToObject:'...

That just might get us the answer we are looking for ...

Do an allInstances on WASession and get the oop of an old instance ...

then logout and to the following (using your oop):

  System abortTransaction.
  SystemRepository findReferencePathToObject: (Object _objectForOop: 119868673)

This will take a bit, but you will get back an array that looks something like this:

(class)@ -> Array
(oop)@   -> 365061121
(size)@  -> 11
1@       -> aWASession
2@       -> true
3@       -> WADispatcher
4@       -> aWADispatcher
5@       -> aDictionary( 'browse'->aWAApplication, 'javascript'->aWADispatcher, 'seaside'->aWALegacyRedirectionHandler, 'examples'->aWADispatcher, 'stat...
6@       -> 'browse'->aWAApplication
7@       -> aWAApplication
8@       -> aWACache
9@       -> aRcKeyValueDictionary( '_pIYr5fSpwCZ3W4b'->aWASession, '8t5reroNR8Typqki'->aWASession, 'eXX4KnQGvkUOFEN7'->aWASession, '1hV8xpORJw_5YSey'->a...
10@      -> aRcCollisionBucket
11@      -> aWASession

The last slot in the array is the target object and the 3rd slot in the array is the root object ... the case I've got here is the expected reference path for an unexpired session ... but if you run this in your data base, you should find an unexpected reference path and we'll be able to clean this up ...

This method only returns _a reference path_ (there could be others) but this should give us a useful path to start identifying the object leak...

Sorry about missing Repository>>#'findReferencePathToObject:' ...

Dale

On 04/03/2015 12:33 PM, James Foster wrote:
It appears that I mistakenly sent this as a private email rather than to the group as I intended. I apologize for the confusion.

On Apr 1, 2015, at 8:09 PM, James Foster <[hidden email]> wrote:

Larry,

I’m glad that the backup scan worked for you (it doesn’t get tested with every release). The object count list is quite interesting. Three hundred million instances of RcCounterElement is certainly a surprise.

What jumps out at me is that there are over a hundred thousand instances of WAPartialContinuation. This suggests that some Seaside cache is still holding things. Dale will likely have some good advice on how to find them but a couple ideas occur to me. 

First, grab one of the instances (Class>>#’allInstances’) and then look at one (or more) reference paths (Repository>>#'findReferencePathToObject:*’ or #’Repository>>#’findAllReferencePathsToObject:’ if you are in 2.4.6 or later). 

Another thing that would be interesting is to replace all the continuations with a new object (each become: Object new). After another MFC/reclaim cycle there should not be any continuation instances and it would be interesting to see what sort of space that frees (Repository>>#’fileSizeReport’). 

Of course, knowing who is holding on to them is the primary question.

James

On Apr 1, 2015, at 7:05 PM, Lawrence Kellogg via Glass <[hidden email]> wrote:


On Mar 31, 2015, at 12:41 PM, Dale Henrichs <[hidden email]> wrote:

Larry,

I'm just going over the old ground again, in case we missed something obvious ... I would hate to spend several more days digging into this only to find that an initial step hadn't completed as expected ...

So it looks like the object log is clear. Next I'd like to double check and make sure that the session state has been expired ...

So let's verify that `UserGlobals at: #'ExpiryCleanup` is no longer present and I'd like to run the WABasicDevelopment reapSeasideCache one more time for good luck.


Yes, the collection at UserGlobals at: #ExpiryCleanup is empty. 

I ran the reapSeasideCache code again.



Assuming that neither of those turn up anything of use, the next step is to find out what's hanging onto the unwanted objects ...

Since I think we've covered the known "object hogs" in the Seaside framework, there are a number of other persistent caches in GLASS, that might as well be cleared out. You can use the workspace here[1] to clean them up ... I don't think that these caches should be holding onto 23G of objects, but run an MFC aftwards to be safe ...


I cleared the caches. 

I ran another MFC

<Screen Shot 2015-04-01 at 10.02.25 PM.png>



At this point there's basically two directions that we can take:

  1. Top down. Start inspecting the data structures in your application and look
      for suspicious collections/objects that could be hanging onto objects above and
      beyond those absolutely needed.

  2. Bottom up. Scan your recent backup and get an instance count report[2] that
      will tell you what class of object is clogging up your data base .... Perhaps you'll
      recognize a big runner or two and know where to look to drop the references.
     If no, we'll have to pick a suspicious class, list the instances of that class, and then
     Repository>>listReferences:  to work our way back to a known root and then
     NUKE THE SUCKER:)


Ok, here is my instance count report. RCCounterElement is a huge winner here, I have no idea why. #63 PracticeJournalLoginTask  and #65 PracticeJournalSession is coming up a lot, so perhaps these are being held onto somewhere.

1 338955617 RcCounterElement
2 17607121 RcCollisionBucket
3 7683895 Association
4 2142624 String
5 2126557 WAValueHolder
6 1959784 VariableContext
7 1629389 CollisionBucket
8 1464171 Dictionary
9 1339617 KeyValueDictionary
10 1339616 Set
11 1243135 OrderedCollection
12 1116296 Array
13 951872 ComplexBlock
14 943639 ComplexVCBlock
15 781212 IdentityDictionary
16 673104 IdentityCollisionBucket
17 666407 WAUserConfiguration
18 664701 WAAttributeSearchContext
19 338617 RcCounter
20 338617 WARcLastAccessEntry
21 332017 RcKeyValueDictionary
22 230240 WAValueCallback
23 226002 WARequestFields
24 226002 WAUrl
25 223641 GRSmallDictionary
26 221821 GRDelayedSend
27 221821 GRUnboundMessage
28 220824 GsStackBuffer
29 219296 WAImageCallback
30 187813 Date
31 176258 MCMethodDefinition
32 146263 WAActionCallback
33 113114 WARenderCanvas
34 113039 WAMimeType
35 113003 WADocumentHandler
36 113003 WAMimeDocument
37 113001 WARenderVisitor
38 113001 WAActionPhaseContinuation
39 113001 WACallbackRegistry
40 113001 WARenderingGuide
41 113001 WARenderContext
42 112684 WASnapshot
43 110804 IdentityBag
44 110720 TransientValue
45 110710 WAToolDecoration
46 110672 TransientMutex
47 110670 WAGemStoneMutex
48 110670 WARcLastAccessExpiryPolicy
49 110670 WACache
50 110670 WANoReapingStrategy
51 110670 WACacheMissStrategy
52 110670 WANotifyRemovalAction
53 110640 WATimingToolFilter
54 110640 WADeprecatedToolFilter
55 110489 WAAnswerHandler
56 110422 WADelegation
57 110412 WAPartialContinuation
58 110412 GsProcess
59 109773 UserPersonalInformation
60 109712 Student
61 109295 WATaskVisitor
62 109285 UserLoginView
63 109285 PracticeJournalLoginTask
64 109259 WAValueExpression
65 109215 PracticeJournalSession
66 56942 Time
67 54394 GsMethod
68 53207 MCVersionInfo
69 53207 UUID
70 45927 MethodVersionRecord
71 41955 MethodBookExercise
72 37223 Symbol
73 29941 MCInstanceVariableDefinition
74 21828 MCClassDefinition
75 19291 SymbolAssociation
76 18065 PracticeDay
77 17218 GsMethodDictionary
78 16617 MusicalPiece
79 16609 SymbolSet
80 11160 FreeformExercise
81 8600 SymbolDictionary
82 7537 DateAndTime
83 6812 Duration
84 6288 Month
85 6288 PracticeMonth
86 4527 WAHtmlAttributes
87 4390 DateTime
88 4247 Metaclass
89 4190 WAGenericTag
90 4142 SimpleBlock
91 4136 WATableColumnTag
92 4136 WACheckboxTag
93 4029 Composer
94 3682 RcIdentityBag
95 3428 ClassHistory
96 3010 PracticeSession
97 2185 MCClassVariableDefinition
98 2017 CanonStringBucket
99 1986 MethodBook
100 1974 WARenderPhaseContinuation
101 1965 PurchaseOptionInformation
102 1843 AmazonPurchase
103 1796 GsDocText
104 1513 GsClassDocumentation
105 1508 209409
106 1425 WASession
107 1218 UserInformationInterface
108 1134 WAValuesCallback
109 1125 WACancelActionCallback
110 751 DepListBucket
111 738 Pragma
112 716 LessonTaskRecording
113 693 UserForgotPasswordView
114 629 MusicalPieceRepertoireItem
115 524 PracticeYear
116 524 Year
117 483 MCOrganizationDefinition
118 480 Repertoire
119 467 MCPackage
120 440 MultiplePageDisplayView
121 403 MethodBookExerciseRepertoireItem
122 352 UserCalendar
123 334 MetacelloValueHolderSpec
124 333 MCVersion
125 333 MCSnapshot
126 313 TimeZoneTransition
127 307 Color
128 269 NumberGenerator
129 216 UserCommunityInformation
130 206 IdentitySet
131 200 RcQueueSessionComponent
132 199 FreeformExerciseRepertoireItem
133 191 WAHtmlCanvas
134 187 PackageInfo
135 182 InvariantArray
136 176 MCRepositoryGroup
137 175 MCWorkingCopy
138 175 MCWorkingAncestry
139 157 PracticeSessionInputView
140 149 MetacelloPackageSpec
141 139 MetacelloRepositoriesSpec
142 132 WAMetaElement
143 131 MCClassInstanceVariableDefinition
144 117 MetacelloMergeMemberSpec
145 106 YouTubeVideoResource
146 101 MusicalPieceRepertoireItemInputView
147 99 UserCommentsView
148 96 LessonTasksView
149 96 LessonTaskView
150 94 WATableTag
151 91 MetacelloMCVersion
152 87 PracticeSessionView
153 81 SortedCollection
154 78 MetacelloMCVersionSpec
155 78 MetacelloVersionNumber
156 78 MetacelloPackagesSpec
157 77 DateRange
158 77 PracticeSessionsView
159 70 MethodBooksView
160 67 UserCalendarView
161 67 PracticeJournalMiniCalendar
162 66 PracticeDayView
163 65 MetacelloAddMemberSpec
164 64 WATextInputTag
165 61 Teacher
166 61 MCPoolImportDefinition
167 61 MCHttpRepository
168 60 CheckScreenNameAvailability
169 58 UserRepertoireItemsView
170 58 UserRepertoireView
171 58 PrivateLesson
172 57 UserRepertoireItemsSummaryView
173 53 MetacelloMCProjectSpec
174 53 MetacelloProjectReferenceSpec
175 52 WAListAttribute
176 48 TimedActivitiesInformationServer
177 46 PracticeSessionTemplate
178 46 WriteStream
179 44 WAFormTag
180 39 UserInstrumentsInputView
181 39 MetacelloRepositorySpec
182 35 UserInstrumentsInputViewGenerator
183 35 CreateLessonTaskRecordingInterface
184 32 WASelectTag
185 32 WADateInput
186 30 WAApplication
187 30 UserComment
188 30 WAExceptionFilter
189 29 WADispatchCallback
190 29 WARadioGroup
191 28 DecimalFloat
192 27 JSStream
193 26 MethodBookExerciseRepertoireItemInputView
194 26 WAStringAttribute
195 24 WAOpeningConditionalComment
196 24 WAScriptElement
197 24 WAClosingConditionalComment
198 24 PracticeSessionTemplateInputView
199 24 WALinkElement
200 23 UserInformationView


Larry



Dale

[1] https://code.google.com/p/glassdb/wiki/ClearPersistentCaches
[2] https://programminggems.wordpress.com/2009/12/15/scanbackup-2/
On 03/31/2015 05:35 AM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 6:24 PM, Dale Henrichs <[hidden email]> wrote:

The initial MFC gave you (pre-backup):

  390,801,691 live objects with 23,382,898 dead

The second MFC gave you (post-backup):

  391,007,811 live objects with 107 dead

Which means that we did not gain nearly as much as anticipated by cleaning up the seaside session state and object log ... so something else is hanging onto a big chunk of objects ...

So yes at this point there is no need to consider a backup and restore to shrink extents until we can free up some more objects ...

I've got to head out on an errand right now, so I can't give you any detailed pointers, to the techniques to use for finding the nasty boy that is hanging onto the "presumably dead objects" ...

I am a bit suspicious that the Object log might still be alive an kicking, so I think you should verify by inspecting the ObjectLog collections ... poke around on the class side ... if you find a big collection (and it blows up your TOC if you try to look at it), then look again at the class-side methods and make sure that you nuke the RCQueue and the OrderedCollection .... close down/logout your vms, and then run another mfc to see if you gained any ground …


Well, the ObjectLog collection on the class side of ObjectLogEntry is empty, and the ObjectQueue class variable has: 

<Mail Attachment.png>


Is it necessary to reinitialize the ObjectQueue?

Is there some report I can run that will tell me what is holding onto so much space?

Best,

Larry



Dale

On 03/30/2015 02:57 PM, Lawrence Kellogg wrote:

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results


Ok, I made it through another mark for collection and here is the result:

<Mail Attachment.png>




Am I wrong in thinking that the file size of the extent will not shrink? It certainly has not shrunk much. 





 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...


Here is the file size report before the mark for collection 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3478.58 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3478.58 Megabytes

and after 

Extent #1
-----------
   Filename = !TCP@localhost#dir:/opt/gemstone/product/seaside/data#<a moz-do-not-send="true" href="log://opt/gemstone/log/%N%P.log#dbf%21/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf" class="">log://opt/gemstone/log/%N%P.log#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf

   File size =       23732.00 Megabytes
   Space available = 3476.47 Megabytes

Totals
------
   Repository size = 23732.00 Megabytes
   Free Space =      3476.47 Megabytes



I await further instructions. 

Best,

Larry





By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass










_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
In reply to this post by GLASS mailing list

On 04/06/2015 04:29 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 7:26 PM, Dale Henrichs <[hidden email]> wrote:

No problem ... no disappointment since I found some logic errors already:) ... So what are the values of the instance variables in the WARcLastAccessEntry ... in other words "why is it answering false?" ...


time - 3593198528
counter - an RcCounter with a whopping 1001 aRcCounterElements
lastCount = 0 

Does that shed any light on the situation?


the value of lastCount is suspicious ... what is the result of `counter value`

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list

On Apr 6, 2015, at 7:37 PM, Dale Henrichs <[hidden email]> wrote:


On 04/06/2015 04:29 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 7:26 PM, Dale Henrichs <[hidden email]> wrote:

No problem ... no disappointment since I found some logic errors already:) ... So what are the values of the instance variables in the WARcLastAccessEntry ... in other words "why is it answering false?" ...


time - 3593198528
counter - an RcCounter with a whopping 1001 aRcCounterElements
lastCount = 0 

Does that shed any light on the situation?


the value of lastCount is suspicious ... what is the result of `counter value`



the value is 1.

For Seaside Gemstone Core: 

MODIFIED
Ancestors: Seaside-GemStone-Core-dkh.63
===========================

Name: Seaside-GemStone-Core-dkh.63
Author: dkh
Time: 09/27/2011, 14:59:38
UUID: 81e7cc84-552b-4806-bf52-2d93f25be69c
Ancestors: Seaside-GemStone-Core-dkh.62

3.0.6.1 (dkh.334):
- remove the GemStone method seasideNextLine and add a test for Issue 289
—————————————



Dale


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Okay,

It looks like I assumed that you were using Seaside 3.1 and you must be using Seaside3.0 ... I don't think that things have actually changed in this area since Seaside 3.0, but it _is_ something to be aware of ... now...

Given that the code (in Seaside-GemStone-Core-dkh.70) in WARcLastAccessEntry>>isExpired:updating: (could you verify that your version is the same?):

isExpired: timeout updating: updating
  | value |
  ^ (value := self counter value) > lastCount
    ifTrue: [
      "accessed since last scan"
      updating
        ifTrue: [
          lastCount := value.
          time := Time totalSeconds ].
      false ]
    ifFalse: [ Time totalSeconds - time > timeout ]

it appears that we have only done a expiration scan (updating = true) only once and I have been under the impression that we've done at least two full scans (the scan before the backup and the scan after we discovered that we had not recovered any data) ... oh well ...

I think that the `time` in this entry corresponds to the date: `2014-11-11T22:42:08+00:00` (Time totalSeconds returns seconds since January 1, 1901) and that implies that the last scan was done a year ago? Unless the code is very different or ...  What does `date` return from the linux shell?

The next step is to see if the code is working like I expect it to. The code below simulates and expiration scan, and then see how the entry has changed along with couple of other things:

  | app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}

I've got to head home, but I'll check in again this evening when I get the chance ...

Dale



On 04/06/2015 04:57 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 7:37 PM, Dale Henrichs <[hidden email]> wrote:


On 04/06/2015 04:29 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 7:26 PM, Dale Henrichs <[hidden email]> wrote:

No problem ... no disappointment since I found some logic errors already:) ... So what are the values of the instance variables in the WARcLastAccessEntry ... in other words "why is it answering false?" ...


time - 3593198528
counter - an RcCounter with a whopping 1001 aRcCounterElements
lastCount = 0 

Does that shed any light on the situation?


the value of lastCount is suspicious ... what is the result of `counter value`



the value is 1.

For Seaside Gemstone Core: 

MODIFIED
Ancestors: Seaside-GemStone-Core-dkh.63
===========================

Name: Seaside-GemStone-Core-dkh.63
Author: dkh
Time: 09/27/2011, 14:59:38
UUID: 81e7cc84-552b-4806-bf52-2d93f25be69c
Ancestors: Seaside-GemStone-Core-dkh.62

3.0.6.1 (dkh.334):
- remove the GemStone method seasideNextLine and add a test for Issue 289
—————————————



Dale



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list

On Apr 6, 2015, at 8:25 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

It looks like I assumed that you were using Seaside 3.1 and you must be using Seaside3.0 ... I don't think that things have actually changed in this area since Seaside 3.0, but it _is_ something to be aware of ... now...

Given that the code (in Seaside-GemStone-Core-dkh.70) in WARcLastAccessEntry>>isExpired:updating: (could you verify that your version is the same?):

isExpired: timeout updating: updating
  | value |
  ^ (value := self counter value) > lastCount
    ifTrue: [
      "accessed since last scan"
      updating
        ifTrue: [
          lastCount := value.
          time := Time totalSeconds ].
      false ]
    ifFalse: [ Time totalSeconds - time > timeout ]

it appears that we have only done a expiration scan (updating = true) only once and I have been under the impression that we've done at least two full scans (the scan before the backup and the scan after we discovered that we had not recovered any data) ... oh well ...

I think that the `time` in this entry corresponds to the date: `2014-11-11T22:42:08+00:00` (Time totalSeconds returns seconds since January 1, 1901) and that implies that the last scan was done a year ago? Unless the code is very different or ...  What does `date` return from the linux shell?

The next step is to see if the code is working like I expect it to. The code below simulates and expiration scan, and then see how the entry has changed along with couple of other things:

  | app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}

I've got to head home, but I'll check in again this evening when I get the chance ...


Thanks, the above code returns:

anArray( aWARcLastAccessExpiryPolicy, 3605819381, false, aWARcLastAccessEntry, 3605819381, 1, 1, true)




Dale



On 04/06/2015 04:57 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 7:37 PM, Dale Henrichs <[hidden email]> wrote:


On 04/06/2015 04:29 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 7:26 PM, Dale Henrichs <[hidden email]> wrote:

No problem ... no disappointment since I found some logic errors already:) ... So what are the values of the instance variables in the WARcLastAccessEntry ... in other words "why is it answering false?" ...


time - 3593198528
counter - an RcCounter with a whopping 1001 aRcCounterElements
lastCount = 0 

Does that shed any light on the situation?


the value of lastCount is suspicious ... what is the result of `counter value`



the value is 1.

For Seaside Gemstone Core: 

MODIFIED
Ancestors: Seaside-GemStone-Core-dkh.63
===========================

Name: Seaside-GemStone-Core-dkh.63
Author: dkh
Time: 09/27/2011, 14:59:38
UUID: 81e7cc84-552b-4806-bf52-2d93f25be69c
Ancestors: Seaside-GemStone-Core-dkh.62

3.0.6.1 (dkh.334):
- remove the GemStone method seasideNextLine and add a test for Issue 289
—————————————



Dale




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list


On 4/6/15 5:30 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 8:25 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

It looks like I assumed that you were using Seaside 3.1 and you must be using Seaside3.0 ... I don't think that things have actually changed in this area since Seaside 3.0, but it _is_ something to be aware of ... now...

Given that the code (in Seaside-GemStone-Core-dkh.70) in WARcLastAccessEntry>>isExpired:updating: (could you verify that your version is the same?):

isExpired: timeout updating: updating
  | value |
  ^ (value := self counter value) > lastCount
    ifTrue: [
      "accessed since last scan"
      updating
        ifTrue: [
          lastCount := value.
          time := Time totalSeconds ].
      false ]
    ifFalse: [ Time totalSeconds - time > timeout ]

it appears that we have only done a expiration scan (updating = true) only once and I have been under the impression that we've done at least two full scans (the scan before the backup and the scan after we discovered that we had not recovered any data) ... oh well ...

I think that the `time` in this entry corresponds to the date: `2014-11-11T22:42:08+00:00` (Time totalSeconds returns seconds since January 1, 1901) and that implies that the last scan was done a year ago? Unless the code is very different or ...  What does `date` return from the linux shell?

The next step is to see if the code is working like I expect it to. The code below simulates and expiration scan, and then see how the entry has changed along with couple of other things:

  | app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}

I've got to head home, but I'll check in again this evening when I get the chance ...


Thanks, the above code returns:

anArray( aWARcLastAccessExpiryPolicy, 3605819381, false, aWARcLastAccessEntry, 3605819381, 1, 1, true)

The time, corresponds to 2015-04-07T00:29:41+00:00...so the date on your server is probably correct...

Oh shoot, the first slot should have been `cache expiryPolicy timeout`, so let's try again, with an edited workspace:

| app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy timeout.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}

From this little experiment it seems that we've only run one expiration run against this particular session ... so after you get me the results from this array and the timeout (first slot) seems to be a reasonable number (less than 12,620,853), then we will try running the expiration run again... BUT before we run anther expiration run, I want to see the code that you are using in WACache>>gemstoneReap, WABasicDevelopment class>>reapHandlerCache:dispatchers: and WABasicDevelopment class>>reapSeasideCache, just to see if I can spot a transaction error before running a second (and third) expiration.

Yes twice more ... for some unknown reason, the session we are looking has apparently not been scanned before (which is why I want to read the code that is in your image) the `lastCount` should have been equal to the `counter value` and I'm not quite sure why the lastCount hasn't been bumped ... unless the first time we touched the session is when we just ran the above workspace ...

I will have to drop offline for awhile, but I will try bringing up a stone that has an older version of Seaside installed to see if there are indeed differences in the expiration logic ...

I assume that you've been running these tests in the stone that we restored from backup ..... or have you been running these tests in a stone in which you've never run the expiration code?

Dale

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Answers below….

On Apr 6, 2015, at 9:59 PM, Dale Henrichs <[hidden email]> wrote:



On 4/6/15 5:30 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 8:25 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

It looks like I assumed that you were using Seaside 3.1 and you must be using Seaside3.0 ... I don't think that things have actually changed in this area since Seaside 3.0, but it _is_ something to be aware of ... now...

Given that the code (in Seaside-GemStone-Core-dkh.70) in WARcLastAccessEntry>>isExpired:updating: (could you verify that your version is the same?):

isExpired: timeout updating: updating
  | value |
  ^ (value := self counter value) > lastCount
    ifTrue: [
      "accessed since last scan"
      updating
        ifTrue: [
          lastCount := value.
          time := Time totalSeconds ].
      false ]
    ifFalse: [ Time totalSeconds - time > timeout ]

it appears that we have only done a expiration scan (updating = true) only once and I have been under the impression that we've done at least two full scans (the scan before the backup and the scan after we discovered that we had not recovered any data) ... oh well ...

I think that the `time` in this entry corresponds to the date: `2014-11-11T22:42:08+00:00` (Time totalSeconds returns seconds since January 1, 1901) and that implies that the last scan was done a year ago? Unless the code is very different or ...  What does `date` return from the linux shell?

The next step is to see if the code is working like I expect it to. The code below simulates and expiration scan, and then see how the entry has changed along with couple of other things:

  | app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}

I've got to head home, but I'll check in again this evening when I get the chance ...


Thanks, the above code returns:

anArray( aWARcLastAccessExpiryPolicy, 3605819381, false, aWARcLastAccessEntry, 3605819381, 1, 1, true)

The time, corresponds to 2015-04-07T00:29:41+00:00...so the date on your server is probably correct...

Oh shoot, the first slot should have been `cache expiryPolicy timeout`, so let's try again, with an edited workspace:

| app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy timeout.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}


Ok, here is the result from the above code, substituting at: for associationAt:

anArray( 600, 3605869347, true, aWARcLastAccessEntry, 3605819381, 1, 1, true)




From this little experiment it seems that we've only run one expiration run against this particular session ... so after you get me the results from this array and the timeout (first slot) seems to be a reasonable number (less than 12,620,853), then we will try running the expiration run again... BUT before we run anther expiration run, I want to see the code that you are using in


WACache>>gemstoneReap,


gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM, 
     so we are already in transaction (assumed to be running in #autoBegin 
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc | 
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [ 
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
Transcript cr; show: 'Scan progress: ' , count printString.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
platform doCommitTransaction.
count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc | 
      count := count + 1.
Transcript cr; show: 'Expire progress: ' , count printString.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
Transcript show: 'Leaving gemstoneReap: ',expired size printString.
  ^ expired size

WABasicDevelopment class>>reapHandlerCache:dispatchers: and

reapHandlerCache: aHandler dispatchers: dispatchers
| expired |
expired := 0.
(aHandler isApplication or: [ aHandler isSession ])
ifTrue: [
expired :=  aHandler cache gemstoneReap.
aHandler sessionsDo: [:session | expired := expired + session continuations gemstoneReap ]]
ifFalse: [
aHandler isDispatcher 
ifTrue: [ dispatchers add: aHandler ]].
^expired


WABasicDevelopment class>>reapSeasideCache, just to see if I can spot a transaction error before running a second (and third) expiration.

reapSeasideCache

| dps expired |
expired := 0.
dps := OrderedCollection with: WADispatcher default.
[ dps isEmpty ] whileFalse: [ | dp |
dp := dps removeFirst.
dp isApplication
ifTrue: [
expired := expired + (self reapHandlerCache: dp dispatchers: dps) ]
ifFalse: [ 
dp handlers valuesDo: [:ep |
expired := expired + (self reapHandlerCache: ep dispatchers: dps) ]]].
^expired



Yes twice more ... for some unknown reason, the session we are looking has apparently not been scanned before (which is why I want to read the code that is in your image) the `lastCount` should have been equal to the `counter value` and I'm not quite sure why the lastCount hasn't been bumped ... unless the first time we touched the session is when we just ran the above workspace ...

I will have to drop offline for awhile, but I will try bringing up a stone that has an older version of Seaside installed to see if there are indeed differences in the expiration logic ...

I assume that you've been running these tests in the stone that we restored from backup ..... or have you been running these tests in a stone in which you've never run the expiration code?


Yes, all of the test have been run from the stone that I restored from backup, and the reap code has been run multiple times, as I recall. 

Best, 

Larry



Dale


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

Okay, I cannot think of a reason that the earlier scans could not work, so I'm thinking that I'd like to pick a smaller subset of the whole system that runs faster and then try to understand why the scan isn't working ...

So I'll start by finding out if UserInformationInterface will make a likely candidate, so let's find out how big the cache is:

  | app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  objectsByKey size

I'll put together another workspace for doing a scan on a per application basis in th emean time ...

Dale

On 04/07/2015 07:28 AM, Lawrence Kellogg wrote:
Answers below….

On Apr 6, 2015, at 9:59 PM, Dale Henrichs <[hidden email]> wrote:



On 4/6/15 5:30 PM, Lawrence Kellogg wrote:

On Apr 6, 2015, at 8:25 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

It looks like I assumed that you were using Seaside 3.1 and you must be using Seaside3.0 ... I don't think that things have actually changed in this area since Seaside 3.0, but it _is_ something to be aware of ... now...

Given that the code (in Seaside-GemStone-Core-dkh.70) in WARcLastAccessEntry>>isExpired:updating: (could you verify that your version is the same?):

isExpired: timeout updating: updating
  | value |
  ^ (value := self counter value) > lastCount
    ifTrue: [
      "accessed since last scan"
      updating
        ifTrue: [
          lastCount := value.
          time := Time totalSeconds ].
      false ]
    ifFalse: [ Time totalSeconds - time > timeout ]

it appears that we have only done a expiration scan (updating = true) only once and I have been under the impression that we've done at least two full scans (the scan before the backup and the scan after we discovered that we had not recovered any data) ... oh well ...

I think that the `time` in this entry corresponds to the date: `2014-11-11T22:42:08+00:00` (Time totalSeconds returns seconds since January 1, 1901) and that implies that the last scan was done a year ago? Unless the code is very different or ...  What does `date` return from the linux shell?

The next step is to see if the code is working like I expect it to. The code below simulates and expiration scan, and then see how the entry has changed along with couple of other things:

  | app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}

I've got to head home, but I'll check in again this evening when I get the chance ...


Thanks, the above code returns:

anArray( aWARcLastAccessExpiryPolicy, 3605819381, false, aWARcLastAccessEntry, 3605819381, 1, 1, true)

The time, corresponds to 2015-04-07T00:29:41+00:00...so the date on your server is probably correct...

Oh shoot, the first slot should have been `cache expiryPolicy timeout`, so let's try again, with an edited workspace:

| app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  assoc := objectsByKey associationAt: '5jTYOuLPzkOYLv4d'.
  lastAccessTable := cache expiryPolicy instVarNamed: #'lastAccessTable'.
  entry := lastAccessTable at: '5jTYOuLPzkOYLv4d'.
  timeout := cache expiryPolicy timeout.
  expired1 := entry isExpiredUpdating: timeout.
  expired2 := entry isExpiredUpdating: -1.
  ^ {timeout.
  Time totalSeconds.
  expired1.
  entry.
  entry time.
  entry lastCount.
  (entry counter value).
  expired2}


Ok, here is the result from the above code, substituting at: for associationAt:

anArray( 600, 3605869347, true, aWARcLastAccessEntry, 3605819381, 1, 1, true)




From this little experiment it seems that we've only run one expiration run against this particular session ... so after you get me the results from this array and the timeout (first slot) seems to be a reasonable number (less than 12,620,853), then we will try running the expiration run again... BUT before we run anther expiration run, I want to see the code that you are using in


WACache>>gemstoneReap,


gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM, 
     so we are already in transaction (assumed to be running in #autoBegin 
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc | 
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [ 
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
Transcript cr; show: 'Scan progress: ' , count printString.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
platform doCommitTransaction.
count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc | 
      count := count + 1.
Transcript cr; show: 'Expire progress: ' , count printString.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
Transcript show: 'Leaving gemstoneReap: ',expired size printString.
  ^ expired size

WABasicDevelopment class>>reapHandlerCache:dispatchers: and

reapHandlerCache: aHandler dispatchers: dispatchers
| expired |
expired := 0.
(aHandler isApplication or: [ aHandler isSession ])
ifTrue: [
expired :=  aHandler cache gemstoneReap.
aHandler sessionsDo: [:session | expired := expired + session continuations gemstoneReap ]]
ifFalse: [
aHandler isDispatcher 
ifTrue: [ dispatchers add: aHandler ]].
^expired


WABasicDevelopment class>>reapSeasideCache, just to see if I can spot a transaction error before running a second (and third) expiration.

reapSeasideCache

| dps expired |
expired := 0.
dps := OrderedCollection with: WADispatcher default.
[ dps isEmpty ] whileFalse: [ | dp |
dp := dps removeFirst.
dp isApplication
ifTrue: [
expired := expired + (self reapHandlerCache: dp dispatchers: dps) ]
ifFalse: [ 
dp handlers valuesDo: [:ep |
expired := expired + (self reapHandlerCache: ep dispatchers: dps) ]]].
^expired



Yes twice more ... for some unknown reason, the session we are looking has apparently not been scanned before (which is why I want to read the code that is in your image) the `lastCount` should have been equal to the `counter value` and I'm not quite sure why the lastCount hasn't been bumped ... unless the first time we touched the session is when we just ran the above workspace ...

I will have to drop offline for awhile, but I will try bringing up a stone that has an older version of Seaside installed to see if there are indeed differences in the expiration logic ...

I assume that you've been running these tests in the stone that we restored from backup ..... or have you been running these tests in a stone in which you've never run the expiration code?


Yes, all of the test have been run from the stone that I restored from backup, and the reap code has been run multiple times, as I recall. 

Best, 

Larry



Dale



_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [Glass] [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

Here's the workspace to use if the size of the cache seems to be reasonable:

  | app cache objectsByKey |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  {(objectsByKey size).
  (cache gemstoneReap)}

If 'UserInformationInterface' appears to be too big, poke around in the WAApplication instances that show up in `WADispatcher default handlers` and see if you can find the smallest cache, and then run the above workspace against that app ... also I'd like to see the result array which gives us the number of entries in the cache and the number of entries expired ...

If they turn out to be significantly different, then try a second run  ...

I'm also tempted to ask you to edit the gemstoneReap method to reduce the amount of logging (but this would be optional):

WACache>>gemstoneReap,

gemstoneReap
  "Iterate through the cache and remove objects that have expired."

  "In GemStone, this method is performed by a separate maintenance VM, 
     so we are already in transaction (assumed to be running in #autoBegin 
     transactionMode) and do not have to worry about acquiring the TransactionMutex.
    Since we are using reducedConflict dictionaries in the first place, we will remove the keys
    and values from the existing dictionaries without using the mutex."

  | expired count platform |
  expired := UserGlobals at: #'ExpiryCleanup' put: OrderedCollection new.
  platform := GRPlatform current.
  platform doCommitTransaction.
  count := 0.
  objectsByKey
    associationsDo: [ :assoc | 
      (self expiryPolicy isExpiredUpdating: assoc value key: assoc key)
        ifTrue: [ 
          self notifyRemoved: assoc value key: assoc key.
          count := count + 1.
          expired add: assoc.
          count \\ 100 == 0
            ifTrue: [platform doCommitTransaction ].
          count \\ 1000 == 0
            ifTrue: [Transcript cr; show: 'Scan progress: ' , count printString.] ] ].
  Transcript cr; show: 'finished scan: ' , count printString.
platform doCommitTransaction.
count := 0.
  (UserGlobals at: #'ExpiryCleanup' )
    do: [ :assoc | 
      count := count + 1.
      objectsByKey removeKey: assoc key ifAbsent: [].
      keysByObject removeKey: assoc value ifAbsent: [  ].
      count \\ 100 == 0
        ifTrue: [ platform doCommitTransaction ].
      count \\ 1000 == 0
        ifTrue: [ Transcript cr; show: 'Expire progress: ' , count printString.]].
  platform doCommitTransaction.
  UserGlobals removeKey: #'ExpiryCleanup'.
  platform doCommitTransaction.
Transcript show: 'Leaving gemstoneReap: ',expired size printString.
  ^ expired size

If you do make these edits then I'd like you to capture the transcript after each run completes ...

Dale

On 04/07/2015 03:20 PM, Dale Henrichs wrote:
Larry,

Okay, I cannot think of a reason that the earlier scans could not work, so I'm thinking that I'd like to pick a smaller subset of the whole system that runs faster and then try to understand why the scan isn't working ...

So I'll start by finding out if UserInformationInterface will make a likely candidate, so let's find out how big the cache is:

  | app cache objectsByKey assoc expired1 expired2 timeout lastAccessTable entry |
  app := WADispatcher default handlers at: 'UserInformationInterface'.
  cache := app cache.
  objectsByKey := cache instVarAt: 3.
  objectsByKey size

I'll put together another workspace for doing a scan on a per application basis in th emean time ...

Dale


_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [GLASS] Seaside - growing extent - normal?

GLASS mailing list
In reply to this post by GLASS mailing list
Hello,

  So, perhaps I will be forced to finally get a handle on my problem with a growing extent. 

  Something funny happened a couple of weeks ago that I thought I would share. 

  I needed a little backend service to do some testing with an iPhone app so I wrote it in Gemstone. 

  I wrote a simple queue mechanism that uses a few RcQueues that are accessible from a call to a WAComponent. That’s it. 

  Well, my iPhone app polls the back-end very fifteen seconds or so, calling this WAComponent. 

  I found that my extent ballooned from 36 gig to over 60 gig and filled up the whole file system! 

  Can we deduce anything as a result of my experience? Is the Seaside session stuff taking up all the space in the extent? 

  I’ll dig up those steps to try to shrink the extent but I don’t remember having much luck. 

  Best, 

  Larry

 
  
On Mar 30, 2015, at 3:23 PM, Lawrence Kellogg <[hidden email]> wrote:


On Mar 30, 2015, at 2:06 PM, Dale Henrichs <[hidden email]> wrote:

A couple of things:

  1. You appear to be doing rpc logins when using topaz? You
      should be able to login in using a linking login by using the -l
      option for topaz ... you will need to do this on the machine that
      is running .. linked logins bypass the netldi





  2. I suspect that you have started the netldi, without using the -g
      option ... so double check how you started the netldi ...


Yesh, I don’t know what the -g option does so I 


Dale
On 03/30/2015 10:06 AM, Lawrence Kellogg wrote:
Thanks Dale, I’ll try again, but, unfortunately, I can no longer log into GemTools. I get this error: 

'Unable to create a GemStone session.
Netldi ''gs64ldi'' on host ''ip-10-32-103-83'' reports the request ''gemnetobject'' failed:
Password validation failed for user seasideuser because getspnam() returned an error: errno=13,EACCES, Authorization failure (permission denied)’

Did the restore somehow change the password on the glass login in the launcher? 

I can’t log into Topaz either and I did in order to do the restore:

|_____________________________________________________________________________|
topaz> set username DataCurator 
topaz> set gemstone seaside
topaz> set password swordfish
topaz> login
-----------------------------------------------------
GemStone: Error         Fatal
Unable to create a GemStone session.
Netldi 'gs64ldi' on host 'ip-10-32-103-83' reports the request 'gemnetobject'
failed:
Password validation failed for user seasideuser because getspnam()
returned an error: errno=13,EACCES, Authorization failure (permission
denied)
Error Category: [GemStone] Number: 4042 Arg Count: 0


Larry

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results

 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...

By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass









_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
Reply | Threaded
Open this post in threaded view
|

Re: [GLASS] Seaside - growing extent - normal?

GLASS mailing list
Larry,

Did you have the maintenance vm running while you did this experiment? The maintenace vm should be expiring sessions every minute ...

If you are running the maintenance vm, then sharing your code with me would allow me to run local experiments and I should be able to figure out what's going on ...

Dale

On 08/07/2015 01:46 PM, Lawrence Kellogg wrote:
Hello,

  So, perhaps I will be forced to finally get a handle on my problem with a growing extent. 

  Something funny happened a couple of weeks ago that I thought I would share. 

  I needed a little backend service to do some testing with an iPhone app so I wrote it in Gemstone. 

  I wrote a simple queue mechanism that uses a few RcQueues that are accessible from a call to a WAComponent. That’s it. 

  Well, my iPhone app polls the back-end very fifteen seconds or so, calling this WAComponent. 

  I found that my extent ballooned from 36 gig to over 60 gig and filled up the whole file system! 

  Can we deduce anything as a result of my experience? Is the Seaside session stuff taking up all the space in the extent? 

  I’ll dig up those steps to try to shrink the extent but I don’t remember having much luck. 

  Best, 

  Larry

 
  
On Mar 30, 2015, at 3:23 PM, Lawrence Kellogg <[hidden email]> wrote:


On Mar 30, 2015, at 2:06 PM, Dale Henrichs <[hidden email]> wrote:

A couple of things:

  1. You appear to be doing rpc logins when using topaz? You
      should be able to login in using a linking login by using the -l
      option for topaz ... you will need to do this on the machine that
      is running .. linked logins bypass the netldi





  2. I suspect that you have started the netldi, without using the -g
      option ... so double check how you started the netldi ...


Yesh, I don’t know what the -g option does so I 


Dale
On 03/30/2015 10:06 AM, Lawrence Kellogg wrote:
Thanks Dale, I’ll try again, but, unfortunately, I can no longer log into GemTools. I get this error: 

'Unable to create a GemStone session.
Netldi ''gs64ldi'' on host ''ip-10-32-103-83'' reports the request ''gemnetobject'' failed:
Password validation failed for user seasideuser because getspnam() returned an error: errno=13,EACCES, Authorization failure (permission denied)’

Did the restore somehow change the password on the glass login in the launcher? 

I can’t log into Topaz either and I did in order to do the restore:

|_____________________________________________________________________________|
topaz> set username DataCurator 
topaz> set gemstone seaside
topaz> set password swordfish
topaz> login
-----------------------------------------------------
GemStone: Error         Fatal
Unable to create a GemStone session.
Netldi 'gs64ldi' on host 'ip-10-32-103-83' reports the request 'gemnetobject'
failed:
Password validation failed for user seasideuser because getspnam()
returned an error: errno=13,EACCES, Authorization failure (permission
denied)
Error Category: [GemStone] Number: 4042 Arg Count: 0


Larry

On Mar 30, 2015, at 12:28 PM, Dale Henrichs <[hidden email]> wrote:

Okay,

I guess you made it through the session expirations okay and according to the MFC results it does look like you did get rid of a big chunk of objects... Presumably the backup was made before the vote on the possible dead was finished so the backup would not have been able to skip all of the dead objects (until the vote was completed) .... there 's also an outside chance that the vm used to expire the sessions would have voted down some of the possible dead if it was still logged in when the backup was made ...

So we need to find out what's going on in the new extent ... so do another mfc and send me the results

 In the new extent, run the MFC again, and provide me with the results ... include an `Admin>>DoIt>>File Size Report`. Then logout of GemTools and stop/start any other seaside servers or maintenance vms that might be running ...

By the time we exchange emails, the vote should have a chance to complete this time... but I want to see the results of the MFC and File SIze Report before deciding what to do next ...

Dale

On 03/30/2015 07:30 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Well, I went though the process as described below, but have not see my extent shrink appreciably, so I am puzzled. 
Here is the screenshot after the mark for collection. Do I have to do something to reclaim the dead objects? Does the maintenance gem need to be run?


<Mail Attachment.png>

After the ObjectLog init, and mark, I did a restore into a fresh extent.

Here is the size of the new extent vs the old, saved extent:

<Mail Attachment.png>



Thoughts?

Larry



On Mar 25, 2015, at 2:15 PM, Dale Henrichs <[hidden email]> wrote:

Okay here's the sequence of steps that I think you should take:

  1. expire all of your sessions:

  | expired |
  Transcript cr; show: 'Unregistering...' , DateAndTime now printString.
  expired := WABasicDevelopment reapSeasideCache.
  expired > 0
    ifTrue: [ (ObjectLogEntry trace: 'MTCE: expired sessions' object: expired) addToLog ].
  Transcript cr; show: '...Expired: ' , expired printString , ' sessions.'.
  System commitTransactions

  2. initalize your object log

  3. run MFC

  [
  System abortTransaction.
  SystemRepository markForCollection ]
    on: Warning
    do: [ :ex |
      Transcript
        cr;
        show: ex description.
      ex resume ]

  4. Then do a backup and restore ... you can use GemTools to do the restore,
      but then you should read the SysAdmin docs[1] for instructions to do the restore
      (I've enclosed link to 3.2 docs, but the procedure and commands should pretty
      much be the same, but it's best to look up the docs for your GemStone version[2]
      and follow those instructions)

As I mentioned earlier, it will probably take a while for each of these operations to complete (object log will be fast and the backup will be fast, if the mfc tosses out the majority of your data) and it is likely that the repository will grow some more during the process (hard to predict this one, tho).

Step 1 will touch every session and every continuation so it is hard to say what percent of the objects are going to be touched (the expensive part), still there are likely to be a lot of those puppies and they will have to be read from disk into the SPC ...

Step 3. is going scan all of the live objects and again it hard to predict exactly how expensive it will be ...

Dale

[1] http://downloads.gemtalksystems.com/docs/GemStone64/3.2.x/GS64-SysAdmin-3.2/GS64-SysAdmin-3.2.htm
[2] http://gemtalksystems.com/techsupport/resources/

On 3/25/15 10:18 AM, Lawrence Kellogg wrote:
Hello Dale, 

  Thanks for the help. I’m a terrible system admin when it comes to maintaining a system with one user, LOL. 

  I’m not running the maintenance VM and I haven’t been doing regular mark for collects. 

  I’m trying to do a fullBackupTo: at the moment, well see if I get through that. Should I have done a markForCollection before the full backup? 

  I’ll also try the ObjectLog trick. 

  I guess I need to start from a fresh extent, as you said, and the extent file will not shrink. I’m at 48% of my available disk space but it does seem slower than usual. 

  
Best, 

Larry


  
On Mar 25, 2015, at 12:58 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Lawrence,

Are you doing regular Mark for collects? Are you running the maintenance vm along with you seaside servers?

Seaside produces persistent garbage (persistent session state that eventually times out) when it processes requests so if you do not run the maintenance vm the sessions are not expired and if you do not run mfc regularly the expired sessions are not cleaned up ...

Another source of growth could be the Object Log ... (use `ObjectLogEntry initalize` to efficiently reset the Object Log ... pay attention to the mispelling ... thats another story). If you are getting continuations saved to the object log, the stacks that are saved, can hang onto a lot of session state, that even though expired will not be garbage collected because of references from the continuation in the object log keep it alive ...

The best way to shrink your extent (once we understand why it is growing) is to do a backup and then restore into a virgin extent ($GEMSTONE/bin/extent0.seaside.dbf)...

Dale

On 3/25/15 8:34 AM, Lawrence Kellogg via Glass wrote:
Well, Amazon sent me a note that they are having hardware trouble on my instance, so they shut it down. It looks like they’re threatening to take the thing offline permanently so I’m trying to save my work with an AMI and move it somewhere else, if I have to.

I finally got Gemstone/Seaside back up and running and noticed these lines in the Seaside log file. These kind of messages go on once a day for weeks. Is this normal? 

--- 03/07/2015 02:44:14 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22528 megabytes.
    Repository has grown to 22528 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22544 megabytes.
    Repository has grown to 22544 megabytes.

--- 03/08/2015 03:31:45 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22560 megabytes.
    Repository has grown to 22560 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22576 megabytes.
    Repository has grown to 22576 megabytes.

--- 03/10/2015 03:19:34 AM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22592 megabytes.
    Repository has grown to 22592 megabytes.

--- 03/10/2015 03:46:39 PM UTC ---
    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22608 megabytes.
    Repository has grown to 22608 megabytes.

    Extent = !#dbf!/opt/gemstone/GemStone64Bit2.4.4.1-x86_64.Linux/seaside/data/extent0.dbf
       has grown to 22624 megabytes.
    Repository has grown to 22624 megabytes.


My extent has now grown to 

-rw------- 1 seasideuser seasideuser 23735566336 Mar 25 15:31 extent0.dbf


I don’t get a lot of traffic so I’m a little surprised at the growth. Should I try to shrink the extent?

I suppose I should also do a SystemRepository backup, if I can remember the commands. 

Best, 

Larry




_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass










_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass
123