Corrupted object preventing garbage collection?

Hello!
I'm having some weird problems :S
It seems that there are a couple of objects "corrupted" that are killing the seaside gems and preventing garbage collection.
I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.

Here is a piece of log of one of the seaside gems:

 daemon Server started on port 9003
object already in in-memory oop map , objId 10390231041
-----------------------------------------------------
GemStone: Error         Nonfatal
The object with object ID 10390231041 is corrupt. Reason: 'object
already in in-memory oop map'
Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context : 10443254785
Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
Arg 2: [10390230785 sz:35 cls: 74753 String] object already in in-memory oop map

Now executing the following command saved from "iferr 1":
   where
==> 1 System class >> _primitiveCommit: @1 line 1   [GsMethod 3226881]
2 System class >> __commit: @1 line 8   [GsMethod 3222017]
3 System class >> _localCommit: @8 line 30   [GsMethod 3222529]
4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3   [GsMethod 7713281]
5 System class >> _commit: @6 line 16   [GsMethod 3222785]
6 System class >> commitTransaction @2 line 28   [GsMethod 3230465]
7 System class >> _commitPrintingDiagnostics @2 line 9   [GsMethod 3151105]
8 SystemCommitTransaction >> defaultAction @1 line 3   [GsMethod 126577409]
9 ExceptionA >> _defaultAction @1 line 4   [GsMethod 10050561]
10 ExceptionA >> signal @7 line 37   [GsMethod 10057473]
11 ExceptionA >> signal: @3 line 12   [GsMethod 10050817]
12 ExceptionA class >> signal: @2 line 3   [GsMethod 11895041]
13 ExceptionA class >> signal @2 line 9   [GsMethod 11896577]
14 GRGemStonePlatform >> doCommitTransaction @3 line 3   [GsMethod 129758209]
15 ComplexVCBlock in GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: @28 line 32   [GsMethod 374752257]
16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
18 TransientRecursionLock >> critical: @15 line 8   [GsMethod 21159937]
19 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: @39 line 5   [GsMethod 374752257]
20 ComplexBlock in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: @7 line 9   [GsMethod 374752513]
21 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
22 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: @32 line 17   [GsMethod 374752513]
25 WAFastCGIAdaptor >> process: @4 line 4   [GsMethod 156093185]
26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line 4   [GsMethod 156095489]
27 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
28 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5   [GsMethod 156095489]
31 FSResponderRole >> answer @2 line 4   [GsMethod 155147521]
32 FSRole >> handleConnection @3 line 5   [GsMethod 155154177]
33 FSConnection >> unsafeServe @4 line 8   [GsMethod 156518913]
34 ComplexBlock in FSConnection >> safeServe @5 line 8   [GsMethod 156521729]
35 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
36 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
38 ComplexVCBlock in FSConnection >> safeServe @8 line 9   [GsMethod 156521729]
39 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
40 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
42 ComplexVCBlock in FSConnection >> safeServe @11 line 12   [GsMethod 156521729]
43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
45 FSConnection >> safeServe @14 line 15   [GsMethod 156521729]
46 FSConnection >> serve @1 line 4   [GsMethod 156525569]
47 ComplexBlock in FSSocketServer >> listen: @9 line 15   [GsMethod 155629057]
48 GsProcess >> _startPart2 @15 line 17   [GsMethod 4501249]
49 GsProcess >> _start @1 line 9   [GsMethod 4501761]

What could it be causing this? My extent keeps growing and growing :(

Cheers,
Carla

Another tip: Gemtools doesn't open the Debug list (well... it actually takes so much time to open it that I give up and interrupt the process).

And yet another tip: this also appears in the logs. anUIOperacion is and instance of one of my objects and I guess it's within the states of the seaside component that triggered this.

 daemon Server started on port 9001
-----------------------------------------------------
GemStone: Error         Nonfatal
No method was found for the selector #'sequenceNumber' when sent to
aWAValueHolder contents: anUIOperacion->aRcCounterElement with arguments
contained in anArray( ).
Error Category: 231169 [GemStone] Number: 2010 Arg Count: 3 Context : 10394577665
Arg 1: [10419891969 sz:2 cls: 67073 Association] an Association
  key             [8217729281 sz:1 cls: 123185665 WAValueHolder] a WAValueHolder
  value           [10391790081 sz:1 cls: 108289 RcCounterElement] a RcCounterElement

Arg 2: [6798849 sz:14 cls: 110849 Symbol] sequenceNumber
Arg 3: [10394577409 sz:0 cls: 66817 Array] an Array

Now executing the following command saved from "iferr 1":
   where

==> 1 System class >> signal:args:signalDictionary: @8 line 15   [GsMethod 3241473]
2 Object >> doesNotUnderstand: @6 line 14   [GsMethod 1926913]
3 Object >> _doesNotUnderstand: @1 line 6   [GsMethod 1926657]
4 RcQueue >> add: @11 line 13   [GsMethod 12454913]
5 ObjectLogEntry >> addToLog @3 line 3   [GsMethod 20842497]
6 ComplexBlock in GRGemStonePlatform >> seasideLogServerStart:port: @12 line 8   [GsMethod 129763585]
7 ComplexBlock in GRGemStonePlatform >> doTransaction: @9 line 15   [GsMethod 129762305]

8 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]

9 ComplexBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
10 ComplexVCBlock in GRGemStonePlatform >> doTransaction: @13 line 16   [GsMethod 129762305]
11 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
12 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
13 TransientRecursionLock >> critical: @15 line 8   [GsMethod 21159937]
14 GRGemStonePlatform >> doTransaction: @14 line 11   [GsMethod 129762305]
15 GRGemStonePlatform >> seasideLogServerStart:port: @13 line 3   [GsMethod 129763585]
16 WAGemStoneRunSeasideGems class >> startGemServerOn: @4 line 7   [GsMethod 144519425]
17 Executed Code @24 line 29   [GsMethod 10394554113]

On Mon, Aug 13, 2012 at 4:47 PM, Carla F. Griggio <[hidden email]> wrote:

Hello!
I'm having some weird problems :S
It seems that there are a couple of objects "corrupted" that are killing the seaside gems and preventing garbage collection.
I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.

Here is a piece of log of one of the seaside gems:

 daemon Server started on port 9003
object already in in-memory oop map , objId 10390231041
-----------------------------------------------------
GemStone: Error         Nonfatal
The object with object ID 10390231041 is corrupt. Reason: 'object
already in in-memory oop map'
Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context : 10443254785
Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
Arg 2: [10390230785 sz:35 cls: 74753 String] object already in in-memory oop map

Now executing the following command saved from "iferr 1":
   where
==> 1 System class >> _primitiveCommit: @1 line 1   [GsMethod 3226881]
2 System class >> __commit: @1 line 8   [GsMethod 3222017]
3 System class >> _localCommit: @8 line 30   [GsMethod 3222529]
4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3   [GsMethod 7713281]
5 System class >> _commit: @6 line 16   [GsMethod 3222785]
6 System class >> commitTransaction @2 line 28   [GsMethod 3230465]
7 System class >> _commitPrintingDiagnostics @2 line 9   [GsMethod 3151105]
8 SystemCommitTransaction >> defaultAction @1 line 3   [GsMethod 126577409]
9 ExceptionA >> _defaultAction @1 line 4   [GsMethod 10050561]
10 ExceptionA >> signal @7 line 37   [GsMethod 10057473]
11 ExceptionA >> signal: @3 line 12   [GsMethod 10050817]
12 ExceptionA class >> signal: @2 line 3   [GsMethod 11895041]
13 ExceptionA class >> signal @2 line 9   [GsMethod 11896577]
14 GRGemStonePlatform >> doCommitTransaction @3 line 3   [GsMethod 129758209]
15 ComplexVCBlock in GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: @28 line 32   [GsMethod 374752257]
16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
18 TransientRecursionLock >> critical: @15 line 8   [GsMethod 21159937]
19 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: @39 line 5   [GsMethod 374752257]
20 ComplexBlock in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: @7 line 9   [GsMethod 374752513]
21 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
22 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: @32 line 17   [GsMethod 374752513]
25 WAFastCGIAdaptor >> process: @4 line 4   [GsMethod 156093185]
26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line 4   [GsMethod 156095489]
27 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
28 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5   [GsMethod 156095489]
31 FSResponderRole >> answer @2 line 4   [GsMethod 155147521]
32 FSRole >> handleConnection @3 line 5   [GsMethod 155154177]
33 FSConnection >> unsafeServe @4 line 8   [GsMethod 156518913]
34 ComplexBlock in FSConnection >> safeServe @5 line 8   [GsMethod 156521729]
35 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
36 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
38 ComplexVCBlock in FSConnection >> safeServe @8 line 9   [GsMethod 156521729]
39 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
40 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
42 ComplexVCBlock in FSConnection >> safeServe @11 line 12   [GsMethod 156521729]
43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
45 FSConnection >> safeServe @14 line 15   [GsMethod 156521729]
46 FSConnection >> serve @1 line 4   [GsMethod 156525569]
47 ComplexBlock in FSSocketServer >> listen: @9 line 15   [GsMethod 155629057]
48 GsProcess >> _startPart2 @15 line 17   [GsMethod 4501249]
49 GsProcess >> _start @1 line 9   [GsMethod 4501761]

What could it be causing this? My extent keeps growing and growing :(

Cheers,
Carla

And anoooother thing.
I tried to empty the ObjectLogEntry using topaz and this is what happened:

topaz 1> printit
ObjectLogEntry emptyLog

%
-----------------------------------------------------
GemStone: Error         Nonfatal

The object with object ID 10399549697 does not exist.
Error Category: [GemStone] Number: 2101 Arg Count: 1
Arg 1: 10399549697


(There are 2 objects IDs bothering me like this, the one from the log I copied before and this one, both appear in the same kind of errors).

On Mon, Aug 13, 2012 at 4:55 PM, Carla F. Griggio <[hidden email]> wrote:

Another tip: Gemtools doesn't open the Debug list (well... it actually takes so much time to open it that I give up and interrupt the process).

And yet another tip: this also appears in the logs. anUIOperacion is and instance of one of my objects and I guess it's within the states of the seaside component that triggered this.

 daemon Server started on port 9001
-----------------------------------------------------
GemStone: Error         Nonfatal
No method was found for the selector #'sequenceNumber' when sent to
aWAValueHolder contents: anUIOperacion->aRcCounterElement with arguments
contained in anArray( ).
Error Category: 231169 [GemStone] Number: 2010 Arg Count: 3 Context : 10394577665
Arg 1: [10419891969 sz:2 cls: 67073 Association] an Association
  key             [8217729281 sz:1 cls: 123185665 WAValueHolder] a WAValueHolder
  value           [10391790081 sz:1 cls: 108289 RcCounterElement] a RcCounterElement

Arg 2: [6798849 sz:14 cls: 110849 Symbol] sequenceNumber
Arg 3: [10394577409 sz:0 cls: 66817 Array] an Array

Now executing the following command saved from "iferr 1":
   where

==> 1 System class >> signal:args:signalDictionary: @8 line 15   [GsMethod 3241473]
2 Object >> doesNotUnderstand: @6 line 14   [GsMethod 1926913]
3 Object >> _doesNotUnderstand: @1 line 6   [GsMethod 1926657]
4 RcQueue >> add: @11 line 13   [GsMethod 12454913]
5 ObjectLogEntry >> addToLog @3 line 3   [GsMethod 20842497]
6 ComplexBlock in GRGemStonePlatform >> seasideLogServerStart:port: @12 line 8   [GsMethod 129763585]
7 ComplexBlock in GRGemStonePlatform >> doTransaction: @9 line 15   [GsMethod 129762305]

8 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]

9 ComplexBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
10 ComplexVCBlock in GRGemStonePlatform >> doTransaction: @13 line 16   [GsMethod 129762305]
11 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
12 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
13 TransientRecursionLock >> critical: @15 line 8   [GsMethod 21159937]
14 GRGemStonePlatform >> doTransaction: @14 line 11   [GsMethod 129762305]
15 GRGemStonePlatform >> seasideLogServerStart:port: @13 line 3   [GsMethod 129763585]
16 WAGemStoneRunSeasideGems class >> startGemServerOn: @4 line 7   [GsMethod 144519425]
17 Executed Code @24 line 29   [GsMethod 10394554113]

On Mon, Aug 13, 2012 at 4:47 PM, Carla F. Griggio <[hidden email]> wrote:

Hello!
I'm having some weird problems :S
It seems that there are a couple of objects "corrupted" that are killing the seaside gems and preventing garbage collection.
I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.

Here is a piece of log of one of the seaside gems:

 daemon Server started on port 9003
object already in in-memory oop map , objId 10390231041
-----------------------------------------------------
GemStone: Error         Nonfatal
The object with object ID 10390231041 is corrupt. Reason: 'object
already in in-memory oop map'
Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context : 10443254785
Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
Arg 2: [10390230785 sz:35 cls: 74753 String] object already in in-memory oop map

Now executing the following command saved from "iferr 1":
   where
==> 1 System class >> _primitiveCommit: @1 line 1   [GsMethod 3226881]
2 System class >> __commit: @1 line 8   [GsMethod 3222017]
3 System class >> _localCommit: @8 line 30   [GsMethod 3222529]
4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3   [GsMethod 7713281]
5 System class >> _commit: @6 line 16   [GsMethod 3222785]
6 System class >> commitTransaction @2 line 28   [GsMethod 3230465]
7 System class >> _commitPrintingDiagnostics @2 line 9   [GsMethod 3151105]
8 SystemCommitTransaction >> defaultAction @1 line 3   [GsMethod 126577409]
9 ExceptionA >> _defaultAction @1 line 4   [GsMethod 10050561]
10 ExceptionA >> signal @7 line 37   [GsMethod 10057473]
11 ExceptionA >> signal: @3 line 12   [GsMethod 10050817]
12 ExceptionA class >> signal: @2 line 3   [GsMethod 11895041]
13 ExceptionA class >> signal @2 line 9   [GsMethod 11896577]
14 GRGemStonePlatform >> doCommitTransaction @3 line 3   [GsMethod 129758209]
15 ComplexVCBlock in GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: @28 line 32   [GsMethod 374752257]
16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
18 TransientRecursionLock >> critical: @15 line 8   [GsMethod 21159937]
19 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock: @39 line 5   [GsMethod 374752257]
20 ComplexBlock in GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: @7 line 9   [GsMethod 374752513]
21 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
22 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock: @32 line 17   [GsMethod 374752513]
25 WAFastCGIAdaptor >> process: @4 line 4   [GsMethod 156093185]
26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line 4   [GsMethod 156095489]
27 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
28 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5   [GsMethod 156095489]
31 FSResponderRole >> answer @2 line 4   [GsMethod 155147521]
32 FSRole >> handleConnection @3 line 5   [GsMethod 155154177]
33 FSConnection >> unsafeServe @4 line 8   [GsMethod 156518913]
34 ComplexBlock in FSConnection >> safeServe @5 line 8   [GsMethod 156521729]
35 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
36 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
38 ComplexVCBlock in FSConnection >> safeServe @8 line 9   [GsMethod 156521729]
39 ExceptionHandler >> doTryBlock: @9 line 7   [GsMethod 10065409]
40 ExceptionHandler >> try:on:do: @15 line 18   [GsMethod 10062081]
41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8   [GsMethod 9005057]
42 ComplexVCBlock in FSConnection >> safeServe @11 line 12   [GsMethod 156521729]
43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11   [GsMethod 2304001]
44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11   [GsMethod 2304001]
45 FSConnection >> safeServe @14 line 15   [GsMethod 156521729]
46 FSConnection >> serve @1 line 4   [GsMethod 156525569]
47 ComplexBlock in FSSocketServer >> listen: @9 line 15   [GsMethod 155629057]
48 GsProcess >> _startPart2 @15 line 17   [GsMethod 4501249]
49 GsProcess >> _start @1 line 9   [GsMethod 4501761]

What could it be causing this? My extent keeps growing and growing :(

Cheers,
Carla

Carla,

I'm on vacation, so I just saw this thread ... the first thing that you should do is to run an object audit (see section 8.2 of the System Administration Guide[1]). This will tell us whether the corruption is persistent or is only showing up in temporary objects...

It's a good sign that you're now having clean mfcs but the object audit will find any data base corruption that might be lurking ...

The 'object already in in-memory oop map' rings a bell for me and I'm finding out more information about that particular error...

Dale

----- Original Message -----
| From: "Carla F. Griggio" <[hidden email]>

| To: "GemStone Seaside beta discussion" <[hidden email]>

| Sent: Monday, August 13, 2012 1:42:43 PM
| Subject: Re: [GS/SS Beta] Corrupted object preventing garbage collection?
|
| Well... I killed the mantainance gem and the garbage collection
| started to work ok again. But I still don't know if the errors in
| the log will appear again, I'll keep you posted.
|
|
| On Mon, Aug 13, 2012 at 5:13 PM, Carla F. Griggio <
| [hidden email] > wrote:
|
|
| And anoooother thing.
| I tried to empty the ObjectLogEntry using topaz and this is what
| happened:
|
| topaz 1> printit
| ObjectLogEntry emptyLog
|
| %
| -----------------------------------------------------
| GemStone: Error Nonfatal
| The object with object ID 10399549697 does not exist.
| Error Category: [GemStone] Number: 2101 Arg Count: 1
| Arg 1: 10399549697
|
|
| (There are 2 objects IDs bothering me like this, the one from the log
| I copied before and this one, both appear in the same kind of
| errors).
|
|
|
|
| On Mon, Aug 13, 2012 at 4:55 PM, Carla F. Griggio <
| [hidden email] > wrote:
|
|
| Another tip: Gemtools doesn't open the Debug list (well... it
| actually takes so much time to open it that I give up and interrupt
| the process).
|
| And yet another tip: this also appears in the logs. anUIOperacion is
| and instance of one of my objects and I guess it's within the states
| of the seaside component that triggered this.
|
| daemon Server started on port 9001
| -----------------------------------------------------
| GemStone: Error Nonfatal
| No method was found for the selector #'sequenceNumber' when sent to
| aWAValueHolder contents: anUIOperacion->aRcCounterElement with
| arguments
| contained in anArray( ).
| Error Category: 231169 [GemStone] Number: 2010 Arg Count: 3 Context :
| 10394577665
| Arg 1: [10419891969 sz:2 cls: 67073 Association] an Association
| key [8217729281 sz:1 cls: 123185665 WAValueHolder] a WAValueHolder
| value [10391790081 sz:1 cls: 108289 RcCounterElement] a
| RcCounterElement
|
| Arg 2: [6798849 sz:14 cls: 110849 Symbol] sequenceNumber
| Arg 3: [10394577409 sz:0 cls: 66817 Array] an Array
|
|
| Now executing the following command saved from "iferr 1":
| where
| ==> 1 System class >> signal:args:signalDictionary: @8 line 15
| [GsMethod 3241473]
| 2 Object >> doesNotUnderstand: @6 line 14 [GsMethod 1926913]
| 3 Object >> _doesNotUnderstand: @1 line 6 [GsMethod 1926657]
| 4 RcQueue >> add: @11 line 13 [GsMethod 12454913]
| 5 ObjectLogEntry >> addToLog @3 line 3 [GsMethod 20842497]
| 6 ComplexBlock in GRGemStonePlatform >> seasideLogServerStart:port:
| @12 line 8 [GsMethod 129763585]
| 7 ComplexBlock in GRGemStonePlatform >> doTransaction: @9 line 15
| [GsMethod 129762305]
|
| 8 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| 2304001]
| 9 ComplexBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| 2304001]
| 10 ComplexVCBlock in GRGemStonePlatform >> doTransaction: @13 line 16
| [GsMethod 129762305]
| 11 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| 2304001]
| 12 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| 2304001]
| 13 TransientRecursionLock >> critical: @15 line 8 [GsMethod 21159937]
| 14 GRGemStonePlatform >> doTransaction: @14 line 11 [GsMethod
| 129762305]
| 15 GRGemStonePlatform >> seasideLogServerStart:port: @13 line 3
| [GsMethod 129763585]
| 16 WAGemStoneRunSeasideGems class >> startGemServerOn: @4 line 7
| [GsMethod 144519425]
| 17 Executed Code @24 line 29 [GsMethod 10394554113]
|
|
|
|
|
|
| On Mon, Aug 13, 2012 at 4:47 PM, Carla F. Griggio <
| [hidden email] > wrote:
|
|
| Hello!
| I'm having some weird problems :S
| It seems that there are a couple of objects "corrupted" that are
| killing the seaside gems and preventing garbage collection.
| I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.
|
| Here is a piece of log of one of the seaside gems:
|
| daemon Server started on port 9003
| object already in in-memory oop map , objId 10390231041
| -----------------------------------------------------
| GemStone: Error Nonfatal
| The object with object ID 10390231041 is corrupt. Reason: 'object
| already in in-memory oop map'
| Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context :
| 10443254785
| Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
| Arg 2: [10390230785 sz:35 cls: 74753 String] object already in
| in-memory oop map
|
| Now executing the following command saved from "iferr 1":
| where
| ==> 1 System class >> _primitiveCommit: @1 line 1 [GsMethod 3226881]
| 2 System class >> __commit: @1 line 8 [GsMethod 3222017]
| 3 System class >> _localCommit: @8 line 30 [GsMethod 3222529]
| 4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3 [GsMethod
| 7713281]
| 5 System class >> _commit: @6 line 16 [GsMethod 3222785]
| 6 System class >> commitTransaction @2 line 28 [GsMethod 3230465]
| 7 System class >> _commitPrintingDiagnostics @2 line 9 [GsMethod
| 3151105]
| 8 SystemCommitTransaction >> defaultAction @1 line 3 [GsMethod
| 126577409]
| 9 ExceptionA >> _defaultAction @1 line 4 [GsMethod 10050561]
| 10 ExceptionA >> signal @7 line 37 [GsMethod 10057473]
| 11 ExceptionA >> signal: @3 line 12 [GsMethod 10050817]
| 12 ExceptionA class >> signal: @2 line 3 [GsMethod 11895041]
| 13 ExceptionA class >> signal @2 line 9 [GsMethod 11896577]
| 14 GRGemStonePlatform >> doCommitTransaction @3 line 3 [GsMethod
| 129758209]
| 15 ComplexVCBlock in GRGemStonePlatform >>
| seasideProcessRequestWithRetry:resultBlock: @28 line 32 [GsMethod
| 374752257]
| 16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| 2304001]
| 17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| 2304001]
| 18 TransientRecursionLock >> critical: @15 line 8 [GsMethod 21159937]
| 19 GRGemStonePlatform >> seasideProcessRequestWithRetry:resultBlock:
| @39 line 5 [GsMethod 374752257]
| 20 ComplexBlock in GRGemStonePlatform >>
| seasideProcessRequest:adaptor:resultBlock: @7 line 9 [GsMethod
| 374752513]
| 21 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| 22 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| 23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| 9005057]
| 24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
| @32 line 17 [GsMethod 374752513]
| 25 WAFastCGIAdaptor >> process: @4 line 4 [GsMethod 156093185]
| 26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line 4
| [GsMethod 156095489]
| 27 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| 28 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| 29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| 9005057]
| 30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5 [GsMethod
| 156095489]
| 31 FSResponderRole >> answer @2 line 4 [GsMethod 155147521]
| 32 FSRole >> handleConnection @3 line 5 [GsMethod 155154177]
| 33 FSConnection >> unsafeServe @4 line 8 [GsMethod 156518913]
| 34 ComplexBlock in FSConnection >> safeServe @5 line 8 [GsMethod
| 156521729]
| 35 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| 36 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| 37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| 9005057]
| 38 ComplexVCBlock in FSConnection >> safeServe @8 line 9 [GsMethod
| 156521729]
| 39 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| 40 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| 41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| 9005057]
| 42 ComplexVCBlock in FSConnection >> safeServe @11 line 12 [GsMethod
| 156521729]
| 43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| 2304001]
| 44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| 2304001]
| 45 FSConnection >> safeServe @14 line 15 [GsMethod 156521729]
| 46 FSConnection >> serve @1 line 4 [GsMethod 156525569]
| 47 ComplexBlock in FSSocketServer >> listen: @9 line 15 [GsMethod
| 155629057]
| 48 GsProcess >> _startPart2 @15 line 17 [GsMethod 4501249]
| 49 GsProcess >> _start @1 line 9 [GsMethod 4501761]
|
| What could it be causing this? My extent keeps growing and growing :(
|
| Cheers,
| Carla
|
|
|
|
|
|
|
|
|

Carla,

Our engineers have looked into the "object already in in-memory oop map" and here's the report:

  There have been a number of reports of this error, some associated with
  conversion, some associated with the MT scan. I didn't see anything But
  the structure is not persistent, so something else is going on.

  I didn't see a C level stack trace... that's usually what we'd ask for.
  Have her set GEM_HALT_ON_ERROR=2261 to get a stack.

As mentioned above the error isn't related to persistent data structures... Still useful to do an object audit to ensure that you've got a clean slate now that you've upgraded to 2.4.4.4.

Dale
----- Original Message -----
| From: "Carla F. Griggio" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>

| Sent: Tuesday, August 14, 2012 9:05:45 AM
| Subject: Re: [GS/SS Beta] Corrupted object preventing garbage collection?
|
| Thank you, Dale! I'll check that.
| I'll also update to 2.4.4.4, the release notes say that one of these
| problems was resolved in that version.
|
| Enjoy your vacations!
|
| Carla.
|
|
| On Tue, Aug 14, 2012 at 12:59 PM, Dale Henrichs < [hidden email]
| > wrote:
|
|
| Carla,
|
| I'm on vacation, so I just saw this thread ... the first thing that
| you should do is to run an object audit (see section 8.2 of the
| System Administration Guide[1]). This will tell us whether the
| corruption is persistent or is only showing up in temporary
| objects...
|
| It's a good sign that you're now having clean mfcs but the object
| audit will find any data base corruption that might be lurking ...
|
| The 'object already in in-memory oop map' rings a bell for me and I'm
| finding out more information about that particular error...
|
| Dale
|
|
| ----- Original Message -----
| | From: "Carla F. Griggio" < [hidden email] >
|
| | To: "GemStone Seaside beta discussion" < [hidden email]
| | >
|
|
| | Sent: Monday, August 13, 2012 1:42:43 PM
| | Subject: Re: [GS/SS Beta] Corrupted object preventing garbage
| | collection?
| |
| | Well... I killed the mantainance gem and the garbage collection
| | started to work ok again. But I still don't know if the errors in
| | the log will appear again, I'll keep you posted.
| |
| |
| | On Mon, Aug 13, 2012 at 5:13 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | And anoooother thing.
| | I tried to empty the ObjectLogEntry using topaz and this is what
| | happened:
| |
| | topaz 1> printit
| | ObjectLogEntry emptyLog
| |
| | %
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10399549697 does not exist.
| | Error Category: [GemStone] Number: 2101 Arg Count: 1
| | Arg 1: 10399549697
| |
| |
| | (There are 2 objects IDs bothering me like this, the one from the
| | log
| | I copied before and this one, both appear in the same kind of
| | errors).
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:55 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Another tip: Gemtools doesn't open the Debug list (well... it
| | actually takes so much time to open it that I give up and interrupt
| | the process).
| |
| | And yet another tip: this also appears in the logs. anUIOperacion
| | is
| | and instance of one of my objects and I guess it's within the
| | states
| | of the seaside component that triggered this.
| |
| | daemon Server started on port 9001
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | No method was found for the selector #'sequenceNumber' when sent to
| | aWAValueHolder contents: anUIOperacion->aRcCounterElement with
| | arguments
| | contained in anArray( ).
| | Error Category: 231169 [GemStone] Number: 2010 Arg Count: 3 Context
| | :
| | 10394577665
| | Arg 1: [10419891969 sz:2 cls: 67073 Association] an Association
| | key [8217729281 sz:1 cls: 123185665 WAValueHolder] a WAValueHolder
| | value [10391790081 sz:1 cls: 108289 RcCounterElement] a
| | RcCounterElement
| |
| | Arg 2: [6798849 sz:14 cls: 110849 Symbol] sequenceNumber
| | Arg 3: [10394577409 sz:0 cls: 66817 Array] an Array
| |
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> signal:args:signalDictionary: @8 line 15
| | [GsMethod 3241473]
| | 2 Object >> doesNotUnderstand: @6 line 14 [GsMethod 1926913]
| | 3 Object >> _doesNotUnderstand: @1 line 6 [GsMethod 1926657]
| | 4 RcQueue >> add: @11 line 13 [GsMethod 12454913]
| | 5 ObjectLogEntry >> addToLog @3 line 3 [GsMethod 20842497]
| | 6 ComplexBlock in GRGemStonePlatform >> seasideLogServerStart:port:
| | @12 line 8 [GsMethod 129763585]
| | 7 ComplexBlock in GRGemStonePlatform >> doTransaction: @9 line 15
| | [GsMethod 129762305]
| |
| | 8 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 9 ComplexBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| | 2304001]
| | 10 ComplexVCBlock in GRGemStonePlatform >> doTransaction: @13 line
| | 16
| | [GsMethod 129762305]
| | 11 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 12 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 13 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 14 GRGemStonePlatform >> doTransaction: @14 line 11 [GsMethod
| | 129762305]
| | 15 GRGemStonePlatform >> seasideLogServerStart:port: @13 line 3
| | [GsMethod 129763585]
| | 16 WAGemStoneRunSeasideGems class >> startGemServerOn: @4 line 7
| | [GsMethod 144519425]
| | 17 Executed Code @24 line 29 [GsMethod 10394554113]
| |
| |
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:47 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Hello!
| | I'm having some weird problems :S
| | It seems that there are a couple of objects "corrupted" that are
| | killing the seaside gems and preventing garbage collection.
| | I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.
| |
| | Here is a piece of log of one of the seaside gems:
| |
| | daemon Server started on port 9003
| | object already in in-memory oop map , objId 10390231041
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10390231041 is corrupt. Reason: 'object
| | already in in-memory oop map'
| | Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context
| | :
| | 10443254785
| | Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
| | Arg 2: [10390230785 sz:35 cls: 74753 String] object already in
| | in-memory oop map
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> _primitiveCommit: @1 line 1 [GsMethod
| | 3226881]
| | 2 System class >> __commit: @1 line 8 [GsMethod 3222017]
| | 3 System class >> _localCommit: @8 line 30 [GsMethod 3222529]
| | 4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3 [GsMethod
| | 7713281]
| | 5 System class >> _commit: @6 line 16 [GsMethod 3222785]
| | 6 System class >> commitTransaction @2 line 28 [GsMethod 3230465]
| | 7 System class >> _commitPrintingDiagnostics @2 line 9 [GsMethod
| | 3151105]
| | 8 SystemCommitTransaction >> defaultAction @1 line 3 [GsMethod
| | 126577409]
| | 9 ExceptionA >> _defaultAction @1 line 4 [GsMethod 10050561]
| | 10 ExceptionA >> signal @7 line 37 [GsMethod 10057473]
| | 11 ExceptionA >> signal: @3 line 12 [GsMethod 10050817]
| | 12 ExceptionA class >> signal: @2 line 3 [GsMethod 11895041]
| | 13 ExceptionA class >> signal @2 line 9 [GsMethod 11896577]
| | 14 GRGemStonePlatform >> doCommitTransaction @3 line 3 [GsMethod
| | 129758209]
| | 15 ComplexVCBlock in GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock: @28 line 32 [GsMethod
| | 374752257]
| | 16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 18 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 19 GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock:
| | @39 line 5 [GsMethod 374752257]
| | 20 ComplexBlock in GRGemStonePlatform >>
| | seasideProcessRequest:adaptor:resultBlock: @7 line 9 [GsMethod
| | 374752513]
| | 21 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 22 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
| | @32 line 17 [GsMethod 374752513]
| | 25 WAFastCGIAdaptor >> process: @4 line 4 [GsMethod 156093185]
| | 26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line
| | 4
| | [GsMethod 156095489]
| | 27 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 28 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5 [GsMethod
| | 156095489]
| | 31 FSResponderRole >> answer @2 line 4 [GsMethod 155147521]
| | 32 FSRole >> handleConnection @3 line 5 [GsMethod 155154177]
| | 33 FSConnection >> unsafeServe @4 line 8 [GsMethod 156518913]
| | 34 ComplexBlock in FSConnection >> safeServe @5 line 8 [GsMethod
| | 156521729]
| | 35 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 36 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 38 ComplexVCBlock in FSConnection >> safeServe @8 line 9 [GsMethod
| | 156521729]
| | 39 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 40 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 42 ComplexVCBlock in FSConnection >> safeServe @11 line 12
| | [GsMethod
| | 156521729]
| | 43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 45 FSConnection >> safeServe @14 line 15 [GsMethod 156521729]
| | 46 FSConnection >> serve @1 line 4 [GsMethod 156525569]
| | 47 ComplexBlock in FSSocketServer >> listen: @9 line 15 [GsMethod
| | 155629057]
| | 48 GsProcess >> _startPart2 @15 line 17 [GsMethod 4501249]
| | 49 GsProcess >> _start @1 line 9 [GsMethod 4501761]
| |
| | What could it be causing this? My extent keeps growing and growing
| | :(
| |
| | Cheers,
| | Carla
| |
| |
| |
| |
| |
| |
| |
| |
| |
|
|

Hi Dale!

Well, I did an object audit and it showed that 923 objects had errors.

I inspected some of them, and they seem to be collections held by seaside components in the application cache.
Those collections point to non existent objects.

I think that the "Object does not exist" error is a consequence of this and not my main issue :) I mean, the 'guilty objects' are those in the repository that are pointing to non existent objects.

So I've just cleaned the application cache and I'm doing an audit now to see if at least I have less objects with errors.

Just for the record: as I suspected that the problems might come from old continuations, I tried to empty the Object Log, but again... the error that an object didn't exist would come up.
So I tried a silly workaround by reseting the instance variables objectLog and objectQueue of the ObjectLogEntry. I've just understood that my silly workaround was good for having a clean Object Log, BUT the old RcQueue with references to non existent objects kept living somewhere...
I found that old RcQueue, and when I tried to empty it the "Object does not exist" error came up. I could empty it by sending the message _initialize.
That released about 30MB from the repository.
I'm just posting this for future reference in case someone does the same mistake I did to force the Object Log to be empty :P

On Wed, Aug 15, 2012 at 3:33 PM, Dale Henrichs <[hidden email]> wrote:

Carla,

Our engineers have looked into the "object already in in-memory oop map" and here's the report:

  There have been a number of reports of this error, some associated with
  conversion, some associated with the MT scan. I didn't see anything But
  the structure is not persistent, so something else is going on.

  I didn't see a C level stack trace... that's usually what we'd ask for.
  Have her set GEM_HALT_ON_ERROR=2261 to get a stack.

As mentioned above the error isn't related to persistent data structures... Still useful to do an object audit to ensure that you've got a clean slate now that you've upgraded to 2.4.4.4.

Dale
----- Original Message -----
| From: "Carla F. Griggio" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>

| Sent: Tuesday, August 14, 2012 9:05:45 AM
| Subject: Re: [GS/SS Beta] Corrupted object preventing garbage collection?
|
| Thank you, Dale! I'll check that.
| I'll also update to 2.4.4.4, the release notes say that one of these
| problems was resolved in that version.
|
| Enjoy your vacations!
|
| Carla.
|
|
| On Tue, Aug 14, 2012 at 12:59 PM, Dale Henrichs < [hidden email]
| > wrote:
|
|
| Carla,
|
| I'm on vacation, so I just saw this thread ... the first thing that
| you should do is to run an object audit (see section 8.2 of the
| System Administration Guide[1]). This will tell us whether the
| corruption is persistent or is only showing up in temporary
| objects...
|
| It's a good sign that you're now having clean mfcs but the object
| audit will find any data base corruption that might be lurking ...
|
| The 'object already in in-memory oop map' rings a bell for me and I'm
| finding out more information about that particular error...
|
| Dale
|
|
| ----- Original Message -----
| | From: "Carla F. Griggio" < [hidden email] >
|
| | To: "GemStone Seaside beta discussion" < [hidden email]
| | >
|
|
| | Sent: Monday, August 13, 2012 1:42:43 PM
| | Subject: Re: [GS/SS Beta] Corrupted object preventing garbage
| | collection?
| |
| | Well... I killed the mantainance gem and the garbage collection
| | started to work ok again. But I still don't know if the errors in
| | the log will appear again, I'll keep you posted.
| |
| |
| | On Mon, Aug 13, 2012 at 5:13 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | And anoooother thing.
| | I tried to empty the ObjectLogEntry using topaz and this is what
| | happened:
| |
| | topaz 1> printit
| | ObjectLogEntry emptyLog
| |
| | %
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10399549697 does not exist.
| | Error Category: [GemStone] Number: 2101 Arg Count: 1
| | Arg 1: 10399549697
| |
| |
| | (There are 2 objects IDs bothering me like this, the one from the
| | log
| | I copied before and this one, both appear in the same kind of
| | errors).
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:55 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Another tip: Gemtools doesn't open the Debug list (well... it
| | actually takes so much time to open it that I give up and interrupt
| | the process).
| |
| | And yet another tip: this also appears in the logs. anUIOperacion
| | is
| | and instance of one of my objects and I guess it's within the
| | states
| | of the seaside component that triggered this.
| |
| | daemon Server started on port 9001
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | No method was found for the selector #'sequenceNumber' when sent to
| | aWAValueHolder contents: anUIOperacion->aRcCounterElement with
| | arguments
| | contained in anArray( ).
| | Error Category: 231169 [GemStone] Number: 2010 Arg Count: 3 Context
| | :
| | 10394577665
| | Arg 1: [10419891969 sz:2 cls: 67073 Association] an Association
| | key [8217729281 sz:1 cls: 123185665 WAValueHolder] a WAValueHolder
| | value [10391790081 sz:1 cls: 108289 RcCounterElement] a
| | RcCounterElement
| |
| | Arg 2: [6798849 sz:14 cls: 110849 Symbol] sequenceNumber
| | Arg 3: [10394577409 sz:0 cls: 66817 Array] an Array
| |
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> signal:args:signalDictionary: @8 line 15
| | [GsMethod 3241473]
| | 2 Object >> doesNotUnderstand: @6 line 14 [GsMethod 1926913]
| | 3 Object >> _doesNotUnderstand: @1 line 6 [GsMethod 1926657]
| | 4 RcQueue >> add: @11 line 13 [GsMethod 12454913]
| | 5 ObjectLogEntry >> addToLog @3 line 3 [GsMethod 20842497]
| | 6 ComplexBlock in GRGemStonePlatform >> seasideLogServerStart:port:
| | @12 line 8 [GsMethod 129763585]
| | 7 ComplexBlock in GRGemStonePlatform >> doTransaction: @9 line 15
| | [GsMethod 129762305]
| |
| | 8 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 9 ComplexBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| | 2304001]
| | 10 ComplexVCBlock in GRGemStonePlatform >> doTransaction: @13 line
| | 16
| | [GsMethod 129762305]
| | 11 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 12 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 13 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 14 GRGemStonePlatform >> doTransaction: @14 line 11 [GsMethod
| | 129762305]
| | 15 GRGemStonePlatform >> seasideLogServerStart:port: @13 line 3
| | [GsMethod 129763585]
| | 16 WAGemStoneRunSeasideGems class >> startGemServerOn: @4 line 7
| | [GsMethod 144519425]
| | 17 Executed Code @24 line 29 [GsMethod 10394554113]
| |
| |
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:47 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Hello!
| | I'm having some weird problems :S
| | It seems that there are a couple of objects "corrupted" that are
| | killing the seaside gems and preventing garbage collection.
| | I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.
| |
| | Here is a piece of log of one of the seaside gems:
| |
| | daemon Server started on port 9003
| | object already in in-memory oop map , objId 10390231041
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10390231041 is corrupt. Reason: 'object
| | already in in-memory oop map'
| | Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context
| | :
| | 10443254785
| | Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
| | Arg 2: [10390230785 sz:35 cls: 74753 String] object already in
| | in-memory oop map
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> _primitiveCommit: @1 line 1 [GsMethod
| | 3226881]
| | 2 System class >> __commit: @1 line 8 [GsMethod 3222017]
| | 3 System class >> _localCommit: @8 line 30 [GsMethod 3222529]
| | 4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3 [GsMethod
| | 7713281]
| | 5 System class >> _commit: @6 line 16 [GsMethod 3222785]
| | 6 System class >> commitTransaction @2 line 28 [GsMethod 3230465]
| | 7 System class >> _commitPrintingDiagnostics @2 line 9 [GsMethod
| | 3151105]
| | 8 SystemCommitTransaction >> defaultAction @1 line 3 [GsMethod
| | 126577409]
| | 9 ExceptionA >> _defaultAction @1 line 4 [GsMethod 10050561]
| | 10 ExceptionA >> signal @7 line 37 [GsMethod 10057473]
| | 11 ExceptionA >> signal: @3 line 12 [GsMethod 10050817]
| | 12 ExceptionA class >> signal: @2 line 3 [GsMethod 11895041]
| | 13 ExceptionA class >> signal @2 line 9 [GsMethod 11896577]
| | 14 GRGemStonePlatform >> doCommitTransaction @3 line 3 [GsMethod
| | 129758209]
| | 15 ComplexVCBlock in GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock: @28 line 32 [GsMethod
| | 374752257]
| | 16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 18 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 19 GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock:
| | @39 line 5 [GsMethod 374752257]
| | 20 ComplexBlock in GRGemStonePlatform >>
| | seasideProcessRequest:adaptor:resultBlock: @7 line 9 [GsMethod
| | 374752513]
| | 21 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 22 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
| | @32 line 17 [GsMethod 374752513]
| | 25 WAFastCGIAdaptor >> process: @4 line 4 [GsMethod 156093185]
| | 26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line
| | 4
| | [GsMethod 156095489]
| | 27 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 28 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5 [GsMethod
| | 156095489]
| | 31 FSResponderRole >> answer @2 line 4 [GsMethod 155147521]
| | 32 FSRole >> handleConnection @3 line 5 [GsMethod 155154177]
| | 33 FSConnection >> unsafeServe @4 line 8 [GsMethod 156518913]
| | 34 ComplexBlock in FSConnection >> safeServe @5 line 8 [GsMethod
| | 156521729]
| | 35 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 36 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 38 ComplexVCBlock in FSConnection >> safeServe @8 line 9 [GsMethod
| | 156521729]
| | 39 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 40 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 42 ComplexVCBlock in FSConnection >> safeServe @11 line 12
| | [GsMethod
| | 156521729]
| | 43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 45 FSConnection >> safeServe @14 line 15 [GsMethod 156521729]
| | 46 FSConnection >> serve @1 line 4 [GsMethod 156525569]
| | 47 ComplexBlock in FSSocketServer >> listen: @9 line 15 [GsMethod
| | 155629057]
| | 48 GsProcess >> _startPart2 @15 line 17 [GsMethod 4501249]
| | 49 GsProcess >> _start @1 line 9 [GsMethod 4501761]
| |
| | What could it be causing this? My extent keeps growing and growing
| | :(
| |
| | Cheers,
| | Carla
| |
| |
| |
| |
| |
| |
| |
| |
| |
|
|

Well, the object audit resulted in even more objects with errors :(  (just 5 more).

Here is something I don't follow. The object audit tells me this (among lots of other things):

Object 50334977, of class 79361, at 1-based offset 9, references nonexistent object 10306495233

So I grab the oop 50334977 and look up it's object:

topaz 1> object @50334977
a KeyValueDictionary
  numElements     2
  numCollisions   0
  collisionLimit  3
  tableSize       3
  #1 nil
  #2 nil
  #3[Object Does Not Exist]
  #4[Object Does Not Exist]
  #5[Object Does Not Exist]
  #6[Object Does Not Exist]

There you can see that the object with oop 50334977 is a KeyValueDictionary pointing to non existent objects.

So, I want to dereference this KeyValueDictionary, and in order to do so I need to know who is pointing to it. So the next step is:

topaz 1> printit
SystemRepository findReferencePathToObject: (System _objectForOop: 50334977 )
%
an Array
  #1 a KeyValueDictionary
  #2 false

The first element of that array is the same KeyValueDictionary with oop 50334977. So... I don't understand what to do. And why does false appear in the reference path?

How can I get these objects with this pattern of reference path to get cleaned up by the GC?

Cheers...
Carla

On Wed, Aug 15, 2012 at 4:15 PM, Carla F. Griggio <[hidden email]> wrote:

Hi Dale!

Well, I did an object audit and it showed that 923 objects had errors.

I inspected some of them, and they seem to be collections held by seaside components in the application cache.
Those collections point to non existent objects.

I think that the "Object does not exist" error is a consequence of this and not my main issue :) I mean, the 'guilty objects' are those in the repository that are pointing to non existent objects.

So I've just cleaned the application cache and I'm doing an audit now to see if at least I have less objects with errors.

Just for the record: as I suspected that the problems might come from old continuations, I tried to empty the Object Log, but again... the error that an object didn't exist would come up.
So I tried a silly workaround by reseting the instance variables objectLog and objectQueue of the ObjectLogEntry. I've just understood that my silly workaround was good for having a clean Object Log, BUT the old RcQueue with references to non existent objects kept living somewhere...
I found that old RcQueue, and when I tried to empty it the "Object does not exist" error came up. I could empty it by sending the message _initialize.
That released about 30MB from the repository.
I'm just posting this for future reference in case someone does the same mistake I did to force the Object Log to be empty :P

On Wed, Aug 15, 2012 at 3:33 PM, Dale Henrichs <[hidden email]> wrote:

Carla,

Our engineers have looked into the "object already in in-memory oop map" and here's the report:

  There have been a number of reports of this error, some associated with
  conversion, some associated with the MT scan. I didn't see anything But
  the structure is not persistent, so something else is going on.

  I didn't see a C level stack trace... that's usually what we'd ask for.
  Have her set GEM_HALT_ON_ERROR=2261 to get a stack.

As mentioned above the error isn't related to persistent data structures... Still useful to do an object audit to ensure that you've got a clean slate now that you've upgraded to 2.4.4.4.

Dale
----- Original Message -----
| From: "Carla F. Griggio" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>

| Sent: Tuesday, August 14, 2012 9:05:45 AM
| Subject: Re: [GS/SS Beta] Corrupted object preventing garbage collection?
|
| Thank you, Dale! I'll check that.
| I'll also update to 2.4.4.4, the release notes say that one of these
| problems was resolved in that version.
|
| Enjoy your vacations!
|
| Carla.
|
|
| On Tue, Aug 14, 2012 at 12:59 PM, Dale Henrichs < [hidden email]
| > wrote:
|
|
| Carla,
|
| I'm on vacation, so I just saw this thread ... the first thing that
| you should do is to run an object audit (see section 8.2 of the
| System Administration Guide[1]). This will tell us whether the
| corruption is persistent or is only showing up in temporary
| objects...
|
| It's a good sign that you're now having clean mfcs but the object
| audit will find any data base corruption that might be lurking ...
|
| The 'object already in in-memory oop map' rings a bell for me and I'm
| finding out more information about that particular error...
|
| Dale
|
|
| ----- Original Message -----
| | From: "Carla F. Griggio" < [hidden email] >
|
| | To: "GemStone Seaside beta discussion" < [hidden email]
| | >
|
|
| | Sent: Monday, August 13, 2012 1:42:43 PM
| | Subject: Re: [GS/SS Beta] Corrupted object preventing garbage
| | collection?
| |
| | Well... I killed the mantainance gem and the garbage collection
| | started to work ok again. But I still don't know if the errors in
| | the log will appear again, I'll keep you posted.
| |
| |
| | On Mon, Aug 13, 2012 at 5:13 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | And anoooother thing.
| | I tried to empty the ObjectLogEntry using topaz and this is what
| | happened:
| |
| | topaz 1> printit
| | ObjectLogEntry emptyLog
| |
| | %
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10399549697 does not exist.
| | Error Category: [GemStone] Number: 2101 Arg Count: 1
| | Arg 1: 10399549697
| |
| |
| | (There are 2 objects IDs bothering me like this, the one from the
| | log
| | I copied before and this one, both appear in the same kind of
| | errors).
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:55 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Another tip: Gemtools doesn't open the Debug list (well... it
| | actually takes so much time to open it that I give up and interrupt
| | the process).
| |
| | And yet another tip: this also appears in the logs. anUIOperacion
| | is
| | and instance of one of my objects and I guess it's within the
| | states
| | of the seaside component that triggered this.
| |
| | daemon Server started on port 9001
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | No method was found for the selector #'sequenceNumber' when sent to
| | aWAValueHolder contents: anUIOperacion->aRcCounterElement with
| | arguments
| | contained in anArray( ).
| | Error Category: 231169 [GemStone] Number: 2010 Arg Count: 3 Context
| | :
| | 10394577665
| | Arg 1: [10419891969 sz:2 cls: 67073 Association] an Association
| | key [8217729281 sz:1 cls: 123185665 WAValueHolder] a WAValueHolder
| | value [10391790081 sz:1 cls: 108289 RcCounterElement] a
| | RcCounterElement
| |
| | Arg 2: [6798849 sz:14 cls: 110849 Symbol] sequenceNumber
| | Arg 3: [10394577409 sz:0 cls: 66817 Array] an Array
| |
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> signal:args:signalDictionary: @8 line 15
| | [GsMethod 3241473]
| | 2 Object >> doesNotUnderstand: @6 line 14 [GsMethod 1926913]
| | 3 Object >> _doesNotUnderstand: @1 line 6 [GsMethod 1926657]
| | 4 RcQueue >> add: @11 line 13 [GsMethod 12454913]
| | 5 ObjectLogEntry >> addToLog @3 line 3 [GsMethod 20842497]
| | 6 ComplexBlock in GRGemStonePlatform >> seasideLogServerStart:port:
| | @12 line 8 [GsMethod 129763585]
| | 7 ComplexBlock in GRGemStonePlatform >> doTransaction: @9 line 15
| | [GsMethod 129762305]
| |
| | 8 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 9 ComplexBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| | 2304001]
| | 10 ComplexVCBlock in GRGemStonePlatform >> doTransaction: @13 line
| | 16
| | [GsMethod 129762305]
| | 11 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 12 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 13 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 14 GRGemStonePlatform >> doTransaction: @14 line 11 [GsMethod
| | 129762305]
| | 15 GRGemStonePlatform >> seasideLogServerStart:port: @13 line 3
| | [GsMethod 129763585]
| | 16 WAGemStoneRunSeasideGems class >> startGemServerOn: @4 line 7
| | [GsMethod 144519425]
| | 17 Executed Code @24 line 29 [GsMethod 10394554113]
| |
| |
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:47 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Hello!
| | I'm having some weird problems :S
| | It seems that there are a couple of objects "corrupted" that are
| | killing the seaside gems and preventing garbage collection.
| | I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.
| |
| | Here is a piece of log of one of the seaside gems:
| |
| | daemon Server started on port 9003
| | object already in in-memory oop map , objId 10390231041
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10390231041 is corrupt. Reason: 'object
| | already in in-memory oop map'
| | Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context
| | :
| | 10443254785
| | Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
| | Arg 2: [10390230785 sz:35 cls: 74753 String] object already in
| | in-memory oop map
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> _primitiveCommit: @1 line 1 [GsMethod
| | 3226881]
| | 2 System class >> __commit: @1 line 8 [GsMethod 3222017]
| | 3 System class >> _localCommit: @8 line 30 [GsMethod 3222529]
| | 4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3 [GsMethod
| | 7713281]
| | 5 System class >> _commit: @6 line 16 [GsMethod 3222785]
| | 6 System class >> commitTransaction @2 line 28 [GsMethod 3230465]
| | 7 System class >> _commitPrintingDiagnostics @2 line 9 [GsMethod
| | 3151105]
| | 8 SystemCommitTransaction >> defaultAction @1 line 3 [GsMethod
| | 126577409]
| | 9 ExceptionA >> _defaultAction @1 line 4 [GsMethod 10050561]
| | 10 ExceptionA >> signal @7 line 37 [GsMethod 10057473]
| | 11 ExceptionA >> signal: @3 line 12 [GsMethod 10050817]
| | 12 ExceptionA class >> signal: @2 line 3 [GsMethod 11895041]
| | 13 ExceptionA class >> signal @2 line 9 [GsMethod 11896577]
| | 14 GRGemStonePlatform >> doCommitTransaction @3 line 3 [GsMethod
| | 129758209]
| | 15 ComplexVCBlock in GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock: @28 line 32 [GsMethod
| | 374752257]
| | 16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 18 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 19 GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock:
| | @39 line 5 [GsMethod 374752257]
| | 20 ComplexBlock in GRGemStonePlatform >>
| | seasideProcessRequest:adaptor:resultBlock: @7 line 9 [GsMethod
| | 374752513]
| | 21 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 22 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
| | @32 line 17 [GsMethod 374752513]
| | 25 WAFastCGIAdaptor >> process: @4 line 4 [GsMethod 156093185]
| | 26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line
| | 4
| | [GsMethod 156095489]
| | 27 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 28 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5 [GsMethod
| | 156095489]
| | 31 FSResponderRole >> answer @2 line 4 [GsMethod 155147521]
| | 32 FSRole >> handleConnection @3 line 5 [GsMethod 155154177]
| | 33 FSConnection >> unsafeServe @4 line 8 [GsMethod 156518913]
| | 34 ComplexBlock in FSConnection >> safeServe @5 line 8 [GsMethod
| | 156521729]
| | 35 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 36 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 38 ComplexVCBlock in FSConnection >> safeServe @8 line 9 [GsMethod
| | 156521729]
| | 39 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 40 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 42 ComplexVCBlock in FSConnection >> safeServe @11 line 12
| | [GsMethod
| | 156521729]
| | 43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 45 FSConnection >> safeServe @14 line 15 [GsMethod 156521729]
| | 46 FSConnection >> serve @1 line 4 [GsMethod 156525569]
| | 47 ComplexBlock in FSSocketServer >> listen: @9 line 15 [GsMethod
| | 155629057]
| | 48 GsProcess >> _startPart2 @15 line 17 [GsMethod 4501249]
| | 49 GsProcess >> _start @1 line 9 [GsMethod 4501761]
| |
| | What could it be causing this? My extent keeps growing and growing
| | :(
| |
| | Cheers,
| | Carla
| |
| |
| |
| |
| |
| |
| |
| |
| |
|
|

More funny stuff:

topaz 1> printit
SystemRepository findReferencePathToObject: (System _objectForOop: 50334977 )

%
an Array
  #1 a KeyValueDictionary
  #2 false

topaz 1> object ** at: 1

a KeyValueDictionary
  numElements     0
  numCollisions   0
  collisionLimit  3
  tableSize       3
  #1 nil
  #2 nil

  #3 nil
  #4 nil
  #5 nil
  #6 nil

How can that be possible? I managed to empty the dictionary, but it still appears as a reference of itself?

Maybe I'm reading wrong the results of the findReferencePathToObject: method, but I thought that if no object pointed to the one I use as the parameter, the result was nil. I checked that doing:

topaz 1> printit
SystemRepository findReferencePathToObject: Set new
%
nil

On Wed, Aug 15, 2012 at 5:03 PM, Carla F. Griggio <[hidden email]> wrote:

Well, the object audit resulted in even more objects with errors :(  (just 5 more).

Here is something I don't follow. The object audit tells me this (among lots of other things):

Object 50334977, of class 79361, at 1-based offset 9, references nonexistent object 10306495233

So I grab the oop 50334977 and look up it's object:

topaz 1> object @50334977
a KeyValueDictionary
  numElements     2
  numCollisions   0
  collisionLimit  3
  tableSize       3
  #1 nil
  #2 nil
  #3[Object Does Not Exist]
  #4[Object Does Not Exist]
  #5[Object Does Not Exist]
  #6[Object Does Not Exist]

There you can see that the object with oop 50334977 is a KeyValueDictionary pointing to non existent objects.

So, I want to dereference this KeyValueDictionary, and in order to do so I need to know who is pointing to it. So the next step is:

topaz 1> printit
SystemRepository findReferencePathToObject: (System _objectForOop: 50334977 )
%
an Array
  #1 a KeyValueDictionary
  #2 false

The first element of that array is the same KeyValueDictionary with oop 50334977. So... I don't understand what to do. And why does false appear in the reference path?

How can I get these objects with this pattern of reference path to get cleaned up by the GC?

Cheers...
Carla

On Wed, Aug 15, 2012 at 4:15 PM, Carla F. Griggio <[hidden email]> wrote:

Hi Dale!

Well, I did an object audit and it showed that 923 objects had errors.

I inspected some of them, and they seem to be collections held by seaside components in the application cache.
Those collections point to non existent objects.

I think that the "Object does not exist" error is a consequence of this and not my main issue :) I mean, the 'guilty objects' are those in the repository that are pointing to non existent objects.

So I've just cleaned the application cache and I'm doing an audit now to see if at least I have less objects with errors.

Just for the record: as I suspected that the problems might come from old continuations, I tried to empty the Object Log, but again... the error that an object didn't exist would come up.
So I tried a silly workaround by reseting the instance variables objectLog and objectQueue of the ObjectLogEntry. I've just understood that my silly workaround was good for having a clean Object Log, BUT the old RcQueue with references to non existent objects kept living somewhere...
I found that old RcQueue, and when I tried to empty it the "Object does not exist" error came up. I could empty it by sending the message _initialize.
That released about 30MB from the repository.
I'm just posting this for future reference in case someone does the same mistake I did to force the Object Log to be empty :P

On Wed, Aug 15, 2012 at 3:33 PM, Dale Henrichs <[hidden email]> wrote:

Carla,

Our engineers have looked into the "object already in in-memory oop map" and here's the report:

  There have been a number of reports of this error, some associated with
  conversion, some associated with the MT scan. I didn't see anything But
  the structure is not persistent, so something else is going on.

  I didn't see a C level stack trace... that's usually what we'd ask for.
  Have her set GEM_HALT_ON_ERROR=2261 to get a stack.

As mentioned above the error isn't related to persistent data structures... Still useful to do an object audit to ensure that you've got a clean slate now that you've upgraded to 2.4.4.4.

Dale
----- Original Message -----
| From: "Carla F. Griggio" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>

| Sent: Tuesday, August 14, 2012 9:05:45 AM
| Subject: Re: [GS/SS Beta] Corrupted object preventing garbage collection?
|
| Thank you, Dale! I'll check that.
| I'll also update to 2.4.4.4, the release notes say that one of these
| problems was resolved in that version.
|
| Enjoy your vacations!
|
| Carla.
|
|
| On Tue, Aug 14, 2012 at 12:59 PM, Dale Henrichs < [hidden email]
| > wrote:
|
|
| Carla,
|
| I'm on vacation, so I just saw this thread ... the first thing that
| you should do is to run an object audit (see section 8.2 of the
| System Administration Guide[1]). This will tell us whether the
| corruption is persistent or is only showing up in temporary
| objects...
|
| It's a good sign that you're now having clean mfcs but the object
| audit will find any data base corruption that might be lurking ...
|
| The 'object already in in-memory oop map' rings a bell for me and I'm
| finding out more information about that particular error...
|
| Dale
|
|
| ----- Original Message -----
| | From: "Carla F. Griggio" < [hidden email] >
|
| | To: "GemStone Seaside beta discussion" < [hidden email]
| | >
|
|
| | Sent: Monday, August 13, 2012 1:42:43 PM
| | Subject: Re: [GS/SS Beta] Corrupted object preventing garbage
| | collection?
| |
| | Well... I killed the mantainance gem and the garbage collection
| | started to work ok again. But I still don't know if the errors in
| | the log will appear again, I'll keep you posted.
| |
| |
| | On Mon, Aug 13, 2012 at 5:13 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | And anoooother thing.
| | I tried to empty the ObjectLogEntry using topaz and this is what
| | happened:
| |
| | topaz 1> printit
| | ObjectLogEntry emptyLog
| |
| | %
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10399549697 does not exist.
| | Error Category: [GemStone] Number: 2101 Arg Count: 1
| | Arg 1: 10399549697
| |
| |
| | (There are 2 objects IDs bothering me like this, the one from the
| | log
| | I copied before and this one, both appear in the same kind of
| | errors).
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:55 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Another tip: Gemtools doesn't open the Debug list (well... it
| | actually takes so much time to open it that I give up and interrupt
| | the process).
| |
| | And yet another tip: this also appears in the logs. anUIOperacion
| | is
| | and instance of one of my objects and I guess it's within the
| | states
| | of the seaside component that triggered this.
| |
| | daemon Server started on port 9001
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | No method was found for the selector #'sequenceNumber' when sent to
| | aWAValueHolder contents: anUIOperacion->aRcCounterElement with
| | arguments
| | contained in anArray( ).
| | Error Category: 231169 [GemStone] Number: 2010 Arg Count: 3 Context
| | :
| | 10394577665
| | Arg 1: [10419891969 sz:2 cls: 67073 Association] an Association
| | key [8217729281 sz:1 cls: 123185665 WAValueHolder] a WAValueHolder
| | value [10391790081 sz:1 cls: 108289 RcCounterElement] a
| | RcCounterElement
| |
| | Arg 2: [6798849 sz:14 cls: 110849 Symbol] sequenceNumber
| | Arg 3: [10394577409 sz:0 cls: 66817 Array] an Array
| |
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> signal:args:signalDictionary: @8 line 15
| | [GsMethod 3241473]
| | 2 Object >> doesNotUnderstand: @6 line 14 [GsMethod 1926913]
| | 3 Object >> _doesNotUnderstand: @1 line 6 [GsMethod 1926657]
| | 4 RcQueue >> add: @11 line 13 [GsMethod 12454913]
| | 5 ObjectLogEntry >> addToLog @3 line 3 [GsMethod 20842497]
| | 6 ComplexBlock in GRGemStonePlatform >> seasideLogServerStart:port:
| | @12 line 8 [GsMethod 129763585]
| | 7 ComplexBlock in GRGemStonePlatform >> doTransaction: @9 line 15
| | [GsMethod 129762305]
| |
| | 8 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 9 ComplexBlock in ExecutableBlock >> ensure: @6 line 11 [GsMethod
| | 2304001]
| | 10 ComplexVCBlock in GRGemStonePlatform >> doTransaction: @13 line
| | 16
| | [GsMethod 129762305]
| | 11 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 12 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 13 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 14 GRGemStonePlatform >> doTransaction: @14 line 11 [GsMethod
| | 129762305]
| | 15 GRGemStonePlatform >> seasideLogServerStart:port: @13 line 3
| | [GsMethod 129763585]
| | 16 WAGemStoneRunSeasideGems class >> startGemServerOn: @4 line 7
| | [GsMethod 144519425]
| | 17 Executed Code @24 line 29 [GsMethod 10394554113]
| |
| |
| |
| |
| |
| |
| | On Mon, Aug 13, 2012 at 4:47 PM, Carla F. Griggio <
| | [hidden email] > wrote:
| |
| |
| | Hello!
| | I'm having some weird problems :S
| | It seems that there are a couple of objects "corrupted" that are
| | killing the seaside gems and preventing garbage collection.
| | I'm using Gemstone 2.4.4.1 and Seaside 3.0.6.
| |
| | Here is a piece of log of one of the seaside gems:
| |
| | daemon Server started on port 9003
| | object already in in-memory oop map , objId 10390231041
| | -----------------------------------------------------
| | GemStone: Error Nonfatal
| | The object with object ID 10390231041 is corrupt. Reason: 'object
| | already in in-memory oop map'
| | Error Category: 231169 [GemStone] Number: 2261 Arg Count: 2 Context
| | :
| | 10443254785
| | Arg 1: [83121848330 sz:0 cls: 74241 SmallInteger] 10390231041
| | Arg 2: [10390230785 sz:35 cls: 74753 String] object already in
| | in-memory oop map
| |
| | Now executing the following command saved from "iferr 1":
| | where
| | ==> 1 System class >> _primitiveCommit: @1 line 1 [GsMethod
| | 3226881]
| | 2 System class >> __commit: @1 line 8 [GsMethod 3222017]
| | 3 System class >> _localCommit: @8 line 30 [GsMethod 3222529]
| | 4 TransactionBoundaryDefaultPolicy >> commit: @2 line 3 [GsMethod
| | 7713281]
| | 5 System class >> _commit: @6 line 16 [GsMethod 3222785]
| | 6 System class >> commitTransaction @2 line 28 [GsMethod 3230465]
| | 7 System class >> _commitPrintingDiagnostics @2 line 9 [GsMethod
| | 3151105]
| | 8 SystemCommitTransaction >> defaultAction @1 line 3 [GsMethod
| | 126577409]
| | 9 ExceptionA >> _defaultAction @1 line 4 [GsMethod 10050561]
| | 10 ExceptionA >> signal @7 line 37 [GsMethod 10057473]
| | 11 ExceptionA >> signal: @3 line 12 [GsMethod 10050817]
| | 12 ExceptionA class >> signal: @2 line 3 [GsMethod 11895041]
| | 13 ExceptionA class >> signal @2 line 9 [GsMethod 11896577]
| | 14 GRGemStonePlatform >> doCommitTransaction @3 line 3 [GsMethod
| | 129758209]
| | 15 ComplexVCBlock in GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock: @28 line 32 [GsMethod
| | 374752257]
| | 16 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 17 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 18 TransientRecursionLock >> critical: @15 line 8 [GsMethod
| | 21159937]
| | 19 GRGemStonePlatform >>
| | seasideProcessRequestWithRetry:resultBlock:
| | @39 line 5 [GsMethod 374752257]
| | 20 ComplexBlock in GRGemStonePlatform >>
| | seasideProcessRequest:adaptor:resultBlock: @7 line 9 [GsMethod
| | 374752513]
| | 21 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 22 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 23 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 24 GRGemStonePlatform >> seasideProcessRequest:adaptor:resultBlock:
| | @32 line 17 [GsMethod 374752513]
| | 25 WAFastCGIAdaptor >> process: @4 line 4 [GsMethod 156093185]
| | 26 ComplexBlock in WAFastCGIAdaptor >> answerResponderRole: @3 line
| | 4
| | [GsMethod 156095489]
| | 27 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 28 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 29 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 30 WAFastCGIAdaptor >> answerResponderRole: @9 line 5 [GsMethod
| | 156095489]
| | 31 FSResponderRole >> answer @2 line 4 [GsMethod 155147521]
| | 32 FSRole >> handleConnection @3 line 5 [GsMethod 155154177]
| | 33 FSConnection >> unsafeServe @4 line 8 [GsMethod 156518913]
| | 34 ComplexBlock in FSConnection >> safeServe @5 line 8 [GsMethod
| | 156521729]
| | 35 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 36 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 37 ComplexBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 38 ComplexVCBlock in FSConnection >> safeServe @8 line 9 [GsMethod
| | 156521729]
| | 39 ExceptionHandler >> doTryBlock: @9 line 7 [GsMethod 10065409]
| | 40 ExceptionHandler >> try:on:do: @15 line 18 [GsMethod 10062081]
| | 41 ComplexVCBlock in ExecutableBlock >> on:do: @2 line 8 [GsMethod
| | 9005057]
| | 42 ComplexVCBlock in FSConnection >> safeServe @11 line 12
| | [GsMethod
| | 156521729]
| | 43 ComplexBlock in ExecutableBlock >> ensure: @4 line 11 [GsMethod
| | 2304001]
| | 44 ComplexVCBlock in ExecutableBlock >> ensure: @6 line 11
| | [GsMethod
| | 2304001]
| | 45 FSConnection >> safeServe @14 line 15 [GsMethod 156521729]
| | 46 FSConnection >> serve @1 line 4 [GsMethod 156525569]
| | 47 ComplexBlock in FSSocketServer >> listen: @9 line 15 [GsMethod
| | 155629057]
| | 48 GsProcess >> _startPart2 @15 line 17 [GsMethod 4501249]
| | 49 GsProcess >> _start @1 line 9 [GsMethod 4501761]
| |
| | What could it be causing this? My extent keeps growing and growing
| | :(
| |
| | Cheers,
| | Carla
| |
| |
| |
| |
| |
| |
| |
| |
| |
|
|