[Glass] fastcgi process unresponsive

Hi,

We've been running FastCGI on our production sites for about 3 weeks
now (GS 2.4.4.4). Periodically, one of the FCGI servers becomes
unhappy, but remains running.

The symptoms are that the topaz process is running, but the listener
on the socket is dead.

More details below, if you're interested. Any ideas will be appreciated.

Thanks
Otto

This is what I see in the logs:

-----------
failure
Read-Write Conflicts...
Write-Write Conflicts...
    20782838529
    .. bla, bla, list of about 30 oops ...
Write-Dependency Conflicts...
Write-ReadLock Conflicts...
Write-WriteLock Conflicts...
Rc-Write-Write Conflicts...
Synchronized-Commit Conflicts...
----------- Commit failure - retrying LOG ENTRY: aSymbolDictionary-----------

The oops are objects in our domain. So, I'm assuming that it may be 2
sessions trying to update the same objects.

Then we get

-----------------------------------------------------
GemStone: Error         Nonfatal
The object with object ID 81065894012259328 does not exist.
Error Category: 231169 [GemStone] Number: 2101 Arg Count: 1 Context :
19918721793
Arg 1: [648527152098074626 sz:0 cls: 74241 SmallInteger] 81065894012259328

Now executing the following command saved from "iferr 1":
   where
==> 1 WAComponent >> lightbox: @9 line 4   [GsMethod 3438252033]
2 PortfolioInformationContractableSectionRenderer >> overview @5 line
3   [GsMethod 350239489]

... and a long stack

149 GsProcess >> _start @1 line 9   [GsMethod 260584961]
  [GsProcess 19918721793]
topaz 1> [268 sz:0 cls: 68097 Boolean] true
topaz 1>
HostCallDebugger: invoked at: Mon Dec 30 11:02:55 SAST 2013

UTL_GUARANTEE failed, File
/export/toronto3/users/buildgss/244x-1/src/omverify.c line 28

Begin attempt to print C-level stack at: Mon Dec 30 11:02:55 SAST 2013


End of C-level stack:


hostcalldebugger invoked in process 90503, at 12/30/2013 11:02:55 AM.065 SAST
 notifying stone of fatal error

The stone log simply says:

--- 30/12/2013 11:02:55 SAST ---
    Fatal Internal Error Notification from Gem, user = DataCurator  Session = 9
      Gem hostName = localhost , Gem processId = 90503,
      current  CommitRecord = 74986
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Otto,

That looks similar to a problem that showed up in 3.x ... the corruption was happening in tempobj space, so no persistent corruption was occuring ... probably should do an object audit to confirm that the corruption is localized to the vm ...

I'll check with the vm guys on monday and see what they say ...

The fact that the vm remains running should probably be tracked down ...

Dale

----- Original Message -----
| From: "Otto Behrens" <[hidden email]>
| To: [hidden email]
| Cc: [hidden email]
| Sent: Monday, December 30, 2013 2:41:56 AM
| Subject: [Glass] fastcgi process unresponsive
|
| Hi,
|
| We've been running FastCGI on our production sites for about 3 weeks
| now (GS 2.4.4.4). Periodically, one of the FCGI servers becomes
| unhappy, but remains running.
|
| The symptoms are that the topaz process is running, but the listener
| on the socket is dead.
|
| More details below, if you're interested. Any ideas will be
| appreciated.
|
| Thanks
| Otto
|
| This is what I see in the logs:
|
| -----------
| failure
| Read-Write Conflicts...
| Write-Write Conflicts...
|     20782838529
|     .. bla, bla, list of about 30 oops ...
| Write-Dependency Conflicts...
| Write-ReadLock Conflicts...
| Write-WriteLock Conflicts...
| Rc-Write-Write Conflicts...
| Synchronized-Commit Conflicts...
| ----------- Commit failure - retrying LOG ENTRY:
| aSymbolDictionary-----------
|
| The oops are objects in our domain. So, I'm assuming that it may be 2
| sessions trying to update the same objects.
|
| Then we get
|
| -----------------------------------------------------
| GemStone: Error         Nonfatal
| The object with object ID 81065894012259328 does not exist.
| Error Category: 231169 [GemStone] Number: 2101 Arg Count: 1 Context :
| 19918721793
| Arg 1: [648527152098074626 sz:0 cls: 74241 SmallInteger]
| 81065894012259328
|
| Now executing the following command saved from "iferr 1":
|    where
| ==> 1 WAComponent >> lightbox: @9 line 4   [GsMethod 3438252033]
| 2 PortfolioInformationContractableSectionRenderer >> overview @5 line
| 3   [GsMethod 350239489]
|
| ... and a long stack
|
| 149 GsProcess >> _start @1 line 9   [GsMethod 260584961]
|   [GsProcess 19918721793]
| topaz 1> [268 sz:0 cls: 68097 Boolean] true
| topaz 1>
| HostCallDebugger: invoked at: Mon Dec 30 11:02:55 SAST 2013
|
| UTL_GUARANTEE failed, File
| /export/toronto3/users/buildgss/244x-1/src/omverify.c line 28
|
| Begin attempt to print C-level stack at: Mon Dec 30 11:02:55 SAST
| 2013
|
|
| End of C-level stack:
|
|
| hostcalldebugger invoked in process 90503, at 12/30/2013 11:02:55
| AM.065 SAST
|  notifying stone of fatal error
|
| The stone log simply says:
|
| --- 30/12/2013 11:02:55 SAST ---
|     Fatal Internal Error Notification from Gem, user = DataCurator
|      Session = 9
|       Gem hostName = localhost , Gem processId = 90503,
|       current  CommitRecord = 74986
| _______________________________________________
| Glass mailing list
| [hidden email]
| http://lists.gemtalksystems.com/mailman/listinfo/glass
|
_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass