We've recently moved our application to a new server, and my Zinc gems are crashing when continuations are saved:
a ImproperOperation occurred (error 2252), Error 'in StackLoad' , 'saved stack is LinuxX86_64, process switch expects interpreted' . Execution cannot continue. This wasn't happening on the old server, and configs are identical as far as I can tell. I disabled native code in the gem, thinking that'd help, but it made no difference. In a way, I'm glad about that because I really want to use native code in production... but now I'm at a loss as to where this error is coming from. We're still on 3.1.0.4. The new server is Debian 7, while the old one was Ubuntu 12.04.4 LTS. Kernels are essentially the same (3.2.0). Both the old server and the new server are cloud instances, old was at Rackspace with AMD processors, and the new is at Amazon with Intel Xeons. -- Ken Treis Miriam Technologies, Inc. (866) 652-2040 x221 ==== _____________________________________________________________________________ | Configuration Files | | | | System File: /opt/gemstone/GemStone64Bit3.1.0.4-x86_64.Linux/seaside/data/system.conf | | | Executable File: /opt/gemstone/GemStone64Bit3.1.0.4-x86_64.Linux/seaside/etc/seaside30.conf |_____________________________________________________________________________| _____________________________________________________________________________ | Gem Configuration Options for process id 15785 | |_____________________________________________________________________________| DUMP_OPTIONS = TRUE; GEM_GCI_LOG_ENABLED = FALSE; GEM_ABORT_MAX_CRS = 0; GEM_FREE_FRAME_CACHE_SIZE = -1; GEM_FREE_FRAME_LIMIT = -1; GEM_FREE_PAGEIDS_CACHE = 200; GEM_HALT_ON_ERROR = 0; GEM_KEEP_MIN_SOFTREFS = 0; GEM_MAX_SMALLTALK_STACK_DEPTH = 1000; GEM_NATIVE_CODE_ENABLED = FALSE; GEM_PRIVATE_PAGE_CACHE_KB = 960; GEM_PGSVR_COMPRESS_PAGE_TRANSFERS = FALSE; GEM_PGSVR_FREE_FRAME_CACHE_SIZE = -1; GEM_PGSVR_FREE_FRAME_LIMIT = -1; GEM_PGSVR_UPDATE_CACHE_ON_READ = FALSE; GEM_RPC_KEEPALIVE_INTERVAL = 0; GEM_RPCGCI_TIMEOUT = 0; GEM_RPC_USE_SSL = TRUE; GEM_SOFTREF_CLEANUP_PERCENT_MEM = 50; GEM_TEMPOBJ_AGGRESSIVE_STUBBING = TRUE; GEM_TEMPOBJ_CACHE_SIZE = 400000; GEM_TEMPOBJ_MESPACE_SIZE = 0; GEM_TEMPOBJ_OOPMAP_SIZE = 0; GEM_TEMPOBJ_SCOPES_SIZE = 2000; GEM_TEMPOBJ_POMGEN_SIZE = 0; GEM_TEMPOBJ_POMGEN_PRUNE_ON_VOTE = 100; GEM_TEMPOBJ_POMGEN_SCAVENGE_INTERVAL = 1800; GEM_TEMPOBJ_START_ADDR not used on this platform LOG_WARNINGS = TRUE; SHR_NUM_FREE_FRAME_SERVERS = -1; SHR_PAGE_CACHE_LOCKED = FALSE; SHR_PAGE_CACHE_NUM_PROCS = 4089; SHR_PAGE_CACHE_NUM_SHARED_COUNTERS = 1900; SHR_PAGE_CACHE_PERMISSIONS = 660; SHR_PAGE_CACHE_SIZE_KB = 4000000; SHR_TARGET_FREE_FRAME_COUNT = -1; SHR_WELL_KNOWN_PORT_NUMBER = 0; (vmGc spaceSizes: eden init 2048K max 74944K , survivor init 448K max 12544K, vmGc old max 299968K, code max 80000K, perm max 40000K, pom 10 * 33344K = 333440K, vmGc remSet 8068K, meSpace max 382432K oopMapSize 2097152 max footprint 1250M) _____________________________________________________________________________ | GemStone/S64 Object-Oriented Data Management System | | Copyright (C) VMware, Inc. 1986-2013 | | All rights reserved. | | Covered by U.S Patents: | | 6,256,637 Transactional virtual machine architecture | | 6,360,219 Object queues with concurrent updating | | 6,567,905 Generational Garbage Collector. | | 6,681,226 Selective Pessimistic Locking for a Concurrently Updateable Database +-----------------------------------------------------------------------------+ | PROGRAM: topaz, Linear GemStone Interface (Linked Session) | | VERSION: 3.1.0.4, Mon Jun 17 13:16:00 2013 | | BUILD: gss64_3_1_0_x_branch-31024 | | BUILT FOR: x86-64 (Linux) | | MODE: 64 bit | | RUNNING ON: 4-CPU www.weco-write.com x86_64 (Linux 3.2.0-4-amd64 #1 SMP Debian | 3.2.57-3) 30488MB | | PROCESS ID: 15785 DATE: 07/17/2014 10:30:24 PDT | | USER IDS: REAL=gsadmin (1004) EFFECTIVE=gsadmin (1004) | |_____________________________________________________________________________| neither .topazini nor $HOME/.topazini were found topaz> topaz> topaz> topaz> topaz> topaz> topaz> topaz> topaz> topaz> topaz> topaz> topaz> topaz> [Info]: LNK client/gem GCI levels = 851/851 [Info]: libssl-3.1.0.4-64.so: loaded [Info]: User ID: DataCurator [Info]: Repository: seaside [Info]: Session ID: 15 [Info]: GCI Client Host: <Linked> [Info]: Page server PID: -1 [Info]: Login Time: 07/17/2014 10:30:24.910 PDT [Info]: libicudata-3.1.0.4-64.so: loaded [Info]: libicuuc-3.1.0.4-64.so: loaded [Info]: libicui18n-3.1.0.4-64.so: loaded Gave this VM preference for OOM killer, Wrote to /proc/15785/oom_score_adj value 250 [07/17/2014 10:30:24.941 PDT] gci login: currSession 1 rpc gem processId -1 OOB keep-alive interval -1 successful login topaz 1> topaz 1> topaz 1> topaz 1> [71841854977 sz:5 cls: 135169 GsFile] aGsFile topaz 1> topaz 1> topaz 1> topaz 1> [76033 sz:19 cls: 779521 System class] System topaz 1> topaz 1> topaz 1> topaz 1> Zinc Server started on port 9008 ----------- Continuation saved to object log LOG ENTRY: anArray----------- ----------- Continuation saved to object log ERROR Encountered: 2014-07-17T10:31:00.24015998840332-07:00 a ImproperOperation occurred (error 2252), Error 'in StackLoad' , 'saved stack is LinuxX86_64, process switch expects interpreted' . Execution cannot continue. 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line 4 [GsNMethod 67373805057] 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod 67373809409] 3 WAErrorHandler >> saveExceptionContinuation: (envId 0) @8 line 6 [GsNMethod 67530876161] _______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
OK, I have a little more detail on this. I misread the error, it turns out that the StackLoad error doesn't occur while saving a continuation for debug. Those are getting saved and I can debug them in GemTools. The error is apparently triggered when trying to run a callback.
Here's the full stack, relevant section emphasized: ----------- Continuation saved to object log ERROR Encountered: 2014-07-17T11:43:56.08789706230164-07:00 a ImproperOperation occurred (error 2252), Error 'in StackLoad' , 'saved stack is LinuxX86_64, process switch expects interpreted' . Execution cannot continue. 1 GRGemStonePlatform >> logError:title:shouldCommit: (envId 0) @2 line 4 [GsNMethod 67373805057] 2 GRGemStonePlatform >> logError:title: (envId 0) @2 line 3 [GsNMethod 67373809409] 3 WAErrorHandler >> saveExceptionContinuation: (envId 0) @8 line 6 [GsNMethod 67530876161] 4 WECOProductionErrorHandler >> saveException: (envId 0) @5 line 5 [GsNMethod 67645394433] 5 WECOProductionErrorHandler >> handleDefault: (envId 0) @2 line 2 [GsNMethod 67645395713] 6 WAErrorHandler >> handleError: (envId 0) @2 line 2 [GsNMethod 67530874113] 7 WECOProductionErrorHandler >> handleError: (envId 0) @7 line 2 [GsNMethod 67645393921] 8 WAErrorHandler >> handleGemStoneException: (envId 0) @4 line 4 [GsNMethod 67530874369] 9 WAHtmlHaltAndErrorHandler >> handleException: (envId 0) @2 line 2 [GsNMethod 67547940609] 10 [] in WAExceptionHandler >> handleExceptionsDuring: (envId 0) @2 line 5 [GsNMethod 67615814145] 11 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 12 [] in WAExceptionHandler >> handleExceptionsDuring: (envId 0) @2 line 8 [GsNMethod 67589087233] 13 [] in ExecBlock >> on:do: (envId 0) @4 line 49 [GsNMethod 37709826305] 14 AbstractException >> _executeHandler: (envId 0) @3 line 8 [GsNMethod 129979137] 15 AbstractException >> _signalFromPrimitive: (envId 0) @1 line 1 [GsNMethod 129984257] 16 WACallback >> evaluateWithFieldValues: (envId 0) @1 line 1 [GsNMethod 67530766081] 17 [] in WACallbackRegistry >> handle: (envId 0) @4 line 10 [GsNMethod 67590116097] 18 Collection >> do: (envId 0) @5 line 10 [GsNMethod 121376513] 19 WACallbackRegistry >> handle: (envId 0) @9 line 9 [GsNMethod 67530772225] 20 [] in WAActionPhaseContinuation >> runCallbacks (envId 0) @5 line 4 [GsNMethod 67615798273] 21 ExecBlock >> onException:do: (envId 0) @2 line 66 [GsNMethod 125587969] 22 ExecBlock >> on:do: (envId 0) @5 line 47 [GsNMethod 125587457] 23 WAExceptionHandler >> handleExceptionsDuring: (envId 0) @2 line 3 [GsNMethod 67533463809] 24 [] in WARenderLoopContinuation >> withNotificationHandlerDo: (envId 0) @2 line 16 [GsNMethod 67591011585] 25 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 26 WARenderLoopContinuation >> withNotificationHandlerDo: (envId 0) @4 line 17 [GsNMethod 67535573761] 27 [] in WAActionPhaseContinuation >> runCallbacks (envId 0) @2 line 3 [GsNMethod 67591003905] 28 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 29 WAActionPhaseContinuation >> runCallbacks (envId 0) @2 line 6 [GsNMethod 67535567105] 30 WAActionPhaseContinuation >> handleRequest (envId 0) @2 line 2 [GsNMethod 67535568129] 31 [] in WASessionContinuation >> basicValue (envId 0) @2 line 2 [GsNMethod 67591037697] 32 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 33 WASessionContinuation >> withUnregisteredHandlerDo: (envId 0) @2 line 3 [GsNMethod 67535609601] 34 WASessionContinuation >> basicValue (envId 0) @2 line 2 [GsNMethod 67535609345] 35 WASessionContinuation >> value (envId 0) @4 line 5 [GsNMethod 67535610369] 36 WASession >> handleFiltered: (envId 0) @9 line 10 [GsNMethod 67535603201] 37 [] in WARequestHandler >> handle: (envId 0) @3 line 4 [GsNMethod 67589540609] 38 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 39 WADynamicVariable class >> use:during: (envId 0) @2 line 4 [GsNMethod 67530848769] 40 [] in WARequestContext >> push:during: (envId 0) @2 line 5 [GsNMethod 67589523201] 41 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 42 WARequestContext >> push:during: (envId 0) @3 line 6 [GsNMethod 67534366721] 43 WARequestHandler >> handle: (envId 0) @2 line 4 [GsNMethod 67534726401] 44 WASession >> handle: (envId 0) @10 line 11 [GsNMethod 67535596545] 45 WARegistry >> dispatch:to: (envId 0) @2 line 5 [GsNMethod 67535205633] 46 WARegistry >> handleKeyed:with: (envId 0) @2 line 5 [GsNMethod 67535202305] 47 WARegistry >> handleFiltered: (envId 0) @33 line 19 [GsNMethod 67535204609] 48 WAApplication >> handleFiltered: (envId 0) @10 line 8 [GsNMethod 67535584257] 49 WARequestFilter >> handleFiltered: (envId 0) @3 line 4 [GsNMethod 67534418177] 50 [] in WAExceptionFilter >> handleFiltered: (envId 0) @2 line 7 [GsNMethod 67615813633] 51 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 52 WADynamicVariable class >> use:during: (envId 0) @2 line 4 [GsNMethod 67530848769] 53 [] in WAExceptionFilter >> handleFiltered: (envId 0) @2 line 6 [GsNMethod 67589083905] 54 ExecBlock >> onException:do: (envId 0) @2 line 66 [GsNMethod 125587969] 55 ExecBlock >> on:do: (envId 0) @5 line 47 [GsNMethod 125587457] 56 WAExceptionHandler >> handleExceptionsDuring: (envId 0) @2 line 3 [GsNMethod 67533463809] 57 WAExceptionFilter >> handleFiltered: (envId 0) @5 line 4 [GsNMethod 67530878977] 58 [] in WARequestHandler >> handle: (envId 0) @3 line 4 [GsNMethod 67589540609] 59 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 60 WADynamicVariable class >> use:during: (envId 0) @2 line 4 [GsNMethod 67530848769] 61 [] in WARequestContext >> push:during: (envId 0) @2 line 5 [GsNMethod 67589523201] 62 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 63 WARequestContext >> push:during: (envId 0) @3 line 6 [GsNMethod 67534366721] 64 WARequestHandler >> handle: (envId 0) @2 line 4 [GsNMethod 67534726401] 65 WADispatcher >> handleFiltered:named: (envId 0) @3 line 5 [GsNMethod 67530834177] 66 WADispatcher >> handleFiltered: (envId 0) @8 line 6 [GsNMethod 67530832129] 67 [] in WARequestHandler >> handle: (envId 0) @3 line 4 [GsNMethod 67589540609] 68 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 69 WADynamicVariable class >> use:during: (envId 0) @2 line 4 [GsNMethod 67530848769] 70 [] in WARequestContext >> push:during: (envId 0) @2 line 5 [GsNMethod 67589523201] 71 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 72 WARequestContext >> push:during: (envId 0) @3 line 6 [GsNMethod 67534366721] 73 WARequestHandler >> handle: (envId 0) @2 line 4 [GsNMethod 67534726401] 74 [] in WAServerAdaptor >> handleRequest: (envId 0) @3 line 4 [GsNMethod 67589601025] 75 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 76 WAServerAdaptor >> handleRequest: (envId 0) @2 line 5 [GsNMethod 67535027713] 77 [] in WAGsZincStreamingAdaptor >> handleRequestCheckingLock: (envId 0) @2 line 11 [GsNMethod 67655441409] 78 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 79 [] in WAGsZincStreamingAdaptor >> handleRequestCheckingLock: (envId 0) @13 line 12 [GsNMethod 67650054401] 80 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 81 TransientRecursionLock >> critical: (envId 0) @11 line 12 [GsNMethod 67077824257] 82 WAGsZincStreamingAdaptor >> handleRequestCheckingLock: (envId 0) @4 line 5 [GsNMethod 67632024577] 83 WAGsZincStreamingAdaptor >> handleRequest: (envId 0) @5 line 5 [GsNMethod 67664767745] 84 WAServerAdaptor >> handle: (envId 0) @2 line 4 [GsNMethod 67535020801] 85 [] in WAServerAdaptor >> process: (envId 0) @2 line 6 [GsNMethod 67589604353] 86 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 87 WAServerAdaptor >> process: (envId 0) @4 line 7 [GsNMethod 67535024897] 88 [] in ZnZincStreamingServerAdaptor >> process: (envId 0) @4 line 8 [GsNMethod 67650065409] 89 ZnDeferredResponse >> writeOn: (envId 0) @2 line 2 [GsNMethod 67632026369] 90 ZnSingleThreadedServer >> writeResponse:on: (envId 0) @2 line 4 [GsNMethod 67548271873] 91 [] in ZnMultiThreadedServer >> writeResponseSafely:on: (envId 0) @2 line 4 [GsNMethod 84589534721] 92 ExecBlock >> on:do: (envId 0) @3 line 42 [GsNMethod 125587457] 93 ZnMultiThreadedServer >> writeResponseSafely:on: (envId 0) @3 line 5 [GsNMethod 84589718785] 94 ZnMultiThreadedServer >> executeRequestResponseLoopOn: (envId 0) @11 line 11 [GsNMethod 67548246273] 95 [] in ZnMultiThreadedServer >> serveConnectionsOn: (envId 0) @2 line 9 [GsNMethod 67629683969] 96 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 97 [] in ZnMultiThreadedServer >> serveConnectionsOn: (envId 0) @2 line 10 [GsNMethod 67615763457] 98 [] in ExecBlock >> ifCurtailed: (envId 0) @2 line 6 [GsNMethod 67381020929] 99 ExecBlock >> ensure: (envId 0) @2 line 12 [GsNMethod 125587201] 100 ExecBlock >> ifCurtailed: (envId 0) @3 line 8 [GsNMethod 67374152193] 101 [] in ZnMultiThreadedServer >> serveConnectionsOn: (envId 0) @2 line 11 [GsNMethod 67591244545] 102 GsProcess >> _start (envId 0) @7 line 16 [GsNMethod 130094081] 103 <Reenter marker> -- Ken TreisMiriam Technologies, Inc. (866) 652-2040 x221 On Jul 17, 2014, at 12:04 PM, Ken Treis <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Look at me, I'm using the list like a blog. Sorry about that.
To my untrained eye, it looks like this is an issue with switching between native and interpreted code. If we saved continuations with native code and then tried to run them non-native, that'd be a plausible explanation for this error. For now, I've re-enabled native code and expired all sessions by sending WAApplication>>clear. So far things seem OK. -- Ken Treis Miriam Technologies, Inc. (866) 652-2040 x221 On Jul 17, 2014, at 12:16 PM, Ken Treis <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Ken, Sorry for the late reply ... I've been busy elsewhere ... Our vm/continuations guy is on vacation this week, so I'll forward this information to him and see what he has to say when he's back ...
You might want to think about upgrading to 3.1.0.6, because there were quite a few bugs fixed in 3.1.0.5. With that said, I don't know if your particular problem has been addressed or not ...
Dale On Thu, Jul 17, 2014 at 12:41 PM, Ken Treis <[hidden email]> wrote:
_______________________________________________ Glass mailing list [hidden email] http://lists.gemtalksystems.com/mailman/listinfo/glass |
Free forum by Nabble | Edit this page |