Gembuilder tutorial documentation

Are you getting a lot of commit conflicts? I repeat that I think the two errors are unrelated ...

The commit conflict handling code is way after the fastCGI read code in the scheme of things...and unless your web server (nginx? lightppd?) is forwarding non-FastCGI traffic to the same ports that it is sending FASTCgi traffic the source of the bad FASTCgi almost has to be your server machine.

Look at GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock: where the commit conflict logic is near the bottom of the method ... when a commit conflict is encountered the http request is retried (without engaging the fastCGI logic)... so it really isn't possible for the two different errors to be related...I don't think...

The commit conflicts are logged in GRGemStonePlatform>>saveLogEntry:shouldCommit: and you could add a timestamp to the server log output ... you've probably also got objectlog entries for the commit conflicts (not for the unreportable errors) and object log entries have time stamps ...

----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Monday, July 9, 2012 5:29:41 PM
| Subject: Re: [GS/SS Beta] Far worse problems…
|
|
| On Jul 9, 2012, at 8:26 PM, Dale Henrichs wrote:
|
| > Is there a timestamp on the commit conflict log entry?
| >
|
| No timestamp on the commit conflict, just from the unrepeatable
| errors around it….
|
|
| -----------  Unreportable ERROR Encountered:
| 2012-07-08T06:20:56.84343409538269-07:00InterpreterError 2089: An
| index range was specified for a sequenceable collection with the
| starting index <1> greater than the ending index <0>.-----------
| retryFailure
| Read-Write Conflicts...
| Write-Write Conflicts...
| Write-Dependency Conflicts...
| Write-ReadLock Conflicts...
| Write-WriteLock Conflicts...
| Rc-Write-Write Conflicts...
|     440885505
| Synchronized-Commit Conflicts...
| ----------- Commit failure - retrying LOG ENTRY:
| aSymbolDictionary-----------
| -----------  Unreportable ERROR Encountered:
| 2012-07-08T06:22:56.88573288917542-07:00InterpreterError 2089: An
| index range was specified for a sequenceable collection with the
| starting index <1> greater than the ending index <0>.--
|
|
|
|
|
| > Occasional commit conflicts are not actually cause for alarm ...
| > according the log, the commit conflict was retried ... without
| > further errors about the the retry limit being exceeded I'm pretty
| > sure that the retry succeeded ...
| >
| > The "Unreportable ERROR" is very liklye coming from some process
| > (most likely on your box) trying to connect to the fastCGI ports
| > without actually passing in valid fastCGI …
| >
|
| I'll have to think about this…. not sure what it could be at the
| moment...
|
|   Larry
|
|
|
|
| > Dale
| >
| > ----- Original Message -----
| > | From: "Lawrence Kellogg" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Monday, July 9, 2012 5:22:14 PM
| > | Subject: Re: [GS/SS Beta] Far worse problems…
| > |
| > |
| > | On Jul 9, 2012, at 8:12 PM, Dale Henrichs wrote:
| > |
| > | > Two document handlers with the same document will compare #= so
| > | > two
| > | > different sessions creating document handlers with the same
| > | > document will conflict ...
| > | >
| > | > it would be interesting to see the document contents of the
| > | > WADocument...that might have meaning for you..
| > | >
| > |
| > | Well, that OOP now points a an empty dictionary. I tried to look
| > | at
| > | the contents of the WADocument but it was a garage string, as far
| > | as
| > | I could tell.
| > |
| > |   You know, it only mentions the Rc-Write-Write conflict once and
| > |   then it spews out hundreds of Unreportable errors. Now, I think
| > |   it
| > |   is still generating those errors, with a stack dump, but I
| > |   can't
| > |   figure out anything from the stack dump as to what is going
| > |   wrong.
| > |
| > |   Larry
| > |
| > |
| > |
| > |
| > | > Dale
| > | >
| > | >
| > | > ----- Original Message -----
| > | > | From: "Dale Henrichs" <[hidden email]>
| > | > | To: "GemStone Seaside beta discussion"
| > | > | <[hidden email]>
| > | > | Sent: Monday, July 9, 2012 5:05:18 PM
| > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > |
| > | > | The sessions are locked with object locks so you shouldn't be
| > | > | able to
| > | > | access the same session in two different servers.
| > | > |
| > | > | The Document handler looks suspicious to me ... perhaps you
| > | > | have
| > | > | different sessions trying to add a document handler... this
| > | > | rings
| > | > | a
| > | > | bell but I can't recall the details .. it doesn't seem right
| > | > | that
| > | > | document handlers are being stored in the session dictionary
| > | > | ...
| > | > |
| > | > | Dale
| > | > |
| > | > | ----- Original Message -----
| > | > | | From: "Lawrence Kellogg" <[hidden email]>
| > | > | | To: "GemStone Seaside beta discussion"
| > | > | | <[hidden email]>
| > | > | | Sent: Monday, July 9, 2012 4:57:33 PM
| > | > | | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | | On Jul 9, 2012, at 7:51 PM, Dale Henrichs wrote:
| > | > | |
| > | > | |
| > | > | |
| > | > | | Well let' see...
| > | > | |
| > | > | | 440885505 is the oop of the object that you are getting the
| > | > | | Rc-Write-Write Conflicts on. You can use:
| > | > | |
| > | > | | Object _objectForOop: 440885505
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | | Thanks, Dale.
| > | > | |
| > | > | |
| > | > | | So, this object is a RcKeyValueDiction with two keys
| > | > | |
| > | > | |
| > | > | | aWASession and a aWADocumentHandler.
| > | > | |
| > | > | |
| > | > | | It does not seem like my code, but perhaps it has something
| > | > | | to
| > | > | | do
| > | > | | with the session stuff.
| > | > | | Out of all the code I wrote, I didn't expect this conflict.
| > | > | | ;-)
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | | Any chance that this call to createCache in my session
| > | > | | class is
| > | > | | causing the problem?
| > | > | |
| > | > | |
| > | > | |
| > | > | | createCache
| > | > | | "To configure the continuation cache you must currently
| > | > | | subclass
| > | > | | WASession and override this method."
| > | > | | | policy |
| > | > | | policy := WARcLastAccessExpiryPolicy new.
| > | > | | policy configuration at: #cacheTimeout put: 3600.
| > | > | | ^ WACache new
| > | > | | setExpiryPolicy: policy;
| > | > | | yourself.
| > | > | |
| > | > | |
| > | > | | Larry
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | |
| > | > | | to inspect the object ... the fact that it is an Rc error
| > | > | | means
| > | > | | that
| > | > | | the type of conflict is one that is not resolvable by retry
| > | > | | ...
| > | > | | for
| > | > | | an Rc Dictionary that means two sessions are using the
| > | > | | adding/removing the same key ... For an Rc set that means
| > | > | | two
| > | > | | sessions are adding removing the same object ...
| > | > | |
| > | > | | Inspect the object and find out ...
| > | > | |
| > | > | |
| > | > | | ... I'll have more in a follow on mail...
| > | > | |
| > | > | | Dale
| > | > | |
| > | > | | ----- Original Message -----
| > | > | | | From: "Lawrence Kellogg" < [hidden email] >
| > | > | | | To: "GemStone Seaside beta discussion" <
| > | > | | | [hidden email]
| > | > | | | >
| > | > | | | Sent: Monday, July 9, 2012 4:42:13 PM
| > | > | | | Subject: [GS/SS Beta] Far worse problems…
| > | > | | |
| > | > | | | Hello,
| > | > | | | Actually, I have far worse problems than just hiding
| > | > | | | changes
| > | > | | | in
| > | > | | | the
| > | > | | | Patch Browser. I just noticed that the log files for my
| > | > | | | Gems
| > | > | | | are
| > | > | | | spewing out this:
| > | > | | |
| > | > | | |
| > | > | | |
| > | > | | |
| > | > | | | etryFailure
| > | > | | | Read-Write Conflicts...
| > | > | | | Write-Write Conflicts...
| > | > | | | Write-Dependency Conflicts...
| > | > | | | Write-ReadLock Conflicts...
| > | > | | | Write-WriteLock Conflicts...
| > | > | | | Rc-Write-Write Conflicts...
| > | > | | | 440885505
| > | > | | | Synchronized-Commit Conflicts...
| > | > | | | ----------- Commit failure - retrying LOG ENTRY:
| > | > | | | aSymbolDictionary-----------
| > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | | | 2012-07-08T18:37:39.67828106880188-07:00InterpreterError
| > | > | | | 2089: An
| > | > | | | index range was specified for a sequenceable collection
| > | > | | | with
| > | > | | | the
| > | > | | | starting index <1> greater than the ending index
| > | > | | | <0>.-----------
| > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | | | 2012-07-08T18:39:39.71508598327637-07:00InterpreterError
| > | > | | | 2089: An
| > | > | | | index range was specified for a sequenceable collection
| > | > | | | with
| > | > | | | the
| > | > | | | starting index <1> greater than the ending index
| > | > | | | <0>.-----------
| > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | | | 2012-07-08T18:41:39.78475308418274-07:00InterpreterError
| > | > | | | 2089: An
| > | > | | | index range was specified for a sequenceable collection
| > | > | | | with
| > | > | | | the
| > | > | | | starting index <1> greater than the ending index
| > | > | | | <0>.-----------
| > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | | | 2012-07-08T18:43:39.84294891357422-07:00InterpreterError
| > | > | | | 2089: An
| > | > | | | index range was specified for a sequenceable collection
| > | > | | | with
| > | > | | | the
| > | > | | | starting index <1> greater than the ending index
| > | > | | | <0>.-----------
| > | > | | |
| > | > | | | on and on and on...
| > | > | | |
| > | > | | | Help! What do I do? I took care of a walk back I add and
| > | > | | | actually
| > | > | | | have no idea what sequenceable collection is causing this
| > | > | | | problem.
| > | > | | | My system is up and running fine but all of these errors
| > | > | | | are
| > | > | | | being
| > | > | | | thrown.
| > | > | | |
| > | > | | |
| > | > | | | Larry
| > | > | |
| > | > | |
| > | > |
| > |
| > |
|
|

On Jul 9, 2012, at 8:42 PM, Dale Henrichs wrote:

> Are you getting a lot of commit conflicts? I repeat that I think the two errors are unrelated ...
>

  You're right, I'm not getting a lot of commit conflicts so I think these problems are unrelated.


> The commit conflict handling code is way after the fastCGI read code in the scheme of things...and unless your web server (nginx? lightppd?) is forwarding non-FastCGI traffic to the same ports that it is sending FASTCgi traffic the source of the bad FASTCgi almost has to be your server machine.
>

  Could a crashed Gem still be sending bad FastCGI packets? I have a monit process in place and I think it is starting multiple Gems on 9001, 9002, and 9003.
At least, I see topaz sessions when I do a ps-ef.


> Look at GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock: where the commit conflict logic is near the bottom of the method ... when a commit conflict is encountered the http request is retried (without engaging the fastCGI logic)... so it really isn't possible for the two different errors to be related...I don't think...
>
> The commit conflicts are logged in GRGemStonePlatform>>saveLogEntry:shouldCommit: and you could add a timestamp to the server log output ... you've probably also got objectlog entries for the commit conflicts (not for the unreportable errors) and object log entries have time stamps …
>

  Ok, thanks….

Lawrence,

If you've got multiple gems trying to listen on a port, only one gem can listen on a port at any one time, so I think that the the gem should fail when it tries to open the port.

The gems are passive as they listen on the port so they can't really affect other gems ...

Do you have a process that periodically "pings" a gem on it's http port to see if it is still alive ... I suspect a process like that has gone haywire and is pinging away on the gems ...

Dale

----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Monday, July 9, 2012 6:30:21 PM
| Subject: Re: [GS/SS Beta] Far worse problems…
|
|
| On Jul 9, 2012, at 8:42 PM, Dale Henrichs wrote:
|
| > Are you getting a lot of commit conflicts? I repeat that I think
| > the two errors are unrelated ...
| >
|
|   You're right, I'm not getting a lot of commit conflicts so I think
|   these problems are unrelated.
|
|
| > The commit conflict handling code is way after the fastCGI read
| > code in the scheme of things...and unless your web server (nginx?
| > lightppd?) is forwarding non-FastCGI traffic to the same ports
| > that it is sending FASTCgi traffic the source of the bad FASTCgi
| > almost has to be your server machine.
| >
|
|   Could a crashed Gem still be sending bad FastCGI packets? I have a
|   monit process in place and I think it is starting multiple Gems on
|   9001, 9002, and 9003.
| At least, I see topaz sessions when I do a ps-ef.
|
|
| > Look at
| > GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock:
| > where the commit conflict logic is near the bottom of the method
| > ... when a commit conflict is encountered the http request is
| > retried (without engaging the fastCGI logic)... so it really isn't
| > possible for the two different errors to be related...I don't
| > think...
| >
| > The commit conflicts are logged in
| > GRGemStonePlatform>>saveLogEntry:shouldCommit: and you could add a
| > timestamp to the server log output ... you've probably also got
| > objectlog entries for the commit conflicts (not for the
| > unreportable errors) and object log entries have time stamps …
| >
|
|   Ok, thanks….
|
| > ----- Original Message -----
| > | From: "Lawrence Kellogg" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Monday, July 9, 2012 5:29:41 PM
| > | Subject: Re: [GS/SS Beta] Far worse problems…
| > |
| > |
| > | On Jul 9, 2012, at 8:26 PM, Dale Henrichs wrote:
| > |
| > | > Is there a timestamp on the commit conflict log entry?
| > | >
| > |
| > | No timestamp on the commit conflict, just from the unrepeatable
| > | errors around it….
| > |
| > |
| > | -----------  Unreportable ERROR Encountered:
| > | 2012-07-08T06:20:56.84343409538269-07:00InterpreterError 2089: An
| > | index range was specified for a sequenceable collection with the
| > | starting index <1> greater than the ending index <0>.-----------
| > | retryFailure
| > | Read-Write Conflicts...
| > | Write-Write Conflicts...
| > | Write-Dependency Conflicts...
| > | Write-ReadLock Conflicts...
| > | Write-WriteLock Conflicts...
| > | Rc-Write-Write Conflicts...
| > |     440885505
| > | Synchronized-Commit Conflicts...
| > | ----------- Commit failure - retrying LOG ENTRY:
| > | aSymbolDictionary-----------
| > | -----------  Unreportable ERROR Encountered:
| > | 2012-07-08T06:22:56.88573288917542-07:00InterpreterError 2089: An
| > | index range was specified for a sequenceable collection with the
| > | starting index <1> greater than the ending index <0>.--
| > |
| > |
| > |
| > |
| > |
| > | > Occasional commit conflicts are not actually cause for alarm
| > | > ...
| > | > according the log, the commit conflict was retried ... without
| > | > further errors about the the retry limit being exceeded I'm
| > | > pretty
| > | > sure that the retry succeeded ...
| > | >
| > | > The "Unreportable ERROR" is very liklye coming from some
| > | > process
| > | > (most likely on your box) trying to connect to the fastCGI
| > | > ports
| > | > without actually passing in valid fastCGI …
| > | >
| > |
| > | I'll have to think about this…. not sure what it could be at the
| > | moment...
| > |
| > |   Larry
| > |
| > |
| > |
| > |
| > | > Dale
| > | >
| > | > ----- Original Message -----
| > | > | From: "Lawrence Kellogg" <[hidden email]>
| > | > | To: "GemStone Seaside beta discussion"
| > | > | <[hidden email]>
| > | > | Sent: Monday, July 9, 2012 5:22:14 PM
| > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > |
| > | > |
| > | > | On Jul 9, 2012, at 8:12 PM, Dale Henrichs wrote:
| > | > |
| > | > | > Two document handlers with the same document will compare
| > | > | > #= so
| > | > | > two
| > | > | > different sessions creating document handlers with the same
| > | > | > document will conflict ...
| > | > | >
| > | > | > it would be interesting to see the document contents of the
| > | > | > WADocument...that might have meaning for you..
| > | > | >
| > | > |
| > | > | Well, that OOP now points a an empty dictionary. I tried to
| > | > | look
| > | > | at
| > | > | the contents of the WADocument but it was a garage string, as
| > | > | far
| > | > | as
| > | > | I could tell.
| > | > |
| > | > |   You know, it only mentions the Rc-Write-Write conflict once
| > | > |   and
| > | > |   then it spews out hundreds of Unreportable errors. Now, I
| > | > |   think
| > | > |   it
| > | > |   is still generating those errors, with a stack dump, but I
| > | > |   can't
| > | > |   figure out anything from the stack dump as to what is going
| > | > |   wrong.
| > | > |
| > | > |   Larry
| > | > |
| > | > |
| > | > |
| > | > |
| > | > | > Dale
| > | > | >
| > | > | >
| > | > | > ----- Original Message -----
| > | > | > | From: "Dale Henrichs" <[hidden email]>
| > | > | > | To: "GemStone Seaside beta discussion"
| > | > | > | <[hidden email]>
| > | > | > | Sent: Monday, July 9, 2012 5:05:18 PM
| > | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > | > |
| > | > | > | The sessions are locked with object locks so you
| > | > | > | shouldn't be
| > | > | > | able to
| > | > | > | access the same session in two different servers.
| > | > | > |
| > | > | > | The Document handler looks suspicious to me ... perhaps
| > | > | > | you
| > | > | > | have
| > | > | > | different sessions trying to add a document handler...
| > | > | > | this
| > | > | > | rings
| > | > | > | a
| > | > | > | bell but I can't recall the details .. it doesn't seem
| > | > | > | right
| > | > | > | that
| > | > | > | document handlers are being stored in the session
| > | > | > | dictionary
| > | > | > | ...
| > | > | > |
| > | > | > | Dale
| > | > | > |
| > | > | > | ----- Original Message -----
| > | > | > | | From: "Lawrence Kellogg" <[hidden email]>
| > | > | > | | To: "GemStone Seaside beta discussion"
| > | > | > | | <[hidden email]>
| > | > | > | | Sent: Monday, July 9, 2012 4:57:33 PM
| > | > | > | | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | | On Jul 9, 2012, at 7:51 PM, Dale Henrichs wrote:
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | | Well let' see...
| > | > | > | |
| > | > | > | | 440885505 is the oop of the object that you are getting
| > | > | > | | the
| > | > | > | | Rc-Write-Write Conflicts on. You can use:
| > | > | > | |
| > | > | > | | Object _objectForOop: 440885505
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | | Thanks, Dale.
| > | > | > | |
| > | > | > | |
| > | > | > | | So, this object is a RcKeyValueDiction with two keys
| > | > | > | |
| > | > | > | |
| > | > | > | | aWASession and a aWADocumentHandler.
| > | > | > | |
| > | > | > | |
| > | > | > | | It does not seem like my code, but perhaps it has
| > | > | > | | something
| > | > | > | | to
| > | > | > | | do
| > | > | > | | with the session stuff.
| > | > | > | | Out of all the code I wrote, I didn't expect this
| > | > | > | | conflict.
| > | > | > | | ;-)
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | | Any chance that this call to createCache in my session
| > | > | > | | class is
| > | > | > | | causing the problem?
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | | createCache
| > | > | > | | "To configure the continuation cache you must currently
| > | > | > | | subclass
| > | > | > | | WASession and override this method."
| > | > | > | | | policy |
| > | > | > | | policy := WARcLastAccessExpiryPolicy new.
| > | > | > | | policy configuration at: #cacheTimeout put: 3600.
| > | > | > | | ^ WACache new
| > | > | > | | setExpiryPolicy: policy;
| > | > | > | | yourself.
| > | > | > | |
| > | > | > | |
| > | > | > | | Larry
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | |
| > | > | > | | to inspect the object ... the fact that it is an Rc
| > | > | > | | error
| > | > | > | | means
| > | > | > | | that
| > | > | > | | the type of conflict is one that is not resolvable by
| > | > | > | | retry
| > | > | > | | ...
| > | > | > | | for
| > | > | > | | an Rc Dictionary that means two sessions are using the
| > | > | > | | adding/removing the same key ... For an Rc set that
| > | > | > | | means
| > | > | > | | two
| > | > | > | | sessions are adding removing the same object ...
| > | > | > | |
| > | > | > | | Inspect the object and find out ...
| > | > | > | |
| > | > | > | |
| > | > | > | | ... I'll have more in a follow on mail...
| > | > | > | |
| > | > | > | | Dale
| > | > | > | |
| > | > | > | | ----- Original Message -----
| > | > | > | | | From: "Lawrence Kellogg" < [hidden email] >
| > | > | > | | | To: "GemStone Seaside beta discussion" <
| > | > | > | | | [hidden email]
| > | > | > | | | >
| > | > | > | | | Sent: Monday, July 9, 2012 4:42:13 PM
| > | > | > | | | Subject: [GS/SS Beta] Far worse problems…
| > | > | > | | |
| > | > | > | | | Hello,
| > | > | > | | | Actually, I have far worse problems than just hiding
| > | > | > | | | changes
| > | > | > | | | in
| > | > | > | | | the
| > | > | > | | | Patch Browser. I just noticed that the log files for
| > | > | > | | | my
| > | > | > | | | Gems
| > | > | > | | | are
| > | > | > | | | spewing out this:
| > | > | > | | |
| > | > | > | | |
| > | > | > | | |
| > | > | > | | |
| > | > | > | | | etryFailure
| > | > | > | | | Read-Write Conflicts...
| > | > | > | | | Write-Write Conflicts...
| > | > | > | | | Write-Dependency Conflicts...
| > | > | > | | | Write-ReadLock Conflicts...
| > | > | > | | | Write-WriteLock Conflicts...
| > | > | > | | | Rc-Write-Write Conflicts...
| > | > | > | | | 440885505
| > | > | > | | | Synchronized-Commit Conflicts...
| > | > | > | | | ----------- Commit failure - retrying LOG ENTRY:
| > | > | > | | | aSymbolDictionary-----------
| > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | | | 2012-07-08T18:37:39.67828106880188-07:00InterpreterError
| > | > | > | | | 2089: An
| > | > | > | | | index range was specified for a sequenceable
| > | > | > | | | collection
| > | > | > | | | with
| > | > | > | | | the
| > | > | > | | | starting index <1> greater than the ending index
| > | > | > | | | <0>.-----------
| > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | | | 2012-07-08T18:39:39.71508598327637-07:00InterpreterError
| > | > | > | | | 2089: An
| > | > | > | | | index range was specified for a sequenceable
| > | > | > | | | collection
| > | > | > | | | with
| > | > | > | | | the
| > | > | > | | | starting index <1> greater than the ending index
| > | > | > | | | <0>.-----------
| > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | | | 2012-07-08T18:41:39.78475308418274-07:00InterpreterError
| > | > | > | | | 2089: An
| > | > | > | | | index range was specified for a sequenceable
| > | > | > | | | collection
| > | > | > | | | with
| > | > | > | | | the
| > | > | > | | | starting index <1> greater than the ending index
| > | > | > | | | <0>.-----------
| > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | | | 2012-07-08T18:43:39.84294891357422-07:00InterpreterError
| > | > | > | | | 2089: An
| > | > | > | | | index range was specified for a sequenceable
| > | > | > | | | collection
| > | > | > | | | with
| > | > | > | | | the
| > | > | > | | | starting index <1> greater than the ending index
| > | > | > | | | <0>.-----------
| > | > | > | | |
| > | > | > | | | on and on and on...
| > | > | > | | |
| > | > | > | | | Help! What do I do? I took care of a walk back I add
| > | > | > | | | and
| > | > | > | | | actually
| > | > | > | | | have no idea what sequenceable collection is causing
| > | > | > | | | this
| > | > | > | | | problem.
| > | > | > | | | My system is up and running fine but all of these
| > | > | > | | | errors
| > | > | > | | | are
| > | > | > | | | being
| > | > | > | | | thrown.
| > | > | > | | |
| > | > | > | | |
| > | > | > | | | Larry
| > | > | > | |
| > | > | > | |
| > | > | > |
| > | > |
| > | > |
| > |
| > |
|
|

On Jul 9, 2012, at 11:11 PM, Dale Henrichs wrote:

> Lawrence,
>
> If you've got multiple gems trying to listen on a port, only one gem can listen on a port at any one time, so I think that the the gem should fail when it tries to open the port.
>
> The gems are passive as they listen on the port so they can't really affect other gems ...
>
> Do you have a process that periodically "pings" a gem on it's http port to see if it is still alive ... I suspect a process like that has gone haywire and is pinging away on the gems …
>

  I suppose it could be monit. I'll kill it and see if I still get the errors. I can't think of any other process that would be pinging the Gems repeatedly. I killed all of the Gems last night, one by one, and watched monit bring them back up cleanly. At first, I thought the errors were gone, but I see this morning that I am still getting the same Unreported errors.
 

Larry

On Jul 9, 2012, at 11:11 PM, Dale Henrichs wrote:

> Lawrence,
>
> If you've got multiple gems trying to listen on a port, only one gem can listen on a port at any one time, so I think that the the gem should fail when it tries to open the port.
>
> The gems are passive as they listen on the port so they can't really affect other gems ...
>
> Do you have a process that periodically "pings" a gem on it's http port to see if it is still alive ... I suspect a process like that has gone haywire and is pinging away on the gems …
>

Well, I went back through monit, it's been a while since I have looked at it, as I think it has been working fine for months.
Anyway, I see this:

                # Empty FastCGI request
                if failed port 9002
                  # Send FastCGI packet: version 1 (0x01), cmd FCGI_GET_VALUES (0x09)
                  # padding 8 bytes (0x08), followed by 8xNULLs padding
                  send "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
                  # Expect FastCGI packet: version 1 (0x01), resp FCGI_GET_VALUES_RESULT (0x0A)
                  expect "\0x01\0x0A"
                  timeout 20 seconds
                then restart

in the monit.conf file, so perhaps this packet to fastcgi is wrong for some reason. I never had a problem before….

Any idea if this is valid or invalid? For now, I have shut down monit to see if the unreported error problem disappears.

Larry

I'm not familiar with Monit..

----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, July 10, 2012 8:36:34 AM
| Subject: Re: [GS/SS Beta] Far worse problems…
|
|
| On Jul 9, 2012, at 11:11 PM, Dale Henrichs wrote:
|
| > Lawrence,
| >
| > If you've got multiple gems trying to listen on a port, only one
| > gem can listen on a port at any one time, so I think that the the
| > gem should fail when it tries to open the port.
| >
| > The gems are passive as they listen on the port so they can't
| > really affect other gems ...
| >
| > Do you have a process that periodically "pings" a gem on it's http
| > port to see if it is still alive ... I suspect a process like that
| > has gone haywire and is pinging away on the gems …
| >
|
| Well, I went back through monit, it's been a while since I have
| looked at it, as I think it has been working fine for months.
| Anyway, I see this:
|
|                 # Empty FastCGI request
|                 if failed port 9002
|                   # Send FastCGI packet: version 1 (0x01), cmd
|                   FCGI_GET_VALUES (0x09)
|                   # padding 8 bytes (0x08), followed by 8xNULLs
|                   padding
|                   send
|                   "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
|                   # Expect FastCGI packet: version 1 (0x01), resp
|                   FCGI_GET_VALUES_RESULT (0x0A)
|                   expect "\0x01\0x0A"
|                   timeout 20 seconds
|                 then restart
|
| in the monit.conf file, so perhaps this packet to fastcgi is wrong
| for some reason. I never had a problem before….
|
| Any idea if this is valid or invalid? For now, I have shut down monit
| to see if the unreported error problem disappears.
|
| Larry
|
|
| > Dale
| >
| > ----- Original Message -----
| > | From: "Lawrence Kellogg" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Monday, July 9, 2012 6:30:21 PM
| > | Subject: Re: [GS/SS Beta] Far worse problems…
| > |
| > |
| > | On Jul 9, 2012, at 8:42 PM, Dale Henrichs wrote:
| > |
| > | > Are you getting a lot of commit conflicts? I repeat that I
| > | > think
| > | > the two errors are unrelated ...
| > | >
| > |
| > |   You're right, I'm not getting a lot of commit conflicts so I
| > |   think
| > |   these problems are unrelated.
| > |
| > |
| > | > The commit conflict handling code is way after the fastCGI read
| > | > code in the scheme of things...and unless your web server
| > | > (nginx?
| > | > lightppd?) is forwarding non-FastCGI traffic to the same ports
| > | > that it is sending FASTCgi traffic the source of the bad
| > | > FASTCgi
| > | > almost has to be your server machine.
| > | >
| > |
| > |   Could a crashed Gem still be sending bad FastCGI packets? I
| > |   have a
| > |   monit process in place and I think it is starting multiple Gems
| > |   on
| > |   9001, 9002, and 9003.
| > | At least, I see topaz sessions when I do a ps-ef.
| > |
| > |
| > | > Look at
| > | > GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock:
| > | > where the commit conflict logic is near the bottom of the
| > | > method
| > | > ... when a commit conflict is encountered the http request is
| > | > retried (without engaging the fastCGI logic)... so it really
| > | > isn't
| > | > possible for the two different errors to be related...I don't
| > | > think...
| > | >
| > | > The commit conflicts are logged in
| > | > GRGemStonePlatform>>saveLogEntry:shouldCommit: and you could
| > | > add a
| > | > timestamp to the server log output ... you've probably also got
| > | > objectlog entries for the commit conflicts (not for the
| > | > unreportable errors) and object log entries have time stamps …
| > | >
| > |
| > |   Ok, thanks….
| > |
| > | > ----- Original Message -----
| > | > | From: "Lawrence Kellogg" <[hidden email]>
| > | > | To: "GemStone Seaside beta discussion"
| > | > | <[hidden email]>
| > | > | Sent: Monday, July 9, 2012 5:29:41 PM
| > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > |
| > | > |
| > | > | On Jul 9, 2012, at 8:26 PM, Dale Henrichs wrote:
| > | > |
| > | > | > Is there a timestamp on the commit conflict log entry?
| > | > | >
| > | > |
| > | > | No timestamp on the commit conflict, just from the
| > | > | unrepeatable
| > | > | errors around it….
| > | > |
| > | > |
| > | > | -----------  Unreportable ERROR Encountered:
| > | > | 2012-07-08T06:20:56.84343409538269-07:00InterpreterError
| > | > | 2089: An
| > | > | index range was specified for a sequenceable collection with
| > | > | the
| > | > | starting index <1> greater than the ending index
| > | > | <0>.-----------
| > | > | retryFailure
| > | > | Read-Write Conflicts...
| > | > | Write-Write Conflicts...
| > | > | Write-Dependency Conflicts...
| > | > | Write-ReadLock Conflicts...
| > | > | Write-WriteLock Conflicts...
| > | > | Rc-Write-Write Conflicts...
| > | > |     440885505
| > | > | Synchronized-Commit Conflicts...
| > | > | ----------- Commit failure - retrying LOG ENTRY:
| > | > | aSymbolDictionary-----------
| > | > | -----------  Unreportable ERROR Encountered:
| > | > | 2012-07-08T06:22:56.88573288917542-07:00InterpreterError
| > | > | 2089: An
| > | > | index range was specified for a sequenceable collection with
| > | > | the
| > | > | starting index <1> greater than the ending index <0>.--
| > | > |
| > | > |
| > | > |
| > | > |
| > | > |
| > | > | > Occasional commit conflicts are not actually cause for
| > | > | > alarm
| > | > | > ...
| > | > | > according the log, the commit conflict was retried ...
| > | > | > without
| > | > | > further errors about the the retry limit being exceeded I'm
| > | > | > pretty
| > | > | > sure that the retry succeeded ...
| > | > | >
| > | > | > The "Unreportable ERROR" is very liklye coming from some
| > | > | > process
| > | > | > (most likely on your box) trying to connect to the fastCGI
| > | > | > ports
| > | > | > without actually passing in valid fastCGI …
| > | > | >
| > | > |
| > | > | I'll have to think about this…. not sure what it could be at
| > | > | the
| > | > | moment...
| > | > |
| > | > |   Larry
| > | > |
| > | > |
| > | > |
| > | > |
| > | > | > Dale
| > | > | >
| > | > | > ----- Original Message -----
| > | > | > | From: "Lawrence Kellogg" <[hidden email]>
| > | > | > | To: "GemStone Seaside beta discussion"
| > | > | > | <[hidden email]>
| > | > | > | Sent: Monday, July 9, 2012 5:22:14 PM
| > | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > | > |
| > | > | > |
| > | > | > | On Jul 9, 2012, at 8:12 PM, Dale Henrichs wrote:
| > | > | > |
| > | > | > | > Two document handlers with the same document will
| > | > | > | > compare
| > | > | > | > #= so
| > | > | > | > two
| > | > | > | > different sessions creating document handlers with the
| > | > | > | > same
| > | > | > | > document will conflict ...
| > | > | > | >
| > | > | > | > it would be interesting to see the document contents of
| > | > | > | > the
| > | > | > | > WADocument...that might have meaning for you..
| > | > | > | >
| > | > | > |
| > | > | > | Well, that OOP now points a an empty dictionary. I tried
| > | > | > | to
| > | > | > | look
| > | > | > | at
| > | > | > | the contents of the WADocument but it was a garage
| > | > | > | string, as
| > | > | > | far
| > | > | > | as
| > | > | > | I could tell.
| > | > | > |
| > | > | > |   You know, it only mentions the Rc-Write-Write conflict
| > | > | > |   once
| > | > | > |   and
| > | > | > |   then it spews out hundreds of Unreportable errors. Now,
| > | > | > |   I
| > | > | > |   think
| > | > | > |   it
| > | > | > |   is still generating those errors, with a stack dump,
| > | > | > |   but I
| > | > | > |   can't
| > | > | > |   figure out anything from the stack dump as to what is
| > | > | > |   going
| > | > | > |   wrong.
| > | > | > |
| > | > | > |   Larry
| > | > | > |
| > | > | > |
| > | > | > |
| > | > | > |
| > | > | > | > Dale
| > | > | > | >
| > | > | > | >
| > | > | > | > ----- Original Message -----
| > | > | > | > | From: "Dale Henrichs" <[hidden email]>
| > | > | > | > | To: "GemStone Seaside beta discussion"
| > | > | > | > | <[hidden email]>
| > | > | > | > | Sent: Monday, July 9, 2012 5:05:18 PM
| > | > | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > | > | > |
| > | > | > | > | The sessions are locked with object locks so you
| > | > | > | > | shouldn't be
| > | > | > | > | able to
| > | > | > | > | access the same session in two different servers.
| > | > | > | > |
| > | > | > | > | The Document handler looks suspicious to me ...
| > | > | > | > | perhaps
| > | > | > | > | you
| > | > | > | > | have
| > | > | > | > | different sessions trying to add a document
| > | > | > | > | handler...
| > | > | > | > | this
| > | > | > | > | rings
| > | > | > | > | a
| > | > | > | > | bell but I can't recall the details .. it doesn't
| > | > | > | > | seem
| > | > | > | > | right
| > | > | > | > | that
| > | > | > | > | document handlers are being stored in the session
| > | > | > | > | dictionary
| > | > | > | > | ...
| > | > | > | > |
| > | > | > | > | Dale
| > | > | > | > |
| > | > | > | > | ----- Original Message -----
| > | > | > | > | | From: "Lawrence Kellogg" <[hidden email]>
| > | > | > | > | | To: "GemStone Seaside beta discussion"
| > | > | > | > | | <[hidden email]>
| > | > | > | > | | Sent: Monday, July 9, 2012 4:57:33 PM
| > | > | > | > | | Subject: Re: [GS/SS Beta] Far worse problems…
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | On Jul 9, 2012, at 7:51 PM, Dale Henrichs wrote:
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | Well let' see...
| > | > | > | > | |
| > | > | > | > | | 440885505 is the oop of the object that you are
| > | > | > | > | | getting
| > | > | > | > | | the
| > | > | > | > | | Rc-Write-Write Conflicts on. You can use:
| > | > | > | > | |
| > | > | > | > | | Object _objectForOop: 440885505
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | Thanks, Dale.
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | So, this object is a RcKeyValueDiction with two
| > | > | > | > | | keys
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | aWASession and a aWADocumentHandler.
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | It does not seem like my code, but perhaps it has
| > | > | > | > | | something
| > | > | > | > | | to
| > | > | > | > | | do
| > | > | > | > | | with the session stuff.
| > | > | > | > | | Out of all the code I wrote, I didn't expect this
| > | > | > | > | | conflict.
| > | > | > | > | | ;-)
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | Any chance that this call to createCache in my
| > | > | > | > | | session
| > | > | > | > | | class is
| > | > | > | > | | causing the problem?
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | createCache
| > | > | > | > | | "To configure the continuation cache you must
| > | > | > | > | | currently
| > | > | > | > | | subclass
| > | > | > | > | | WASession and override this method."
| > | > | > | > | | | policy |
| > | > | > | > | | policy := WARcLastAccessExpiryPolicy new.
| > | > | > | > | | policy configuration at: #cacheTimeout put: 3600.
| > | > | > | > | | ^ WACache new
| > | > | > | > | | setExpiryPolicy: policy;
| > | > | > | > | | yourself.
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | Larry
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | to inspect the object ... the fact that it is an Rc
| > | > | > | > | | error
| > | > | > | > | | means
| > | > | > | > | | that
| > | > | > | > | | the type of conflict is one that is not resolvable
| > | > | > | > | | by
| > | > | > | > | | retry
| > | > | > | > | | ...
| > | > | > | > | | for
| > | > | > | > | | an Rc Dictionary that means two sessions are using
| > | > | > | > | | the
| > | > | > | > | | adding/removing the same key ... For an Rc set that
| > | > | > | > | | means
| > | > | > | > | | two
| > | > | > | > | | sessions are adding removing the same object ...
| > | > | > | > | |
| > | > | > | > | | Inspect the object and find out ...
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > | | ... I'll have more in a follow on mail...
| > | > | > | > | |
| > | > | > | > | | Dale
| > | > | > | > | |
| > | > | > | > | | ----- Original Message -----
| > | > | > | > | | | From: "Lawrence Kellogg" < [hidden email] >
| > | > | > | > | | | To: "GemStone Seaside beta discussion" <
| > | > | > | > | | | [hidden email]
| > | > | > | > | | | >
| > | > | > | > | | | Sent: Monday, July 9, 2012 4:42:13 PM
| > | > | > | > | | | Subject: [GS/SS Beta] Far worse problems…
| > | > | > | > | | |
| > | > | > | > | | | Hello,
| > | > | > | > | | | Actually, I have far worse problems than just
| > | > | > | > | | | hiding
| > | > | > | > | | | changes
| > | > | > | > | | | in
| > | > | > | > | | | the
| > | > | > | > | | | Patch Browser. I just noticed that the log files
| > | > | > | > | | | for
| > | > | > | > | | | my
| > | > | > | > | | | Gems
| > | > | > | > | | | are
| > | > | > | > | | | spewing out this:
| > | > | > | > | | |
| > | > | > | > | | |
| > | > | > | > | | |
| > | > | > | > | | |
| > | > | > | > | | | etryFailure
| > | > | > | > | | | Read-Write Conflicts...
| > | > | > | > | | | Write-Write Conflicts...
| > | > | > | > | | | Write-Dependency Conflicts...
| > | > | > | > | | | Write-ReadLock Conflicts...
| > | > | > | > | | | Write-WriteLock Conflicts...
| > | > | > | > | | | Rc-Write-Write Conflicts...
| > | > | > | > | | | 440885505
| > | > | > | > | | | Synchronized-Commit Conflicts...
| > | > | > | > | | | ----------- Commit failure - retrying LOG ENTRY:
| > | > | > | > | | | aSymbolDictionary-----------
| > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | > | | | 2012-07-08T18:37:39.67828106880188-07:00InterpreterError
| > | > | > | > | | | 2089: An
| > | > | > | > | | | index range was specified for a sequenceable
| > | > | > | > | | | collection
| > | > | > | > | | | with
| > | > | > | > | | | the
| > | > | > | > | | | starting index <1> greater than the ending index
| > | > | > | > | | | <0>.-----------
| > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | > | | | 2012-07-08T18:39:39.71508598327637-07:00InterpreterError
| > | > | > | > | | | 2089: An
| > | > | > | > | | | index range was specified for a sequenceable
| > | > | > | > | | | collection
| > | > | > | > | | | with
| > | > | > | > | | | the
| > | > | > | > | | | starting index <1> greater than the ending index
| > | > | > | > | | | <0>.-----------
| > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | > | | | 2012-07-08T18:41:39.78475308418274-07:00InterpreterError
| > | > | > | > | | | 2089: An
| > | > | > | > | | | index range was specified for a sequenceable
| > | > | > | > | | | collection
| > | > | > | > | | | with
| > | > | > | > | | | the
| > | > | > | > | | | starting index <1> greater than the ending index
| > | > | > | > | | | <0>.-----------
| > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| > | > | > | > | | | 2012-07-08T18:43:39.84294891357422-07:00InterpreterError
| > | > | > | > | | | 2089: An
| > | > | > | > | | | index range was specified for a sequenceable
| > | > | > | > | | | collection
| > | > | > | > | | | with
| > | > | > | > | | | the
| > | > | > | > | | | starting index <1> greater than the ending index
| > | > | > | > | | | <0>.-----------
| > | > | > | > | | |
| > | > | > | > | | | on and on and on...
| > | > | > | > | | |
| > | > | > | > | | | Help! What do I do? I took care of a walk back I
| > | > | > | > | | | add
| > | > | > | > | | | and
| > | > | > | > | | | actually
| > | > | > | > | | | have no idea what sequenceable collection is
| > | > | > | > | | | causing
| > | > | > | > | | | this
| > | > | > | > | | | problem.
| > | > | > | > | | | My system is up and running fine but all of these
| > | > | > | > | | | errors
| > | > | > | > | | | are
| > | > | > | > | | | being
| > | > | > | > | | | thrown.
| > | > | > | > | | |
| > | > | > | > | | |
| > | > | > | > | | | Larry
| > | > | > | > | |
| > | > | > | > | |
| > | > | > | > |
| > | > | > |
| > | > | > |
| > | > |
| > | > |
| > |
| > |
|
|

On Jul 10, 2012, at 12:14 PM, Dale Henrichs wrote:

> I'm not familiar with Monit..
>

  That's ok, maybe Johan can comment. He's the guy who sent me down the path of using monit.
It's been working great, except for the sudden appearance of these Unreported errors.

  I restarted all of the Gems again so I can see that they're running fine in the log files, and
I have killed monit. So far, I don't see these walk backs getting written to the logs but it is probably too early to tell.

  Thanks for your help.

  Larry

On Jul 9, 2012, at 7:45 PM, Dale Henrichs wrote:

> Lawrence,
>
> The only way to "suppress the source same but rev changed" is to reload (or revert) the mcz file and get the timestamps from the mcz file set correctly, once you do this you shouldn't have to do it over and over again …
>

  I'm not sure what it means to revert the file. Is there a revert option somewhere? I can't find it.


> I have seen this every once in a while, but I haven't tracked the bug down (a revert seems to clean it up) ...
>
> Do you happen to be using FileTree packages?

  No, sorry, I'm not using FileTree packages.

>
> What kind of repositories are you using (Directory-based or Http) for the affect packages?
>


  I'm using a directory based repository. I've been meaning to work everything into Metacello but I never find the time.



P.S. Ever since I took down 'monit', I have not seen those  walkbacks show up.

----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, July 10, 2012 1:50:43 PM
| Subject: Re: [GS/SS Beta] Any way to hide (source same but rev changed) methods?
|
|
| On Jul 9, 2012, at 7:45 PM, Dale Henrichs wrote:
|
| > Lawrence,
| >
| > The only way to "suppress the source same but rev changed" is to
| > reload (or revert) the mcz file and get the timestamps from the
| > mcz file set correctly, once you do this you shouldn't have to do
| > it over and over again …
| >
|
|   I'm not sure what it means to revert the file. Is there a revert
|   option somewhere? I can't find it.

There is a revert package menu item in the repository pane of the MonticelloBrowser (select the package and the repository to see the menu item).
|
|
| > I have seen this every once in a while, but I haven't tracked the
| > bug down (a revert seems to clean it up) ...
| >
| > Do you happen to be using FileTree packages?
|
|   No, sorry, I'm not using FileTree packages.

I was asking, because the FileTree packages might be the source of the bug:)

Just trying to characterize the bug.

|
| >
| > What kind of repositories are you using (Directory-based or Http)
| > for the affect packages?
| >
|
|
|   I'm using a directory based repository. I've been meaning to work
|   everything into Metacello but I never find the time.

Do you load using a script in topaz or manually from GemTools?

I'm just trying to get an idea of what may be the root cause ...

|
|
|
| P.S. Ever since I took down 'monit', I have not seen those  walkbacks
| show up.


Okay, so monit was the culprit .. presumably you had things working at one point in time with monit, so something must have changed in the setup...I'm suspicious when you said you had multiple gems being started, that something was funky with monit ...

Dale

On Jul 10, 2012, at 5:14 PM, Dale Henrichs wrote:

  Do you mean Refresh? I can't for the life of me find a Revert option. I'm also a little unclear on what it does, to tell you the truth.

  Is there some way I can just hack the PatchBrowser to not show those lines? ;-) If the code is the same, everything is fine with me.


  I load manually from Gemtools. I carefully load in the changes I need.


   It seems to me that that Gems crashed and did not get killed from the process list. Monit could not communicate with the Gems that crashed so it started other Gems. Even when I got rid of the crashed Gems, by saving the Gems that had current pids in the pid files, and killing the others, I was still getting those unreported errors. I think the data being sent to a Gem to check its status is somehow wrong. I hope someone who knows more about FastCGI and the data being sent can comment.

  monit was a great way to keep the system up and running and I would like to have it back.




> Dale

I've attached a screenshot highlighting the location of the revert package menu item...

Dale

----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, July 10, 2012 2:27:09 PM
| Subject: Re: [GS/SS Beta] Any way to hide (source same but rev changed) methods?
|
|
| On Jul 10, 2012, at 5:14 PM, Dale Henrichs wrote:
|
| >
| >
| > ----- Original Message -----
| > | From: "Lawrence Kellogg" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Tuesday, July 10, 2012 1:50:43 PM
| > | Subject: Re: [GS/SS Beta] Any way to hide (source same but rev
| > | changed) methods?
| > |
| > |
| > | On Jul 9, 2012, at 7:45 PM, Dale Henrichs wrote:
| > |
| > | > Lawrence,
| > | >
| > | > The only way to "suppress the source same but rev changed" is
| > | > to
| > | > reload (or revert) the mcz file and get the timestamps from the
| > | > mcz file set correctly, once you do this you shouldn't have to
| > | > do
| > | > it over and over again …
| > | >
| > |
| > |   I'm not sure what it means to revert the file. Is there a
| > |   revert
| > |   option somewhere? I can't find it.
| >
| > There is a revert package menu item in the repository pane of the
| > MonticelloBrowser (select the package and the repository to see
| > the menu item).
|
|   Do you mean Refresh? I can't for the life of me find a Revert
|   option. I'm also a little unclear on what it does, to tell you the
|   truth.
|
|   Is there some way I can just hack the PatchBrowser to not show
|   those lines? ;-) If the code is the same, everything is fine with
|   me.
|
|
| > |
| > |
| > | > I have seen this every once in a while, but I haven't tracked
| > | > the
| > | > bug down (a revert seems to clean it up) ...
| > | >
| > | > Do you happen to be using FileTree packages?
| > |
| > |   No, sorry, I'm not using FileTree packages.
| >
| > I was asking, because the FileTree packages might be the source of
| > the bug:)
| >
| > Just trying to characterize the bug.
| >
| > |
| > | >
| > | > What kind of repositories are you using (Directory-based or
| > | > Http)
| > | > for the affect packages?
| > | >
| > |
| > |
| > |   I'm using a directory based repository. I've been meaning to
| > |   work
| > |   everything into Metacello but I never find the time.
| >
| > Do you load using a script in topaz or manually from GemTools?
| >
|
|   I load manually from Gemtools. I carefully load in the changes I
|   need.
|
|
| > I'm just trying to get an idea of what may be the root cause ...
| >
| > |
| > |
| > |
| > | P.S. Ever since I took down 'monit', I have not seen those
| > |  walkbacks
| > | show up.
| >
| >
| > Okay, so monit was the culprit .. presumably you had things working
| > at one point in time with monit, so something must have changed in
| > the setup...I'm suspicious when you said you had multiple gems
| > being started, that something was funky with monit …
| >
|
|    It seems to me that that Gems crashed and did not get killed from
|    the process list. Monit could not communicate with the Gems that
|    crashed so it started other Gems. Even when I got rid of the
|    crashed Gems, by saving the Gems that had current pids in the pid
|    files, and killing the others, I was still getting those
|    unreported errors. I think the data being sent to a Gem to check
|    its status is somehow wrong. I hope someone who knows more about
|    FastCGI and the data being sent can comment.
|
|   monit was a great way to keep the system up and running and I would
|   like to have it back.
|
|
|
|
| > Dale
|
|

On Jul 10, 2012, at 6:13 PM, Dale Henrichs wrote:

> I've attached a screenshot highlighting the location of the revert package menu item…
>

 Thanks, I was down at a lower level where you select the file you want to apply changes from.
I'm afraid of that option, though. What does it mean to Revert Package? Revert to what?

Larry

On Fri, Jul 6, 2012 at 9:25 PM, Lisa Almarode <[hidden email]> wrote:


--

On Fri, Jul 6, 2012 at 6:37 PM, James Foster <[hidden email]> wrote:


--

When you select that menu item, you are prompted for which version of the package to load and the top version is the version that you've loaded, so you end up reloading the package ... which is all that I think needs to be done for fixing the method timestamps ...

if you reload (or revert) the package and then do a diff and see that the messages about "revision changed" then there is something very odd going on and I'd like to get my hands on the package to see if I can reproduce the problem ...

Dale

----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Tuesday, July 10, 2012 4:05:06 PM
| Subject: Re: [GS/SS Beta] Any way to hide (source same but rev changed) methods?
|
|
| On Jul 10, 2012, at 6:13 PM, Dale Henrichs wrote:
|
| > I've attached a screenshot highlighting the location of the revert
| > package menu item…
| >
|
|  Thanks, I was down at a lower level where you select the file you
|  want to apply changes from.
| I'm afraid of that option, though. What does it mean to Revert
| Package? Revert to what?
|
| Larry
|
| > Dale
| >
| > ----- Original Message -----
| > | From: "Lawrence Kellogg" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Tuesday, July 10, 2012 2:27:09 PM
| > | Subject: Re: [GS/SS Beta] Any way to hide (source same but rev
| > | changed) methods?
| > |
| > |
| > | On Jul 10, 2012, at 5:14 PM, Dale Henrichs wrote:
| > |
| > | >
| > | >
| > | > ----- Original Message -----
| > | > | From: "Lawrence Kellogg" <[hidden email]>
| > | > | To: "GemStone Seaside beta discussion"
| > | > | <[hidden email]>
| > | > | Sent: Tuesday, July 10, 2012 1:50:43 PM
| > | > | Subject: Re: [GS/SS Beta] Any way to hide (source same but
| > | > | rev
| > | > | changed) methods?
| > | > |
| > | > |
| > | > | On Jul 9, 2012, at 7:45 PM, Dale Henrichs wrote:
| > | > |
| > | > | > Lawrence,
| > | > | >
| > | > | > The only way to "suppress the source same but rev changed"
| > | > | > is
| > | > | > to
| > | > | > reload (or revert) the mcz file and get the timestamps from
| > | > | > the
| > | > | > mcz file set correctly, once you do this you shouldn't have
| > | > | > to
| > | > | > do
| > | > | > it over and over again …
| > | > | >
| > | > |
| > | > |   I'm not sure what it means to revert the file. Is there a
| > | > |   revert
| > | > |   option somewhere? I can't find it.
| > | >
| > | > There is a revert package menu item in the repository pane of
| > | > the
| > | > MonticelloBrowser (select the package and the repository to see
| > | > the menu item).
| > |
| > |   Do you mean Refresh? I can't for the life of me find a Revert
| > |   option. I'm also a little unclear on what it does, to tell you
| > |   the
| > |   truth.
| > |
| > |   Is there some way I can just hack the PatchBrowser to not show
| > |   those lines? ;-) If the code is the same, everything is fine
| > |   with
| > |   me.
| > |
| > |
| > | > |
| > | > |
| > | > | > I have seen this every once in a while, but I haven't
| > | > | > tracked
| > | > | > the
| > | > | > bug down (a revert seems to clean it up) ...
| > | > | >
| > | > | > Do you happen to be using FileTree packages?
| > | > |
| > | > |   No, sorry, I'm not using FileTree packages.
| > | >
| > | > I was asking, because the FileTree packages might be the source
| > | > of
| > | > the bug:)
| > | >
| > | > Just trying to characterize the bug.
| > | >
| > | > |
| > | > | >
| > | > | > What kind of repositories are you using (Directory-based or
| > | > | > Http)
| > | > | > for the affect packages?
| > | > | >
| > | > |
| > | > |
| > | > |   I'm using a directory based repository. I've been meaning
| > | > |   to
| > | > |   work
| > | > |   everything into Metacello but I never find the time.
| > | >
| > | > Do you load using a script in topaz or manually from GemTools?
| > | >
| > |
| > |   I load manually from Gemtools. I carefully load in the changes
| > |   I
| > |   need.
| > |
| > |
| > | > I'm just trying to get an idea of what may be the root cause
| > | > ...
| > | >
| > | > |
| > | > |
| > | > |
| > | > | P.S. Ever since I took down 'monit', I have not seen those
| > | > |  walkbacks
| > | > | show up.
| > | >
| > | >
| > | > Okay, so monit was the culprit .. presumably you had things
| > | > working
| > | > at one point in time with monit, so something must have changed
| > | > in
| > | > the setup...I'm suspicious when you said you had multiple gems
| > | > being started, that something was funky with monit …
| > | >
| > |
| > |    It seems to me that that Gems crashed and did not get killed
| > |    from
| > |    the process list. Monit could not communicate with the Gems
| > |    that
| > |    crashed so it started other Gems. Even when I got rid of the
| > |    crashed Gems, by saving the Gems that had current pids in the
| > |    pid
| > |    files, and killing the others, I was still getting those
| > |    unreported errors. I think the data being sent to a Gem to
| > |    check
| > |    its status is somehow wrong. I hope someone who knows more
| > |    about
| > |    FastCGI and the data being sent can comment.
| > |
| > |   monit was a great way to keep the system up and running and I
| > |   would
| > |   like to have it back.
| > |
| > |
| > |
| > |
| > | > Dale
| > |
| > |
| > <revertPackageMenuItem.png>
|
|

Now, things seem to be getting worse in FastCGI land.

I'm getting these errors:

2012/07/11 15:03:55 [error] 12493#0: *157316 connect() failed (111: Connection refused) while connecting to upstream, client:

which I have never seen before….

But, I can login and my website is served, and my iPhone client talks to the server.
But it seems like sometimes my iPhone client encounters those upstream errors.

Totally puzzled.



On Jul 10, 2012, at 12:23 PM, Lawrence Kellogg wrote:

That appears to be from the nginx log? The information following the ':' should tell you which server is the "bad boy"...

The seaside server that it is trying to connect to is not accepting the connection ... now that can be for a couple of different reasons:

  - the number of queued accept requests is exceeded for the port on that server
  - the server has stopped actively accepting connections

You should check the logs for your seaside servers and see if all of your topaz sessions are healthy.

If you are consistently getting the same error for the same server, then it is likely that the server is "broken". If you to a `kill -USER1 <pid>` you should get a stack trace from the offending server and see what it is doing.

If you are running statmonitor, you can use vsd and my looking at SessionStat0 you can see the number of http requests being handled by each seaside server. If the server being called out by nginx is continuing to respond to requests after the nginx event, then it is likely that you were momentarily overwhelmed with http requests and you might need to add addtional seaside servers ...

Dale


----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Wednesday, July 11, 2012 8:37:35 AM
| Subject: Re: [GS/SS Beta] Far worse problems…
|
| Now, things seem to be getting worse in FastCGI land.
|
| I'm getting these errors:
|
| 2012/07/11 15:03:55 [error] 12493#0: *157316 connect() failed (111:
| Connection refused) while connecting to upstream, client:
|
| which I have never seen before….
|
| But, I can login and my website is served, and my iPhone client talks
| to the server.
| But it seems like sometimes my iPhone client encounters those
| upstream errors.
|
| Totally puzzled.
|
|
|
| On Jul 10, 2012, at 12:23 PM, Lawrence Kellogg wrote:
|
| >
| > On Jul 10, 2012, at 12:14 PM, Dale Henrichs wrote:
| >
| >> I'm not familiar with Monit..
| >>
| >
| >  That's ok, maybe Johan can comment. He's the guy who sent me down
| >  the path of using monit.
| > It's been working great, except for the sudden appearance of these
| > Unreported errors.
| >
| >  I restarted all of the Gems again so I can see that they're
| >  running fine in the log files, and
| > I have killed monit. So far, I don't see these walk backs getting
| > written to the logs but it is probably too early to tell.
| >
| >  Thanks for your help.
| >
| >  Larry
| >
| >
| >
| >
| >> ----- Original Message -----
| >> | From: "Lawrence Kellogg" <[hidden email]>
| >> | To: "GemStone Seaside beta discussion"
| >> | <[hidden email]>
| >> | Sent: Tuesday, July 10, 2012 8:36:34 AM
| >> | Subject: Re: [GS/SS Beta] Far worse problems…
| >> |
| >> |
| >> | On Jul 9, 2012, at 11:11 PM, Dale Henrichs wrote:
| >> |
| >> | > Lawrence,
| >> | >
| >> | > If you've got multiple gems trying to listen on a port, only
| >> | > one
| >> | > gem can listen on a port at any one time, so I think that the
| >> | > the
| >> | > gem should fail when it tries to open the port.
| >> | >
| >> | > The gems are passive as they listen on the port so they can't
| >> | > really affect other gems ...
| >> | >
| >> | > Do you have a process that periodically "pings" a gem on it's
| >> | > http
| >> | > port to see if it is still alive ... I suspect a process like
| >> | > that
| >> | > has gone haywire and is pinging away on the gems …
| >> | >
| >> |
| >> | Well, I went back through monit, it's been a while since I have
| >> | looked at it, as I think it has been working fine for months.
| >> | Anyway, I see this:
| >> |
| >> |                 # Empty FastCGI request
| >> |                 if failed port 9002
| >> |                   # Send FastCGI packet: version 1 (0x01), cmd
| >> |                   FCGI_GET_VALUES (0x09)
| >> |                   # padding 8 bytes (0x08), followed by 8xNULLs
| >> |                   padding
| >> |                   send
| >> |                   "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
| >> |                   # Expect FastCGI packet: version 1 (0x01),
| >> |                   resp
| >> |                   FCGI_GET_VALUES_RESULT (0x0A)
| >> |                   expect "\0x01\0x0A"
| >> |                   timeout 20 seconds
| >> |                 then restart
| >> |
| >> | in the monit.conf file, so perhaps this packet to fastcgi is
| >> | wrong
| >> | for some reason. I never had a problem before….
| >> |
| >> | Any idea if this is valid or invalid? For now, I have shut down
| >> | monit
| >> | to see if the unreported error problem disappears.
| >> |
| >> | Larry
| >> |
| >> |
| >> | > Dale
| >> | >
| >> | > ----- Original Message -----
| >> | > | From: "Lawrence Kellogg" <[hidden email]>
| >> | > | To: "GemStone Seaside beta discussion"
| >> | > | <[hidden email]>
| >> | > | Sent: Monday, July 9, 2012 6:30:21 PM
| >> | > | Subject: Re: [GS/SS Beta] Far worse problems…
| >> | > |
| >> | > |
| >> | > | On Jul 9, 2012, at 8:42 PM, Dale Henrichs wrote:
| >> | > |
| >> | > | > Are you getting a lot of commit conflicts? I repeat that I
| >> | > | > think
| >> | > | > the two errors are unrelated ...
| >> | > | >
| >> | > |
| >> | > |   You're right, I'm not getting a lot of commit conflicts so
| >> | > |   I
| >> | > |   think
| >> | > |   these problems are unrelated.
| >> | > |
| >> | > |
| >> | > | > The commit conflict handling code is way after the fastCGI
| >> | > | > read
| >> | > | > code in the scheme of things...and unless your web server
| >> | > | > (nginx?
| >> | > | > lightppd?) is forwarding non-FastCGI traffic to the same
| >> | > | > ports
| >> | > | > that it is sending FASTCgi traffic the source of the bad
| >> | > | > FASTCgi
| >> | > | > almost has to be your server machine.
| >> | > | >
| >> | > |
| >> | > |   Could a crashed Gem still be sending bad FastCGI packets?
| >> | > |   I
| >> | > |   have a
| >> | > |   monit process in place and I think it is starting multiple
| >> | > |   Gems
| >> | > |   on
| >> | > |   9001, 9002, and 9003.
| >> | > | At least, I see topaz sessions when I do a ps-ef.
| >> | > |
| >> | > |
| >> | > | > Look at
| >> | > | > GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock:
| >> | > | > where the commit conflict logic is near the bottom of the
| >> | > | > method
| >> | > | > ... when a commit conflict is encountered the http request
| >> | > | > is
| >> | > | > retried (without engaging the fastCGI logic)... so it
| >> | > | > really
| >> | > | > isn't
| >> | > | > possible for the two different errors to be related...I
| >> | > | > don't
| >> | > | > think...
| >> | > | >
| >> | > | > The commit conflicts are logged in
| >> | > | > GRGemStonePlatform>>saveLogEntry:shouldCommit: and you
| >> | > | > could
| >> | > | > add a
| >> | > | > timestamp to the server log output ... you've probably
| >> | > | > also got
| >> | > | > objectlog entries for the commit conflicts (not for the
| >> | > | > unreportable errors) and object log entries have time
| >> | > | > stamps …
| >> | > | >
| >> | > |
| >> | > |   Ok, thanks….
| >> | > |
| >> | > | > ----- Original Message -----
| >> | > | > | From: "Lawrence Kellogg" <[hidden email]>
| >> | > | > | To: "GemStone Seaside beta discussion"
| >> | > | > | <[hidden email]>
| >> | > | > | Sent: Monday, July 9, 2012 5:29:41 PM
| >> | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| >> | > | > |
| >> | > | > |
| >> | > | > | On Jul 9, 2012, at 8:26 PM, Dale Henrichs wrote:
| >> | > | > |
| >> | > | > | > Is there a timestamp on the commit conflict log entry?
| >> | > | > | >
| >> | > | > |
| >> | > | > | No timestamp on the commit conflict, just from the
| >> | > | > | unrepeatable
| >> | > | > | errors around it….
| >> | > | > |
| >> | > | > |
| >> | > | > | -----------  Unreportable ERROR Encountered:
| >> | > | > | 2012-07-08T06:20:56.84343409538269-07:00InterpreterError
| >> | > | > | 2089: An
| >> | > | > | index range was specified for a sequenceable collection
| >> | > | > | with
| >> | > | > | the
| >> | > | > | starting index <1> greater than the ending index
| >> | > | > | <0>.-----------
| >> | > | > | retryFailure
| >> | > | > | Read-Write Conflicts...
| >> | > | > | Write-Write Conflicts...
| >> | > | > | Write-Dependency Conflicts...
| >> | > | > | Write-ReadLock Conflicts...
| >> | > | > | Write-WriteLock Conflicts...
| >> | > | > | Rc-Write-Write Conflicts...
| >> | > | > |     440885505
| >> | > | > | Synchronized-Commit Conflicts...
| >> | > | > | ----------- Commit failure - retrying LOG ENTRY:
| >> | > | > | aSymbolDictionary-----------
| >> | > | > | -----------  Unreportable ERROR Encountered:
| >> | > | > | 2012-07-08T06:22:56.88573288917542-07:00InterpreterError
| >> | > | > | 2089: An
| >> | > | > | index range was specified for a sequenceable collection
| >> | > | > | with
| >> | > | > | the
| >> | > | > | starting index <1> greater than the ending index <0>.--
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > | > Occasional commit conflicts are not actually cause for
| >> | > | > | > alarm
| >> | > | > | > ...
| >> | > | > | > according the log, the commit conflict was retried ...
| >> | > | > | > without
| >> | > | > | > further errors about the the retry limit being
| >> | > | > | > exceeded I'm
| >> | > | > | > pretty
| >> | > | > | > sure that the retry succeeded ...
| >> | > | > | >
| >> | > | > | > The "Unreportable ERROR" is very liklye coming from
| >> | > | > | > some
| >> | > | > | > process
| >> | > | > | > (most likely on your box) trying to connect to the
| >> | > | > | > fastCGI
| >> | > | > | > ports
| >> | > | > | > without actually passing in valid fastCGI …
| >> | > | > | >
| >> | > | > |
| >> | > | > | I'll have to think about this…. not sure what it could
| >> | > | > | be at
| >> | > | > | the
| >> | > | > | moment...
| >> | > | > |
| >> | > | > |   Larry
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > |
| >> | > | > | > Dale
| >> | > | > | >
| >> | > | > | > ----- Original Message -----
| >> | > | > | > | From: "Lawrence Kellogg" <[hidden email]>
| >> | > | > | > | To: "GemStone Seaside beta discussion"
| >> | > | > | > | <[hidden email]>
| >> | > | > | > | Sent: Monday, July 9, 2012 5:22:14 PM
| >> | > | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| >> | > | > | > |
| >> | > | > | > |
| >> | > | > | > | On Jul 9, 2012, at 8:12 PM, Dale Henrichs wrote:
| >> | > | > | > |
| >> | > | > | > | > Two document handlers with the same document will
| >> | > | > | > | > compare
| >> | > | > | > | > #= so
| >> | > | > | > | > two
| >> | > | > | > | > different sessions creating document handlers with
| >> | > | > | > | > the
| >> | > | > | > | > same
| >> | > | > | > | > document will conflict ...
| >> | > | > | > | >
| >> | > | > | > | > it would be interesting to see the document
| >> | > | > | > | > contents of
| >> | > | > | > | > the
| >> | > | > | > | > WADocument...that might have meaning for you..
| >> | > | > | > | >
| >> | > | > | > |
| >> | > | > | > | Well, that OOP now points a an empty dictionary. I
| >> | > | > | > | tried
| >> | > | > | > | to
| >> | > | > | > | look
| >> | > | > | > | at
| >> | > | > | > | the contents of the WADocument but it was a garage
| >> | > | > | > | string, as
| >> | > | > | > | far
| >> | > | > | > | as
| >> | > | > | > | I could tell.
| >> | > | > | > |
| >> | > | > | > |   You know, it only mentions the Rc-Write-Write
| >> | > | > | > |   conflict
| >> | > | > | > |   once
| >> | > | > | > |   and
| >> | > | > | > |   then it spews out hundreds of Unreportable errors.
| >> | > | > | > |   Now,
| >> | > | > | > |   I
| >> | > | > | > |   think
| >> | > | > | > |   it
| >> | > | > | > |   is still generating those errors, with a stack
| >> | > | > | > |   dump,
| >> | > | > | > |   but I
| >> | > | > | > |   can't
| >> | > | > | > |   figure out anything from the stack dump as to what
| >> | > | > | > |   is
| >> | > | > | > |   going
| >> | > | > | > |   wrong.
| >> | > | > | > |
| >> | > | > | > |   Larry
| >> | > | > | > |
| >> | > | > | > |
| >> | > | > | > |
| >> | > | > | > |
| >> | > | > | > | > Dale
| >> | > | > | > | >
| >> | > | > | > | >
| >> | > | > | > | > ----- Original Message -----
| >> | > | > | > | > | From: "Dale Henrichs" <[hidden email]>
| >> | > | > | > | > | To: "GemStone Seaside beta discussion"
| >> | > | > | > | > | <[hidden email]>
| >> | > | > | > | > | Sent: Monday, July 9, 2012 5:05:18 PM
| >> | > | > | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| >> | > | > | > | > |
| >> | > | > | > | > | The sessions are locked with object locks so you
| >> | > | > | > | > | shouldn't be
| >> | > | > | > | > | able to
| >> | > | > | > | > | access the same session in two different
| >> | > | > | > | > | servers.
| >> | > | > | > | > |
| >> | > | > | > | > | The Document handler looks suspicious to me ...
| >> | > | > | > | > | perhaps
| >> | > | > | > | > | you
| >> | > | > | > | > | have
| >> | > | > | > | > | different sessions trying to add a document
| >> | > | > | > | > | handler...
| >> | > | > | > | > | this
| >> | > | > | > | > | rings
| >> | > | > | > | > | a
| >> | > | > | > | > | bell but I can't recall the details .. it
| >> | > | > | > | > | doesn't
| >> | > | > | > | > | seem
| >> | > | > | > | > | right
| >> | > | > | > | > | that
| >> | > | > | > | > | document handlers are being stored in the
| >> | > | > | > | > | session
| >> | > | > | > | > | dictionary
| >> | > | > | > | > | ...
| >> | > | > | > | > |
| >> | > | > | > | > | Dale
| >> | > | > | > | > |
| >> | > | > | > | > | ----- Original Message -----
| >> | > | > | > | > | | From: "Lawrence Kellogg" <[hidden email]>
| >> | > | > | > | > | | To: "GemStone Seaside beta discussion"
| >> | > | > | > | > | | <[hidden email]>
| >> | > | > | > | > | | Sent: Monday, July 9, 2012 4:57:33 PM
| >> | > | > | > | > | | Subject: Re: [GS/SS Beta] Far worse problems…
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | On Jul 9, 2012, at 7:51 PM, Dale Henrichs
| >> | > | > | > | > | | wrote:
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | Well let' see...
| >> | > | > | > | > | |
| >> | > | > | > | > | | 440885505 is the oop of the object that you
| >> | > | > | > | > | | are
| >> | > | > | > | > | | getting
| >> | > | > | > | > | | the
| >> | > | > | > | > | | Rc-Write-Write Conflicts on. You can use:
| >> | > | > | > | > | |
| >> | > | > | > | > | | Object _objectForOop: 440885505
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | Thanks, Dale.
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | So, this object is a RcKeyValueDiction with
| >> | > | > | > | > | | two
| >> | > | > | > | > | | keys
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | aWASession and a aWADocumentHandler.
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | It does not seem like my code, but perhaps it
| >> | > | > | > | > | | has
| >> | > | > | > | > | | something
| >> | > | > | > | > | | to
| >> | > | > | > | > | | do
| >> | > | > | > | > | | with the session stuff.
| >> | > | > | > | > | | Out of all the code I wrote, I didn't expect
| >> | > | > | > | > | | this
| >> | > | > | > | > | | conflict.
| >> | > | > | > | > | | ;-)
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | Any chance that this call to createCache in my
| >> | > | > | > | > | | session
| >> | > | > | > | > | | class is
| >> | > | > | > | > | | causing the problem?
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | createCache
| >> | > | > | > | > | | "To configure the continuation cache you must
| >> | > | > | > | > | | currently
| >> | > | > | > | > | | subclass
| >> | > | > | > | > | | WASession and override this method."
| >> | > | > | > | > | | | policy |
| >> | > | > | > | > | | policy := WARcLastAccessExpiryPolicy new.
| >> | > | > | > | > | | policy configuration at: #cacheTimeout put:
| >> | > | > | > | > | | 3600.
| >> | > | > | > | > | | ^ WACache new
| >> | > | > | > | > | | setExpiryPolicy: policy;
| >> | > | > | > | > | | yourself.
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | Larry
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | to inspect the object ... the fact that it is
| >> | > | > | > | > | | an Rc
| >> | > | > | > | > | | error
| >> | > | > | > | > | | means
| >> | > | > | > | > | | that
| >> | > | > | > | > | | the type of conflict is one that is not
| >> | > | > | > | > | | resolvable
| >> | > | > | > | > | | by
| >> | > | > | > | > | | retry
| >> | > | > | > | > | | ...
| >> | > | > | > | > | | for
| >> | > | > | > | > | | an Rc Dictionary that means two sessions are
| >> | > | > | > | > | | using
| >> | > | > | > | > | | the
| >> | > | > | > | > | | adding/removing the same key ... For an Rc set
| >> | > | > | > | > | | that
| >> | > | > | > | > | | means
| >> | > | > | > | > | | two
| >> | > | > | > | > | | sessions are adding removing the same object
| >> | > | > | > | > | | ...
| >> | > | > | > | > | |
| >> | > | > | > | > | | Inspect the object and find out ...
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > | | ... I'll have more in a follow on mail...
| >> | > | > | > | > | |
| >> | > | > | > | > | | Dale
| >> | > | > | > | > | |
| >> | > | > | > | > | | ----- Original Message -----
| >> | > | > | > | > | | | From: "Lawrence Kellogg" < [hidden email] >
| >> | > | > | > | > | | | To: "GemStone Seaside beta discussion" <
| >> | > | > | > | > | | | [hidden email]
| >> | > | > | > | > | | | >
| >> | > | > | > | > | | | Sent: Monday, July 9, 2012 4:42:13 PM
| >> | > | > | > | > | | | Subject: [GS/SS Beta] Far worse problems…
| >> | > | > | > | > | | |
| >> | > | > | > | > | | | Hello,
| >> | > | > | > | > | | | Actually, I have far worse problems than
| >> | > | > | > | > | | | just
| >> | > | > | > | > | | | hiding
| >> | > | > | > | > | | | changes
| >> | > | > | > | > | | | in
| >> | > | > | > | > | | | the
| >> | > | > | > | > | | | Patch Browser. I just noticed that the log
| >> | > | > | > | > | | | files
| >> | > | > | > | > | | | for
| >> | > | > | > | > | | | my
| >> | > | > | > | > | | | Gems
| >> | > | > | > | > | | | are
| >> | > | > | > | > | | | spewing out this:
| >> | > | > | > | > | | |
| >> | > | > | > | > | | |
| >> | > | > | > | > | | |
| >> | > | > | > | > | | |
| >> | > | > | > | > | | | etryFailure
| >> | > | > | > | > | | | Read-Write Conflicts...
| >> | > | > | > | > | | | Write-Write Conflicts...
| >> | > | > | > | > | | | Write-Dependency Conflicts...
| >> | > | > | > | > | | | Write-ReadLock Conflicts...
| >> | > | > | > | > | | | Write-WriteLock Conflicts...
| >> | > | > | > | > | | | Rc-Write-Write Conflicts...
| >> | > | > | > | > | | | 440885505
| >> | > | > | > | > | | | Synchronized-Commit Conflicts...
| >> | > | > | > | > | | | ----------- Commit failure - retrying LOG
| >> | > | > | > | > | | | ENTRY:
| >> | > | > | > | > | | | aSymbolDictionary-----------
| >> | > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| >> | > | > | > | > | | | 2012-07-08T18:37:39.67828106880188-07:00InterpreterError
| >> | > | > | > | > | | | 2089: An
| >> | > | > | > | > | | | index range was specified for a sequenceable
| >> | > | > | > | > | | | collection
| >> | > | > | > | > | | | with
| >> | > | > | > | > | | | the
| >> | > | > | > | > | | | starting index <1> greater than the ending
| >> | > | > | > | > | | | index
| >> | > | > | > | > | | | <0>.-----------
| >> | > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| >> | > | > | > | > | | | 2012-07-08T18:39:39.71508598327637-07:00InterpreterError
| >> | > | > | > | > | | | 2089: An
| >> | > | > | > | > | | | index range was specified for a sequenceable
| >> | > | > | > | > | | | collection
| >> | > | > | > | > | | | with
| >> | > | > | > | > | | | the
| >> | > | > | > | > | | | starting index <1> greater than the ending
| >> | > | > | > | > | | | index
| >> | > | > | > | > | | | <0>.-----------
| >> | > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| >> | > | > | > | > | | | 2012-07-08T18:41:39.78475308418274-07:00InterpreterError
| >> | > | > | > | > | | | 2089: An
| >> | > | > | > | > | | | index range was specified for a sequenceable
| >> | > | > | > | > | | | collection
| >> | > | > | > | > | | | with
| >> | > | > | > | > | | | the
| >> | > | > | > | > | | | starting index <1> greater than the ending
| >> | > | > | > | > | | | index
| >> | > | > | > | > | | | <0>.-----------
| >> | > | > | > | > | | | ----------- Unreportable ERROR Encountered:
| >> | > | > | > | > | | | 2012-07-08T18:43:39.84294891357422-07:00InterpreterError
| >> | > | > | > | > | | | 2089: An
| >> | > | > | > | > | | | index range was specified for a sequenceable
| >> | > | > | > | > | | | collection
| >> | > | > | > | > | | | with
| >> | > | > | > | > | | | the
| >> | > | > | > | > | | | starting index <1> greater than the ending
| >> | > | > | > | > | | | index
| >> | > | > | > | > | | | <0>.-----------
| >> | > | > | > | > | | |
| >> | > | > | > | > | | | on and on and on...
| >> | > | > | > | > | | |
| >> | > | > | > | > | | | Help! What do I do? I took care of a walk
| >> | > | > | > | > | | | back I
| >> | > | > | > | > | | | add
| >> | > | > | > | > | | | and
| >> | > | > | > | > | | | actually
| >> | > | > | > | > | | | have no idea what sequenceable collection is
| >> | > | > | > | > | | | causing
| >> | > | > | > | > | | | this
| >> | > | > | > | > | | | problem.
| >> | > | > | > | > | | | My system is up and running fine but all of
| >> | > | > | > | > | | | these
| >> | > | > | > | > | | | errors
| >> | > | > | > | > | | | are
| >> | > | > | > | > | | | being
| >> | > | > | > | > | | | thrown.
| >> | > | > | > | > | | |
| >> | > | > | > | > | | |
| >> | > | > | > | > | | | Larry
| >> | > | > | > | > | |
| >> | > | > | > | > | |
| >> | > | > | > | > |
| >> | > | > | > |
| >> | > | > | > |
| >> | > | > |
| >> | > | > |
| >> | > |
| >> | > |
| >> |
| >> |
| >
|
|

On Jul 11, 2012, at 12:14 PM, Dale Henrichs wrote:

> That appears to be from the nginx log? The information following the ':' should tell you which server is the "bad boy"…
>

Yes, this is a printout from the nginx error.log file.

  Ok, thanks for all this information. I will keep an eye on it and try to figure out what is going on. I recycled all the Gems and it seems like this problem has stopped. Even with those errors, it looks like that traffic did update my database so it's a little confusing.

  By the way, Johan said that the FastCGI packet was not meant to be valid and that he gets the same errors in the log! Sorry for leading you on this wild goose chase.

  Larry

No problem ... we've created google fodder and learned a few things:)

Dale

----- Original Message -----
| From: "Lawrence Kellogg" <[hidden email]>
| To: "GemStone Seaside beta discussion" <[hidden email]>
| Sent: Wednesday, July 11, 2012 1:48:58 PM
| Subject: Re: [GS/SS Beta] Far worse problems…
|
|
| On Jul 11, 2012, at 12:14 PM, Dale Henrichs wrote:
|
| > That appears to be from the nginx log? The information following
| > the ':' should tell you which server is the "bad boy"…
| >
|
| Yes, this is a printout from the nginx error.log file.
|
| > The seaside server that it is trying to connect to is not accepting
| > the connection ... now that can be for a couple of different
| > reasons:
| >
| >  - the number of queued accept requests is exceeded for the port on
| >  that server
| >  - the server has stopped actively accepting connections
| >
| > You should check the logs for your seaside servers and see if all
| > of your topaz sessions are healthy.
| >
| > If you are consistently getting the same error for the same server,
| > then it is likely that the server is "broken". If you to a `kill
| > -USER1 <pid>` you should get a stack trace from the offending
| > server and see what it is doing.
| >
| > If you are running statmonitor, you can use vsd and my looking at
| > SessionStat0 you can see the number of http requests being handled
| > by each seaside server. If the server being called out by nginx is
| > continuing to respond to requests after the nginx event, then it
| > is likely that you were momentarily overwhelmed with http requests
| > and you might need to add addtional seaside servers …
| >
|
|   Ok, thanks for all this information. I will keep an eye on it and
|   try to figure out what is going on. I recycled all the Gems and it
|   seems like this problem has stopped. Even with those errors, it
|   looks like that traffic did update my database so it's a little
|   confusing.
|
|   By the way, Johan said that the FastCGI packet was not meant to be
|   valid and that he gets the same errors in the log! Sorry for
|   leading you on this wild goose chase.
|
|   Larry
|
|
|
|
| > Dale
| >
| >
| > ----- Original Message -----
| > | From: "Lawrence Kellogg" <[hidden email]>
| > | To: "GemStone Seaside beta discussion"
| > | <[hidden email]>
| > | Sent: Wednesday, July 11, 2012 8:37:35 AM
| > | Subject: Re: [GS/SS Beta] Far worse problems…
| > |
| > | Now, things seem to be getting worse in FastCGI land.
| > |
| > | I'm getting these errors:
| > |
| > | 2012/07/11 15:03:55 [error] 12493#0: *157316 connect() failed
| > | (111:
| > | Connection refused) while connecting to upstream, client:
| > |
| > | which I have never seen before….
| > |
| > | But, I can login and my website is served, and my iPhone client
| > | talks
| > | to the server.
| > | But it seems like sometimes my iPhone client encounters those
| > | upstream errors.
| > |
| > | Totally puzzled.
| > |
| > |
| > |
| > | On Jul 10, 2012, at 12:23 PM, Lawrence Kellogg wrote:
| > |
| > | >
| > | > On Jul 10, 2012, at 12:14 PM, Dale Henrichs wrote:
| > | >
| > | >> I'm not familiar with Monit..
| > | >>
| > | >
| > | >  That's ok, maybe Johan can comment. He's the guy who sent me
| > | >  down
| > | >  the path of using monit.
| > | > It's been working great, except for the sudden appearance of
| > | > these
| > | > Unreported errors.
| > | >
| > | >  I restarted all of the Gems again so I can see that they're
| > | >  running fine in the log files, and
| > | > I have killed monit. So far, I don't see these walk backs
| > | > getting
| > | > written to the logs but it is probably too early to tell.
| > | >
| > | >  Thanks for your help.
| > | >
| > | >  Larry
| > | >
| > | >
| > | >
| > | >
| > | >> ----- Original Message -----
| > | >> | From: "Lawrence Kellogg" <[hidden email]>
| > | >> | To: "GemStone Seaside beta discussion"
| > | >> | <[hidden email]>
| > | >> | Sent: Tuesday, July 10, 2012 8:36:34 AM
| > | >> | Subject: Re: [GS/SS Beta] Far worse problems…
| > | >> |
| > | >> |
| > | >> | On Jul 9, 2012, at 11:11 PM, Dale Henrichs wrote:
| > | >> |
| > | >> | > Lawrence,
| > | >> | >
| > | >> | > If you've got multiple gems trying to listen on a port,
| > | >> | > only
| > | >> | > one
| > | >> | > gem can listen on a port at any one time, so I think that
| > | >> | > the
| > | >> | > the
| > | >> | > gem should fail when it tries to open the port.
| > | >> | >
| > | >> | > The gems are passive as they listen on the port so they
| > | >> | > can't
| > | >> | > really affect other gems ...
| > | >> | >
| > | >> | > Do you have a process that periodically "pings" a gem on
| > | >> | > it's
| > | >> | > http
| > | >> | > port to see if it is still alive ... I suspect a process
| > | >> | > like
| > | >> | > that
| > | >> | > has gone haywire and is pinging away on the gems …
| > | >> | >
| > | >> |
| > | >> | Well, I went back through monit, it's been a while since I
| > | >> | have
| > | >> | looked at it, as I think it has been working fine for
| > | >> | months.
| > | >> | Anyway, I see this:
| > | >> |
| > | >> |                 # Empty FastCGI request
| > | >> |                 if failed port 9002
| > | >> |                   # Send FastCGI packet: version 1 (0x01),
| > | >> |                   cmd
| > | >> |                   FCGI_GET_VALUES (0x09)
| > | >> |                   # padding 8 bytes (0x08), followed by
| > | >> |                   8xNULLs
| > | >> |                   padding
| > | >> |                   send
| > | >> |                   "\0x01\0x09\0x00\0x00\0x00\0x00\0x08\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00\0x00"
| > | >> |                   # Expect FastCGI packet: version 1 (0x01),
| > | >> |                   resp
| > | >> |                   FCGI_GET_VALUES_RESULT (0x0A)
| > | >> |                   expect "\0x01\0x0A"
| > | >> |                   timeout 20 seconds
| > | >> |                 then restart
| > | >> |
| > | >> | in the monit.conf file, so perhaps this packet to fastcgi is
| > | >> | wrong
| > | >> | for some reason. I never had a problem before….
| > | >> |
| > | >> | Any idea if this is valid or invalid? For now, I have shut
| > | >> | down
| > | >> | monit
| > | >> | to see if the unreported error problem disappears.
| > | >> |
| > | >> | Larry
| > | >> |
| > | >> |
| > | >> | > Dale
| > | >> | >
| > | >> | > ----- Original Message -----
| > | >> | > | From: "Lawrence Kellogg" <[hidden email]>
| > | >> | > | To: "GemStone Seaside beta discussion"
| > | >> | > | <[hidden email]>
| > | >> | > | Sent: Monday, July 9, 2012 6:30:21 PM
| > | >> | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | >> | > |
| > | >> | > |
| > | >> | > | On Jul 9, 2012, at 8:42 PM, Dale Henrichs wrote:
| > | >> | > |
| > | >> | > | > Are you getting a lot of commit conflicts? I repeat
| > | >> | > | > that I
| > | >> | > | > think
| > | >> | > | > the two errors are unrelated ...
| > | >> | > | >
| > | >> | > |
| > | >> | > |   You're right, I'm not getting a lot of commit
| > | >> | > |   conflicts so
| > | >> | > |   I
| > | >> | > |   think
| > | >> | > |   these problems are unrelated.
| > | >> | > |
| > | >> | > |
| > | >> | > | > The commit conflict handling code is way after the
| > | >> | > | > fastCGI
| > | >> | > | > read
| > | >> | > | > code in the scheme of things...and unless your web
| > | >> | > | > server
| > | >> | > | > (nginx?
| > | >> | > | > lightppd?) is forwarding non-FastCGI traffic to the
| > | >> | > | > same
| > | >> | > | > ports
| > | >> | > | > that it is sending FASTCgi traffic the source of the
| > | >> | > | > bad
| > | >> | > | > FASTCgi
| > | >> | > | > almost has to be your server machine.
| > | >> | > | >
| > | >> | > |
| > | >> | > |   Could a crashed Gem still be sending bad FastCGI
| > | >> | > |   packets?
| > | >> | > |   I
| > | >> | > |   have a
| > | >> | > |   monit process in place and I think it is starting
| > | >> | > |   multiple
| > | >> | > |   Gems
| > | >> | > |   on
| > | >> | > |   9001, 9002, and 9003.
| > | >> | > | At least, I see topaz sessions when I do a ps-ef.
| > | >> | > |
| > | >> | > |
| > | >> | > | > Look at
| > | >> | > | > GRGemStonePlatform>>seasideProcessRequestWithRetry:resultBlock:
| > | >> | > | > where the commit conflict logic is near the bottom of
| > | >> | > | > the
| > | >> | > | > method
| > | >> | > | > ... when a commit conflict is encountered the http
| > | >> | > | > request
| > | >> | > | > is
| > | >> | > | > retried (without engaging the fastCGI logic)... so it
| > | >> | > | > really
| > | >> | > | > isn't
| > | >> | > | > possible for the two different errors to be
| > | >> | > | > related...I
| > | >> | > | > don't
| > | >> | > | > think...
| > | >> | > | >
| > | >> | > | > The commit conflicts are logged in
| > | >> | > | > GRGemStonePlatform>>saveLogEntry:shouldCommit: and you
| > | >> | > | > could
| > | >> | > | > add a
| > | >> | > | > timestamp to the server log output ... you've probably
| > | >> | > | > also got
| > | >> | > | > objectlog entries for the commit conflicts (not for
| > | >> | > | > the
| > | >> | > | > unreportable errors) and object log entries have time
| > | >> | > | > stamps …
| > | >> | > | >
| > | >> | > |
| > | >> | > |   Ok, thanks….
| > | >> | > |
| > | >> | > | > ----- Original Message -----
| > | >> | > | > | From: "Lawrence Kellogg" <[hidden email]>
| > | >> | > | > | To: "GemStone Seaside beta discussion"
| > | >> | > | > | <[hidden email]>
| > | >> | > | > | Sent: Monday, July 9, 2012 5:29:41 PM
| > | >> | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > | On Jul 9, 2012, at 8:26 PM, Dale Henrichs wrote:
| > | >> | > | > |
| > | >> | > | > | > Is there a timestamp on the commit conflict log
| > | >> | > | > | > entry?
| > | >> | > | > | >
| > | >> | > | > |
| > | >> | > | > | No timestamp on the commit conflict, just from the
| > | >> | > | > | unrepeatable
| > | >> | > | > | errors around it….
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > | -----------  Unreportable ERROR Encountered:
| > | >> | > | > | 2012-07-08T06:20:56.84343409538269-07:00InterpreterError
| > | >> | > | > | 2089: An
| > | >> | > | > | index range was specified for a sequenceable
| > | >> | > | > | collection
| > | >> | > | > | with
| > | >> | > | > | the
| > | >> | > | > | starting index <1> greater than the ending index
| > | >> | > | > | <0>.-----------
| > | >> | > | > | retryFailure
| > | >> | > | > | Read-Write Conflicts...
| > | >> | > | > | Write-Write Conflicts...
| > | >> | > | > | Write-Dependency Conflicts...
| > | >> | > | > | Write-ReadLock Conflicts...
| > | >> | > | > | Write-WriteLock Conflicts...
| > | >> | > | > | Rc-Write-Write Conflicts...
| > | >> | > | > |     440885505
| > | >> | > | > | Synchronized-Commit Conflicts...
| > | >> | > | > | ----------- Commit failure - retrying LOG ENTRY:
| > | >> | > | > | aSymbolDictionary-----------
| > | >> | > | > | -----------  Unreportable ERROR Encountered:
| > | >> | > | > | 2012-07-08T06:22:56.88573288917542-07:00InterpreterError
| > | >> | > | > | 2089: An
| > | >> | > | > | index range was specified for a sequenceable
| > | >> | > | > | collection
| > | >> | > | > | with
| > | >> | > | > | the
| > | >> | > | > | starting index <1> greater than the ending index
| > | >> | > | > | <0>.--
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > | > Occasional commit conflicts are not actually cause
| > | >> | > | > | > for
| > | >> | > | > | > alarm
| > | >> | > | > | > ...
| > | >> | > | > | > according the log, the commit conflict was retried
| > | >> | > | > | > ...
| > | >> | > | > | > without
| > | >> | > | > | > further errors about the the retry limit being
| > | >> | > | > | > exceeded I'm
| > | >> | > | > | > pretty
| > | >> | > | > | > sure that the retry succeeded ...
| > | >> | > | > | >
| > | >> | > | > | > The "Unreportable ERROR" is very liklye coming
| > | >> | > | > | > from
| > | >> | > | > | > some
| > | >> | > | > | > process
| > | >> | > | > | > (most likely on your box) trying to connect to the
| > | >> | > | > | > fastCGI
| > | >> | > | > | > ports
| > | >> | > | > | > without actually passing in valid fastCGI …
| > | >> | > | > | >
| > | >> | > | > |
| > | >> | > | > | I'll have to think about this…. not sure what it
| > | >> | > | > | could
| > | >> | > | > | be at
| > | >> | > | > | the
| > | >> | > | > | moment...
| > | >> | > | > |
| > | >> | > | > |   Larry
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > | > | > Dale
| > | >> | > | > | >
| > | >> | > | > | > ----- Original Message -----
| > | >> | > | > | > | From: "Lawrence Kellogg" <[hidden email]>
| > | >> | > | > | > | To: "GemStone Seaside beta discussion"
| > | >> | > | > | > | <[hidden email]>
| > | >> | > | > | > | Sent: Monday, July 9, 2012 5:22:14 PM
| > | >> | > | > | > | Subject: Re: [GS/SS Beta] Far worse problems…
| > | >> | > | > | > |
| > | >> | > | > | > |
| > | >> | > | > | > | On Jul 9, 2012, at 8:12 PM, Dale Henrichs wrote:
| > | >> | > | > | > |
| > | >> | > | > | > | > Two document handlers with the same document
| > | >> | > | > | > | > will
| > | >> | > | > | > | > compare
| > | >> | > | > | > | > #= so
| > | >> | > | > | > | > two
| > | >> | > | > | > | > different sessions creating document handlers
| > | >> | > | > | > | > with
| > | >> | > | > | > | > the
| > | >> | > | > | > | > same
| > | >> | > | > | > | > document will conflict ...
| > | >> | > | > | > | >
| > | >> | > | > | > | > it would be interesting to see the document
| > | >> | > | > | > | > contents of
| > | >> | > | > | > | > the
| > | >> | > | > | > | > WADocument...that might have meaning for you..
| > | >> | > | > | > | >
| > | >> | > | > | > |
| > | >> | > | > | > | Well, that OOP now points a an empty dictionary.
| > | >> | > | > | > | I
| > | >> | > | > | > | tried
| > | >> | > | > | > | to
| > | >> | > | > | > | look
| > | >> | > | > | > | at
| > | >> | > | > | > | the contents of the WADocument but it was a
| > | >> | > | > | > | garage
| > | >> | > | > | > | string, as
| > | >> | > | > | > | far
| > | >> | > | > | > | as
| > | >> | > | > | > | I could tell.
| > | >> | > | > | > |
| > | >> | > | > | > |   You know, it only mentions the Rc-Write-Write
| > | >> | > | > | > |   conflict
| > | >> | > | > | > |   once
| > | >> | > | > | > |   and
| > | >> | > | > | > |   then it spews out hundreds of Unreportable
| > | >> | > | > | > |   errors.
| > | >> | > | > | > |   Now,
| > | >> | > | > | > |   I
| > | >> | > | > | > |   think
| > | >> | > | > | > |   it
| > | >> | > | > | > |   is still generating those errors, with a stack
| > | >> | > | > | > |   dump,
| > | >> | > | > | > |   but I
| > | >> | > | > | > |   can't
| > | >> | > | > | > |   figure out anything from the stack dump as to
| > | >> | > | > | > |   what
| > | >> | > | > | > |   is
| > | >> | > | > | > |   going
| > | >> | > | > | > |   wrong.
| > | >> | > | > | > |
| > | >> | > | > | > |   Larry
| > | >> | > | > | > |
| > | >> | > | > | > |
| > | >> | > | > | > |
| > | >> | > | > | > |
| > | >> | > | > | > | > Dale
| > | >> | > | > | > | >
| > | >> | > | > | > | >
| > | >> | > | > | > | > ----- Original Message -----
| > | >> | > | > | > | > | From: "Dale Henrichs" <[hidden email]>
| > | >> | > | > | > | > | To: "GemStone Seaside beta discussion"
| > | >> | > | > | > | > | <[hidden email]>
| > | >> | > | > | > | > | Sent: Monday, July 9, 2012 5:05:18 PM
| > | >> | > | > | > | > | Subject: Re: [GS/SS Beta] Far worse
| > | >> | > | > | > | > | problems…
| > | >> | > | > | > | > |
| > | >> | > | > | > | > | The sessions are locked with object locks so
| > | >> | > | > | > | > | you
| > | >> | > | > | > | > | shouldn't be
| > | >> | > | > | > | > | able to
| > | >> | > | > | > | > | access the same session in two different
| > | >> | > | > | > | > | servers.
| > | >> | > | > | > | > |
| > | >> | > | > | > | > | The Document handler looks suspicious to me
| > | >> | > | > | > | > | ...
| > | >> | > | > | > | > | perhaps
| > | >> | > | > | > | > | you
| > | >> | > | > | > | > | have
| > | >> | > | > | > | > | different sessions trying to add a document
| > | >> | > | > | > | > | handler...
| > | >> | > | > | > | > | this
| > | >> | > | > | > | > | rings
| > | >> | > | > | > | > | a
| > | >> | > | > | > | > | bell but I can't recall the details .. it
| > | >> | > | > | > | > | doesn't
| > | >> | > | > | > | > | seem
| > | >> | > | > | > | > | right
| > | >> | > | > | > | > | that
| > | >> | > | > | > | > | document handlers are being stored in the
| > | >> | > | > | > | > | session
| > | >> | > | > | > | > | dictionary
| > | >> | > | > | > | > | ...
| > | >> | > | > | > | > |
| > | >> | > | > | > | > | Dale
| > | >> | > | > | > | > |
| > | >> | > | > | > | > | ----- Original Message -----
| > | >> | > | > | > | > | | From: "Lawrence Kellogg" <[hidden email]>
| > | >> | > | > | > | > | | To: "GemStone Seaside beta discussion"
| > | >> | > | > | > | > | | <[hidden email]>
| > | >> | > | > | > | > | | Sent: Monday, July 9, 2012 4:57:33 PM
| > | >> | > | > | > | > | | Subject: Re: [GS/SS Beta] Far worse
| > | >> | > | > | > | > | | problems…
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | On Jul 9, 2012, at 7:51 PM, Dale Henrichs
| > | >> | > | > | > | > | | wrote:
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | Well let' see...
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | 440885505 is the oop of the object that
| > | >> | > | > | > | > | | you
| > | >> | > | > | > | > | | are
| > | >> | > | > | > | > | | getting
| > | >> | > | > | > | > | | the
| > | >> | > | > | > | > | | Rc-Write-Write Conflicts on. You can use:
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | Object _objectForOop: 440885505
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | Thanks, Dale.
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | So, this object is a RcKeyValueDiction
| > | >> | > | > | > | > | | with
| > | >> | > | > | > | > | | two
| > | >> | > | > | > | > | | keys
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | aWASession and a aWADocumentHandler.
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | It does not seem like my code, but perhaps
| > | >> | > | > | > | > | | it
| > | >> | > | > | > | > | | has
| > | >> | > | > | > | > | | something
| > | >> | > | > | > | > | | to
| > | >> | > | > | > | > | | do
| > | >> | > | > | > | > | | with the session stuff.
| > | >> | > | > | > | > | | Out of all the code I wrote, I didn't
| > | >> | > | > | > | > | | expect
| > | >> | > | > | > | > | | this
| > | >> | > | > | > | > | | conflict.
| > | >> | > | > | > | > | | ;-)
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | Any chance that this call to createCache
| > | >> | > | > | > | > | | in my
| > | >> | > | > | > | > | | session
| > | >> | > | > | > | > | | class is
| > | >> | > | > | > | > | | causing the problem?
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | createCache
| > | >> | > | > | > | > | | "To configure the continuation cache you
| > | >> | > | > | > | > | | must
| > | >> | > | > | > | > | | currently
| > | >> | > | > | > | > | | subclass
| > | >> | > | > | > | > | | WASession and override this method."
| > | >> | > | > | > | > | | | policy |
| > | >> | > | > | > | > | | policy := WARcLastAccessExpiryPolicy new.
| > | >> | > | > | > | > | | policy configuration at: #cacheTimeout
| > | >> | > | > | > | > | | put:
| > | >> | > | > | > | > | | 3600.
| > | >> | > | > | > | > | | ^ WACache new
| > | >> | > | > | > | > | | setExpiryPolicy: policy;
| > | >> | > | > | > | > | | yourself.
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | Larry
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | to inspect the object ... the fact that it
| > | >> | > | > | > | > | | is
| > | >> | > | > | > | > | | an Rc
| > | >> | > | > | > | > | | error
| > | >> | > | > | > | > | | means
| > | >> | > | > | > | > | | that
| > | >> | > | > | > | > | | the type of conflict is one that is not
| > | >> | > | > | > | > | | resolvable
| > | >> | > | > | > | > | | by
| > | >> | > | > | > | > | | retry
| > | >> | > | > | > | > | | ...
| > | >> | > | > | > | > | | for
| > | >> | > | > | > | > | | an Rc Dictionary that means two sessions
| > | >> | > | > | > | > | | are
| > | >> | > | > | > | > | | using
| > | >> | > | > | > | > | | the
| > | >> | > | > | > | > | | adding/removing the same key ... For an Rc
| > | >> | > | > | > | > | | set
| > | >> | > | > | > | > | | that
| > | >> | > | > | > | > | | means
| > | >> | > | > | > | > | | two
| > | >> | > | > | > | > | | sessions are adding removing the same
| > | >> | > | > | > | > | | object
| > | >> | > | > | > | > | | ...
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | Inspect the object and find out ...
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | ... I'll have more in a follow on mail...
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | Dale
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | | ----- Original Message -----
| > | >> | > | > | > | > | | | From: "Lawrence Kellogg" <
| > | >> | > | > | > | > | | | [hidden email] >
| > | >> | > | > | > | > | | | To: "GemStone Seaside beta discussion" <
| > | >> | > | > | > | > | | | [hidden email]
| > | >> | > | > | > | > | | | >
| > | >> | > | > | > | > | | | Sent: Monday, July 9, 2012 4:42:13 PM
| > | >> | > | > | > | > | | | Subject: [GS/SS Beta] Far worse
| > | >> | > | > | > | > | | | problems…
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | | Hello,
| > | >> | > | > | > | > | | | Actually, I have far worse problems than
| > | >> | > | > | > | > | | | just
| > | >> | > | > | > | > | | | hiding
| > | >> | > | > | > | > | | | changes
| > | >> | > | > | > | > | | | in
| > | >> | > | > | > | > | | | the
| > | >> | > | > | > | > | | | Patch Browser. I just noticed that the
| > | >> | > | > | > | > | | | log
| > | >> | > | > | > | > | | | files
| > | >> | > | > | > | > | | | for
| > | >> | > | > | > | > | | | my
| > | >> | > | > | > | > | | | Gems
| > | >> | > | > | > | > | | | are
| > | >> | > | > | > | > | | | spewing out this:
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | | etryFailure
| > | >> | > | > | > | > | | | Read-Write Conflicts...
| > | >> | > | > | > | > | | | Write-Write Conflicts...
| > | >> | > | > | > | > | | | Write-Dependency Conflicts...
| > | >> | > | > | > | > | | | Write-ReadLock Conflicts...
| > | >> | > | > | > | > | | | Write-WriteLock Conflicts...
| > | >> | > | > | > | > | | | Rc-Write-Write Conflicts...
| > | >> | > | > | > | > | | | 440885505
| > | >> | > | > | > | > | | | Synchronized-Commit Conflicts...
| > | >> | > | > | > | > | | | ----------- Commit failure - retrying
| > | >> | > | > | > | > | | | LOG
| > | >> | > | > | > | > | | | ENTRY:
| > | >> | > | > | > | > | | | aSymbolDictionary-----------
| > | >> | > | > | > | > | | | ----------- Unreportable ERROR
| > | >> | > | > | > | > | | | Encountered:
| > | >> | > | > | > | > | | | 2012-07-08T18:37:39.67828106880188-07:00InterpreterError
| > | >> | > | > | > | > | | | 2089: An
| > | >> | > | > | > | > | | | index range was specified for a
| > | >> | > | > | > | > | | | sequenceable
| > | >> | > | > | > | > | | | collection
| > | >> | > | > | > | > | | | with
| > | >> | > | > | > | > | | | the
| > | >> | > | > | > | > | | | starting index <1> greater than the
| > | >> | > | > | > | > | | | ending
| > | >> | > | > | > | > | | | index
| > | >> | > | > | > | > | | | <0>.-----------
| > | >> | > | > | > | > | | | ----------- Unreportable ERROR
| > | >> | > | > | > | > | | | Encountered:
| > | >> | > | > | > | > | | | 2012-07-08T18:39:39.71508598327637-07:00InterpreterError
| > | >> | > | > | > | > | | | 2089: An
| > | >> | > | > | > | > | | | index range was specified for a
| > | >> | > | > | > | > | | | sequenceable
| > | >> | > | > | > | > | | | collection
| > | >> | > | > | > | > | | | with
| > | >> | > | > | > | > | | | the
| > | >> | > | > | > | > | | | starting index <1> greater than the
| > | >> | > | > | > | > | | | ending
| > | >> | > | > | > | > | | | index
| > | >> | > | > | > | > | | | <0>.-----------
| > | >> | > | > | > | > | | | ----------- Unreportable ERROR
| > | >> | > | > | > | > | | | Encountered:
| > | >> | > | > | > | > | | | 2012-07-08T18:41:39.78475308418274-07:00InterpreterError
| > | >> | > | > | > | > | | | 2089: An
| > | >> | > | > | > | > | | | index range was specified for a
| > | >> | > | > | > | > | | | sequenceable
| > | >> | > | > | > | > | | | collection
| > | >> | > | > | > | > | | | with
| > | >> | > | > | > | > | | | the
| > | >> | > | > | > | > | | | starting index <1> greater than the
| > | >> | > | > | > | > | | | ending
| > | >> | > | > | > | > | | | index
| > | >> | > | > | > | > | | | <0>.-----------
| > | >> | > | > | > | > | | | ----------- Unreportable ERROR
| > | >> | > | > | > | > | | | Encountered:
| > | >> | > | > | > | > | | | 2012-07-08T18:43:39.84294891357422-07:00InterpreterError
| > | >> | > | > | > | > | | | 2089: An
| > | >> | > | > | > | > | | | index range was specified for a
| > | >> | > | > | > | > | | | sequenceable
| > | >> | > | > | > | > | | | collection
| > | >> | > | > | > | > | | | with
| > | >> | > | > | > | > | | | the
| > | >> | > | > | > | > | | | starting index <1> greater than the
| > | >> | > | > | > | > | | | ending
| > | >> | > | > | > | > | | | index
| > | >> | > | > | > | > | | | <0>.-----------
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | | on and on and on...
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | | Help! What do I do? I took care of a
| > | >> | > | > | > | > | | | walk
| > | >> | > | > | > | > | | | back I
| > | >> | > | > | > | > | | | add
| > | >> | > | > | > | > | | | and
| > | >> | > | > | > | > | | | actually
| > | >> | > | > | > | > | | | have no idea what sequenceable
| > | >> | > | > | > | > | | | collection is
| > | >> | > | > | > | > | | | causing
| > | >> | > | > | > | > | | | this
| > | >> | > | > | > | > | | | problem.
| > | >> | > | > | > | > | | | My system is up and running fine but all
| > | >> | > | > | > | > | | | of
| > | >> | > | > | > | > | | | these
| > | >> | > | > | > | > | | | errors
| > | >> | > | > | > | > | | | are
| > | >> | > | > | > | > | | | being
| > | >> | > | > | > | > | | | thrown.
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | |
| > | >> | > | > | > | > | | | Larry
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > | |
| > | >> | > | > | > | > |
| > | >> | > | > | > |
| > | >> | > | > | > |
| > | >> | > | > |
| > | >> | > | > |
| > | >> | > |
| > | >> | > |
| > | >> |
| > | >> |
| > | >
| > |
| > |
|
|