dkh 3/8/2017 16:56 ------------------ xxx_4_0 [25e936f] Start moving down the new path: 78 run, 61 passes, 2 expected defects, 12 failures, 2 errors, 1 unexpected passes 78 tests 78 run 61 passes 2 expected defects 12 failures BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateA BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testIndexUpdateTroubleA BtreePlusEqualityTests>>#testIndexUpdateTroubleB BtreePlusEqualityTests>>#testIndexUpdateTroubleC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 2 errors BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD_ModifyingInstVarAtOffset_to_ 1 unexpected passes dkh 3/8/2017 16:42 ------------------ Major shift in emphasis ...I've been approaching the problem using an incorrect algorithm for finding the root elements ... filtering should not be necessary and I need to adjust how the removal and adds are being done ... I'm missing some of the entries... In test testMultiEnumeratedIndexUpdateD, for the key|value.key|value.key|value terms, and a given root object: -60.00->60.00->-60.00->-60.00->-60.00->60.00->-60.00->-60.00 I expect to see `2 raisedTo: 3` entries in the btree. 6 of the values will be sorted into the -60.00 section and 2 of them are in the 60.00 section ... right now we are not correctly removing and re-adding all 8 entries that leads to all sorts of gyrations ... I also have reason to believe that the audit should be used, because it may actually give us useful information ... see /homeComposition/gs/bugs/bug46550/enumeratedIndexUpdateD_0 for detailed analysis. Part of the equation is to use add: instead of _intersect: and _union: when collecting the results ... or be smarter since we really should be removing 6 of the entries in the above case ... dkh 3/8/2017 14:51 ------------------ BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD (/homeComposition/gs/bugs/bug46550/enumeratedIndexUpdateD_0) expected: 1@ -> -60.00->60.00->-63.00->63.00->-60.00->60.00->-63.00->63.00 2@ -> -60.00->60.00->-63.00->63.00->-60.00->60.00->-63.00->63.00 3@ -> -63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00->-63.00 4@ -> -63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00->-63.00 result: 1@ -> -60.00->60.00->-63.00->63.00->-60.00->60.00->-63.00->63.00 2@ -> -60.00->60.00->-63.00->63.00->-60.00->60.00->-63.00->63.00 3@ -> -60.00->60.00->-63.00->63.00->-60.00->60.00->-63.00->63.00 4@ -> -60.00->60.00->-63.00->63.00->-60.00->60.00->-63.00->63.00 5@ -> -63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00->-63.00 6@ -> -63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00->-63.00 dkh 3/8/2017 14:49 ------------------ xxx_4[f864c01] BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf, testMultiEnumeratedIndexUpdateF, and testMultiEnumeratedIndexUpdateC passing ... testMultiEnumeratedIndexUpdateD and testMultiEnumeratedIndexUpdateE failing 77 run, 69 passes, 2 expected defects, 5 failures, 0 errors, 1 unexpected passes 77 tests 77 run 69 passes 2 expected defects 5 failures *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 0 errors 1 unexpected passes dkh 3/8/2017 12:03 ------------------ BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf expected: 1@ -> 102->102->'2038'->-2038.00->2038.00 2@ -> 2038->2038->'2038'->2038.00->-2038.00 3@ -> 100->100->'2038'->2038.00->-2038.00 -->4@ -> 100->100->'2038'->2038.00->-2038.00 5@ -> 101->101->'2038'->2038.00->-2038.00 result: 1@ -> 102->102->'2038'->-2038.00->2038.00 2@ -> 100->100->'2038'->2038.00->-2038.00 3@ -> 2038->2038->'2038'->2038.00->-2038.00 4@ -> 101->101->'2038'->2038.00->-2038.00 ==== the factor should be on a reachable set basis ... ==== dkh 3/8/2017 11:57 ------------------ xxx_4[148632] patch error in BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf 77 run, 68 passes, 2 expected defects, 6 failures, 0 errors, 1 unexpected passes 77 tests 77 run 68 passes 2 expected defects 6 failures BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 0 errors 1 unexpected passes dkh 3/8/2017 11:33 ------------------ xxx_4[7e1fab9] get rid of nscNum limit. 77 run, 70 passes, 2 expected defects, 2 failures, 2 errors, 1 unexpected passes 77 tests 77 run 70 passes 2 expected defects 2 failures BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 2 errors BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf_ModifyingInstVarAtOffset_to 1 unexpected passes Actually I like this result. dkh 3/8/2017 11:26 ------------------ testMultiEnumeratedIndexUpdateF ... expected: 1@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 2@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 3@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 4@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 result: 1@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 2@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 3@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 at end of day, I actually think that it is wrong to limit to nscNum ... dkh 3/8/2017 11:23 ------------------ 77 run, 69 passes, 2 expected defects, 5 failures, 0 errors, 1 unexpected passes 77 tests 77 run ['done'] 69 passes 2 expected defects 5 failures *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 0 errors 1 unexpected passes Both BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC and BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF pass with `occurrences := (num * factor)` and `referenceCount + sz`!!! dkh 3/8/2017 11:14 ------------------ revert to d95f320? dkh 3/8/2017 11:09 ------------------ BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF pass with `occurrences := (num * factor)` .. others still fail ... enumerated guys with identity involved are not surprising ... I think that the limit for nsc size is required dkh 3/8/2017 10:27 ------------------ testMultiEnumeratedIndexUpdateF ... expected: 1@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 2@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 3@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 4@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 result: 1@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 2@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 3@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 reachableRoots: 1@ -> -60.00->-60.00->60.00->-60.00->-60.00->-60.00->60.00->-60.00 2@ -> -60.00->-60.00->60.00->-60.00->-60.00->-60.00->60.00->-60.00 dkh 3/8/2017 10:22 ------------------ xxx_4 [67bf723] Back to all MultiEnumerated tests failing ... but I like the algorithm better ... less magical, mystery adjustments ... 77 run, 67 passes, 2 expected defects, 7 failures, 0 errors, 1 unexpected passes 77 tests 77 run 67 passes 2 expected defects 7 failures BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 0 errors 1 unexpected passes dkh 3/7/2017 18:26 ------------------ xxx_4 [d95f320] One more test passing: BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 77 run, 69 passes, 2 expected defects, 5 failures, 0 errors, 1 unexpected passes 77 tests 77 run ['done'] 69 passes 2 expected defects 5 failures *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 0 errors 1 unexpected passes dkh 3/7/2017 16:57 ------------------ testMultiEnumeratedIndexUpdateF ---- expected: 1@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 2@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 3@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 4@ -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 result: 1@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 2@ -> -63.00->-63.00->63.00->-63.00->-63.00->-63.00->63.00->-63.00 "victim key value: commonAssoc." commonAssoc -> 63.00->-63.00 commonBinding -> 63.00 query -> (each.key|value.key|value.key|value = x) victim -> -60.00->-60.00->63.00->-63.00->-60.00->-60.00->63.00->-63.00 key@ -> -60.00->-60.00->63.00->-63.00 value@ -> -60.00->-60.00->63.00->-63.00 I think there is something else wrong here ... the bruteforce query is finding it but the btree query is not ... the update that was done (via tracing the update:at:to: added the -63.00 as new value not 63.00 and that may have been missed completely .... an assoc is being added and the key of the assoc us 63.00 so in addition to the -63.00 key being added, there should be a 63.00 being added... I think there is something wrong at the level of GsEnumeratedPathTerm>>updateBtreeFor:at:oldValue:to:roots: where the children are being added and removed ... not enough action is going on! dkh 3/7/2017 16:37 ------------------ testEnumeratedIndexUpdateLeaf passing, but now we're seeing deplist failures return: 77 run, 68 passes, 2 expected defects, 6 failures, 0 errors, 1 unexpected passes 77 tests 77 run ['done'] 68 passes 2 expected defects 6 failures *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 0 errors 1 unexpected passes dkh 3/7/2017 16:22 ------------------ add new test BtreeTests>>testMultiEnumeratedIndexUpdateF (failing) 77 run, 69 passes, 2 expected defects, 3 failures, 2 errors, 1 unexpected passes 77 tests 77 run ['done'] 69 passes 2 expected defects 3 failures BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateF 2 errors BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf_ModifyingInstVarAtOffset_to 1 unexpected passes dkh 3/7/2017 15:17 ------------------ commit: xxx_4 [eb16cdb] better version of GsEnumeratedPathTerm>>findReachableRootsFor:ivOffset: ... beats test results for xx_4[32cfb5a, f9dec56] (finally!!): 75 run, 68 passes, 2 expected defects, 2 failures, 2 errors, 1 unexpected passes 75 tests 75 run ['done'] 68 passes 2 expected defects 2 failures BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 2 errors BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf_ModifyingInstVarAtOffset_to 1 unexpected passes dkh 3/7/2017 14:41 ------------------ beginning revamp GsEnumeratedPathTerm>>findReachableRootsFor:ivOffset:. 75 run, 62 passes, 2 expected defects, 10 failures, 0 errors, 1 unexpected passes 75 tests 75 run ['done'] 62 passes BtreeTests>>testClassicEnumeratedIndexUpdateD BtreeTests>>testEnumeratedIndexUpdate BtreeTests>>testMultiEnumeratedIndexUpdateA 2 expected defects 10 failures BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateA BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 0 errors 1 unexpected passes dkh 3/7/2017 12:21 ------------------ The last time that BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB passed, we were reducing the results to a set from a bag ... the reachable roots for this test is a bag with two occurrences for the same object and there "should only be one occurrence in the bag". In GsEnumeratedPathTerm>>findReachableRootsFor:ivOffset:, I need to call PathTerm>>findReachableRootsFor:ivOffset: for EACH of the enumerated objects. That way I can account for non-identical terms that give same reachable objects. dkh 3/7/2017 12:13 ------------------ and the current winner is: factor := referenceCount * (enumeratedSet occurrencesOf: targetObject). 74 run, 64 passes, 3 expected defects, 7 failures, 0 errors, 0 unexpected passes 74 tests 74 run ['done'] 64 passes 3 expected defects 7 failures BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB[] *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC[] BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD[factor := referenceCount * enumeratedSet size] BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE[] 0 errors 0 unexpected passes dkh 3/7/2017 11:38 ------------------ factor := 1 74 run, 62 passes, 3 expected defects, 9 failures, 0 errors, 0 unexpected passes 74 tests 74 run ['done'] 62 passes 3 expected defects 9 failures BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateA BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 0 errors 0 unexpected passes -->factor := referenceCount * (enumeratedSet occurrencesOf: targetObject). 74 run, 64 passes, 3 expected defects, 7 failures, 0 errors, 0 unexpected passes 74 tests 74 run ['done'] 64 passes 3 expected defects 7 failures BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB[] *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC[] BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD[factor := referenceCount * enumeratedSet size] BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE[] 0 errors 0 unexpected passes factor := referenceCount * enumeratedSet size. 74 run, 63 passes, 3 expected defects, 2 failures, 6 errors, 0 unexpected passes 74 tests 74 run ['done'] 63 passes 3 expected defects 2 failures BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 6 errors BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateA BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB_ModifyingInstVarAtOffset_to_ BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf_ModifyingInstVarAtOffset_to 0 unexpected passes dkh 3/7/2017 10:26 ------------------ address test errors: 74 run, 64 passes, 3 expected defects, 7 failures, 0 errors, 0 unexpected passes 74 tests 74 run ['done'] 64 passes 3 expected defects 7 failures BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 0 errors 0 unexpected passes dkh 3/7/2017 10:14 ------------------ enable the new approach (GsEnumeratedPathTerm>>addMappingsForObject:at:root:logging: and GsEnumeratedPathTerm>>removeMappingsFor:at:root:lastOne:logging:): 74 run, 61 passes, 3 expected defects, 6 failures, 4 errors, 0 unexpected passes 74 tests 74 run ['done'] 61 passes 3 expected defects 6 failures *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 4 errors BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateA BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB_ModifyingInstVarAtOffset_to_ BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateC 0 unexpected passes dkh 3/7/2017 10:06 ------------------ branch xxx_4 [f9dec56] Playing with a new approach (commented out right now: GsEnumeratedPathTerm>>updateBtreeFor:at:oldValue:to:roots:) and create a couple tests with slightly different variants of testMultiEnumeratedIndexUpdateC ... testMultiEnumeratedIndexUpdateB is a dep list failure: 74 run, 65 passes, 3 expected defects, 6 failures, 0 errors, 0 unexpected passes 74 tests 74 run ['done'] 65 passes 3 expected defects 6 failures *BtreePlusEqualityTests>>#testIndexRemoveAll *BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements *BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateE 0 errors 0 unexpected passes * - dep list failures dkh 3/6/2017 15:17 ------------------ The current approach is not completely without merit ... I think that it could be more correct, than existing, since the ivOffset is being honored. save on a new branch: xxx_4 ... dkh 3/6/2017 14:58 ------------------ new test (BtreeTests>>testMultiEnumeratedIndexUpdateC ), new failure, what's new? 70 run, 65 passes, 1 expected defects, 4 failures, 0 errors, 0 unexpected passes 70 tests 70 run ['done'] 65 passes 1 expected defects 4 failures BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateC 0 errors 0 unexpected passes dkh 3/6/2017 13:44 ------------------ 68 run, 64 passes, 1 expected defects, 3 failures, 0 errors, 0 unexpected passes 68 tests 68 run ['done'] 64 passes 1 expected defects 3 failures BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB 0 errors 0 unexpected passes dkh 3/6/2017 12:12 ------------------ xxx_3_3_1 [7803820] Using _intersect: to slightly better effect ... error turned into failure (testMultiEnumeratedIndexUpdateB) 68 run, 62 passes, 1 expected defects, 3 failures, 2 errors, 0 unexpected passes 68 tests 68 run ['done'] 62 passes 1 expected defects 3 failures BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB 2 errors BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD_ModifyingInstVarAtOffset_to_ 0 unexpected passes dkh 3/6/2017 11:52 ------------------ in GsPathTerm>>findRootObjectsFor:pathterm:key:value: we are finding more root objects that there are entries in the btree ... I don't think we can have more entries that the count in the btree! dkh 3/6/2017 10:58 ------------------ xxx_3_3_0 [47e88ab] add GsEnumeratedPathTerm>>findReachableRootsFor:at: to filter reachable roots number of roots accessible from the ivOffset of the enumerated term. BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf is now passing. 68 run, 61 passes, 1 expected defects, 2 failures, 4 errors, 0 unexpected passes 68 tests 68 run ['done'] 61 passes 1 expected defects 2 failures BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements 4 errors BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD_ModifyingInstVarAtOffset_to_ BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB_ModifyingInstVarAtOffset_to 0 unexpected passes dkh 3/6/2017 09:47 ------------------ GsPathTerm>>findReachableRootsFor:at: is the spot to filter out duplicates (apply quota?), not findRootObjectsFor:ivOffset:pathterm:key:value: dkh 3/3/2017 18:08 ------------------ xxx_3_3 [5d1a340] Implement GsPathTerm>>findRootObjectsFor:ivOffset:pathterm:key:value: and GsEnumeratedPathTerm>>findRootObjectsFor:ivOffset:pathterm:key:value: ... GsEnumeratedPathTerm variant needs to be adjusted to be when returning rootObjects ... 68 run, 60 passes, 1 expected defects, 3 failures, 4 errors, 0 unexpected passes 68 tests 68 run ['done'] 60 passes 1 expected defects 3 failures BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements 4 errors BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD_ModifyingInstVarAtOffset_to_ BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB_ModifyingInstVarAtOffset_to 0 unexpected passes dkh 3/3/2017 17:54 ------------------ Somehow we need to recognize that only half of the objects should be moved ... traversal from the iv slot seems to be the key ... our query in GsPathTerm>>findRootObjectsFor:pathterm:key:value: is not selective enough ... the path is "value.value.value.key|value" which should give us the ability to be selective ... perhaps passing the ivOffset into GsPathTerm>>findRootObjectsFor:pathterm:key:value: for enumerated guys. dkh 3/3/2017 17:21 ------------------ BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf ------------ ------------ testEnumeratedIndexUpdateLeaf {commonBinding. expected. result.} 1@ -> 2038.00 2@ -> aBag( 102->102->'2038'->-2038.00->2038.00, 2038->2038->'2038'->2038.00->-2038.00, 100->100->'2038'->2038.00->-2038.00, 100->100->'2038'->... 1@ -> 102->102->'2038'->-2038.00->2038.00 2@ -> 2038->2038->'2038'->2038.00->-2038.00 3@ -> 100->100->'2038'->2038.00->-2038.00 4@ -> 100->100->'2038'->2038.00->-2038.00 5@ -> 101->101->'2038'->2038.00->-2038.00 3@ -> aBag( 102->102->'2038'->-2038.00->2038.00) --- none of the objects that were supposed to be updated got changed. 102->102->'2038'->-2038.00->2038.00 is a copy at each level and is not touched by modification tracking ...--- and the changes simply reversed the key and value: common value key: oldValue. common value value: oldKey. Clearly the btree was properly updated the first time and not updated the second time as all of the entries were stranded in the negative land. --- 1@ -> 2038.00->-2038.00 2@ -> -2038.00 3@ -> 101->101->'2038'->2038.00->-2038.00 4@ -> 2038.00->-2038.00 5@ -> -2038.00 6@ -> 101->101->'2038'->2038.00->-2038.00 7@ -> 2038.00->-2038.00 8@ -> -2038.00 9@ -> 100->100->'2038'->2038.00->-2038.00 10@ -> 2038.00->-2038.00 11@ -> -2038.00 12@ -> 100->100->'2038'->2038.00->-2038.00 13@ -> 2038.00->-2038.00 14@ -> -2038.00 15@ -> 100->100->'2038'->2038.00->-2038.00 16@ -> 2038.00->-2038.00 17@ -> -2038.00 18@ -> 100->100->'2038'->2038.00->-2038.00 19@ -> 2038.00->-2038.00 20@ -> -2038.00 21@ -> 2038->2038->'2038'->2038.00->-2038.00 22@ -> 2038.00->-2038.00 23@ -> -2038.00 24@ -> 2038->2038->'2038'->2038.00->-2038.00 25@ -> -2038.00->2038.00 26@ -> -2038.00 27@ -> 102->102->'2038'->-2038.00->2038.00 --- 1507@ -> 2037.00->-2037.00 1508@ -> 2037.00 1509@ -> 2037->2037->'2037'->2037.00->-2037.00 1510@ -> -2038.00->2038.00 1511@ -> 2038.00 1512@ -> 102->102->'2038'->-2038.00->2038.00 --- first the key is changed from negative to positive: --- updateBtreeDirectFor: {anObject. ivOffset. oldValue. newValue. reachableRoots} 1@ -> 2038.00->-2038.00 2@ -> 1 3@ -> -2038.00 4@ -> 2038.00 5@ -> anIdentityBag( 101->101->'2038'->2038.00->-2038.00, 100->100->'2038'->2038.00->-2038.00, 100->100->'2038'->2038.00->-2038.00, 2038->2038-... 1@ -> 101->101->'2038'->2038.00->-2038.00 2@ -> 100->100->'2038'->2038.00->-2038.00 3@ -> 100->100->'2038'->2038.00->-2038.00 4@ -> 2038->2038->'2038'->2038.00->-2038.00 --- and from the above reachable roots, it really appears that the 4 objects were re-inserted in the positive region of the btree properly ... so it's the second update that has gone awry... btree immediately before the fateful update -- half in negative range and half in positive range: 1@ -> -2038.00->2038.00 2@ -> -2038.00 3@ -> 102->102->'2038'->-2038.00->2038.00 4@ -> -2038.00->2038.00 5@ -> -2038.00 6@ -> 100->100->'2038'->-2038.00->2038.00 7@ -> -2038.00->2038.00 8@ -> -2038.00 9@ -> 100->100->'2038'->-2038.00->2038.00 10@ -> -2038.00->2038.00 11@ -> -2038.00 12@ -> 101->101->'2038'->-2038.00->2038.00 13@ -> -2038.00->2038.00 14@ -> -2038.00 15@ -> 2038->2038->'2038'->-2038.00->2038.00 --- 1138@ -> -2038.00->2038.00 1139@ -> 2038.00 1140@ -> 102->102->'2038'->-2038.00->2038.00 1141@ -> -2038.00->2038.00 1142@ -> 2038.00 1143@ -> 100->100->'2038'->-2038.00->2038.00 1144@ -> -2038.00->2038.00 1145@ -> 2038.00 1146@ -> 100->100->'2038'->-2038.00->2038.00 1147@ -> -2038.00->2038.00 1148@ -> 2038.00 1149@ -> 101->101->'2038'->-2038.00->2038.00 1150@ -> -2038.00->2038.00 1151@ -> 2038.00 1152@ -> 2038->2038->'2038'->-2038.00->2038.00 --- Here's the btree immediately after the key is updated (first update): 1@ -> -2038.00->2038.00 2@ -> -2038.00 3@ -> 102->102->'2038'->-2038.00->2038.00 --- 1387@ -> 2038.00->2038.00 1388@ -> 2038.00 1389@ -> 101->101->'2038'->2038.00->2038.00 1390@ -> 2038.00->2038.00 1391@ -> 2038.00 1392@ -> 101->101->'2038'->2038.00->2038.00 1393@ -> 2038.00->2038.00 1394@ -> 2038.00 1395@ -> 100->100->'2038'->2038.00->2038.00 1396@ -> 2038.00->2038.00 1397@ -> 2038.00 1398@ -> 100->100->'2038'->2038.00->2038.00 1399@ -> 2038.00->2038.00 1400@ -> 2038.00 1401@ -> 100->100->'2038'->2038.00->2038.00 1402@ -> 2038.00->2038.00 1403@ -> 2038.00 1404@ -> 100->100->'2038'->2038.00->2038.00 1405@ -> 2038.00->2038.00 1406@ -> 2038.00 1407@ -> 2038->2038->'2038'->2038.00->2038.00 1408@ -> 2038.00->2038.00 1409@ -> 2038.00 1410@ -> 2038->2038->'2038'->2038.00->2038.00 1411@ -> -2038.00->2038.00 1412@ -> 2038.00 1413@ -> 102->102->'2038'->-2038.00->2038.00 and at this point it is obvious that all of the entries were properly moved to the positive regions. Then when the second update occurred ALL of the entries were moved to the negative region instead of half of them ... The reachable objects for the second update confirms this conclusion: 1@ -> 101->101->'2038'->2038.00->-2038.00 2@ -> 101->101->'2038'->2038.00->-2038.00 3@ -> 100->100->'2038'->2038.00->-2038.00 4@ -> 100->100->'2038'->2038.00->-2038.00 5@ -> 100->100->'2038'->2038.00->-2038.00 6@ -> 100->100->'2038'->2038.00->-2038.00 7@ -> 2038->2038->'2038'->2038.00->-2038.00 8@ -> 2038->2038->'2038'->2038.00->-2038.00 ------- Somehow we need to recognize that only half of the objects should be moved ... traversal from the iv slot seems to be the key dkh 3/3/2017 16:35 ------------------ xxx_3_1 [d290860] different set of test failures, but I am intrigued that the the common factor is that there are multiple entries in the btree and/or nsc ... interesting cross-section of failures and passes: 68 run, 58 passes, 1 expected defects, 9 failures, 0 errors, 0 unexpected passes 68 tests 68 run ['done'] 58 passes 1 expected defects 9 failures BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneAndUpdate BtreePlusEqualityTests>>#testIndexRemoveOneAndUpdateAlternate BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements BtreePlusEqualityTests>>#testIndexUpdateAlternate BtreePlusEqualityTests>>#testIndexUpdateSimple BtreePlusEqualityTests>>#testIndexUpdateTroubleC BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB 0 errors 0 unexpected passes dkh 3/3/2017 14:43 ------------------ xxx_3_0 [f337717] switch to using IdentityBag instead of IdentitySet to do counting ... I think it is a better approach, however, more test failures (which may be a good thing ... most if not all of the enumerated guys are blowing up): 68 run, 56 passes, 1 expected defects, 3 failures, 8 errors, 0 unexpected passes 68 tests 68 run ['done'] 56 passes 1 expected defects 3 failures BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testIndexRemoveAll BtreePlusEqualityTests>>#testIndexRemoveOneOnNscElements 8 errors BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateA BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateB_ModifyingInstVarAtOffset_to_ BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateC BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD BtreePlusEqualityTests>>#testClassicEnumeratedIndexUpdateD_ModifyingInstVarAtOffset_to_ BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB_ModifyingInstVarAtOffset_to 0 unexpected passes dkh 3/3/2017 09:35 ------------------ xxx_3 [ab19384] dkh 3/2/2017 17:26 ------------------ Turns out that updateBtreeFor:... can get called and the reachable roots turn out to be empty (after updateBtreeDirectFor: gets done ... so it is worth combining the loops and/or using the original set. dkh 3/2/2017 17:00 ------------------ GsPathTerm>>updateBtreeDirectForX:at:oldValue:to: and GsPathTerm>>updateBtreeDirectForY:at:oldValue:to: allows me to flip back and forth between the two implementations and test results ... Probably easier to debug testMultiEnumeratedIndexUpdateB since only one update is involved ... testEnumeratedIndexUpdateLeaf swaps keys and values and thats harder to debug inline. dkh 3/2/2017 15:46 ------------------ Confirm that we're flip flopping test failure/mode between b2ee059 and ba1f3c8). master branch commit f5ae70b (ba1f3c8 plust test mods from b2ee059) dkh 3/2/2017 15:38 ------------------ flip flop on test failures ... testMultiEnumeratedIndexUpdateB dependendency failure; testEnumeratedIndexUpdateLeaf result set failure: branch xxx_2 (b2ee059) 68 run, 65 passes, 1 expected defects, 2 failures, 0 errors, 0 unexpected passes 68 tests 68 run ['done'] 65 passes 1 expected defects 2 failures BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB 0 errors 0 unexpected passes dkh 3/1/2017 19:19 ------------------ looks like I've got the final fix for GsPathTerm>>updateBtreeDirectFor:at:oldValue:to:. BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf fails with an unbalanced Dependency list error ... and BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB fails with a wrong result set, but I suspect that it is also a dependency list error ... getting very close: branch master (ba1f3c8) 68 run, 65 passes, 1 expected defects, 2 failures, 0 errors, 0 unexpected passes 68 tests 68 run ['done'] 65 passes 1 expected defects 2 failures BtreePlusEqualityTests>>#testEnumeratedIndexUpdateLeaf BtreePlusEqualityTests>>#testMultiEnumeratedIndexUpdateB 0 errors 0 unexpected passes dkh 3/2/2017 07:32 ------------------ remainingElementSize for stream ... provide a count of number of elements remaining ... I think that we can get an accurate count of the remaining elements without costing too much ... signal a GsIndexError instead of returning arrays .. at least on the new index side ... dkh 3/2/2017 02:52 ------------------ new index option: buildModificationCache. the modification cache is a reverse lookup cache used to resolve the root object given an object along the term path (only used when an object along the path is modified)... caches are built when you get a pages worth of keys with the same value during index creation and addition of objects to nsc. When an object modification occurs and there a pages worth of keys with the same value the cache is built as a side effect of doing the lookup during GsPathTerm>>findReachableRootsFor:at:. The logic in GsPathTerm>>findReachableRootsFor:at: should be moved to the index/btree root ... key and key value btrees don't need a cache, since the key is the root in both cases and no lookup is needed ... cache always created on modification when there are a large number of keys with the same value .... more efficient to build index during index creation and nsc addition ... less surprising transaction size ... if all values are the same for entire collection then a very large write set will be created if a value is updated ... OTOH if you never intend to modify instance variables on indexed objects, then there is no reason to build the modification cache. the cache is simply a btree whose key is an object and whose value is the object's parent ... and of course the root object is a bag of root objects --- which means that with the cache we can immediately find the root objects and root object counts for any object along the path ... right now the cache is envisioned to be on an index by index basis and only for those objects with a large number of equivalent keys ... the brute force method is not that expensive for small numbers of root objects ... dkh 3/1/2017 09:34 ------------------ Index on each term of enumerated path term examples ... sprinkle some nils in the different enumerated path term slots ... identical partial paths and copied partial paths ... dkh 2/15/2017 17:44 ------------------- primitive created (1038): tode 1 > logout tode 1 > ./daveIndex --new --queryResultSetWithIndexes=date Test starting TestCase GsQuery Result Set, Index - Date Predicate QueryString: date1 <= each.adate <= date2 TestCase GsQuery Result Set, Index - Date Predicate. Results:266539 elapsedTime 262 marksweeps 0 objectsRead 576 cpuTime 262 pageReads 0 objectsRefreshed 0 scavenges 3 tode 1 > ./daveIndex --new --queryResultSetWithIndexes=date Test starting TestCase GsQuery Result Set, Index - Date Predicate QueryString: date1 <= each.adate <= date2 TestCase GsQuery Result Set, Index - Date Predicate. Results:266539 elapsedTime 258 marksweeps 0 objectsRead 2 cpuTime 259 pageReads 2 objectsRefreshed 0 scavenges 3 tode 1 > logout tode 1 > ./go [74 sz:0 SmallInteger] 9 tode 1 > ./daveIndex --new --queryWithIndexes=date Test starting TestCase GsQuery, Index - Date Predicate QueryString: date1 <= each.adate <= date2 TestCase GsQuery, Index - Date Predicate. Results:266539 elapsedTime 579 marksweeps 0 objectsRead 267113 cpuTime 579 pageReads 2510 objectsRefreshed 0 scavenges 0 tode 1 > ./daveIndex --new --queryWithIndexes=date Test starting TestCase GsQuery, Index - Date Predicate QueryString: date1 <= each.adate <= date2 TestCase GsQuery, Index - Date Predicate. Results:266539 elapsedTime 58 marksweeps 0 objectsRead 0 cpuTime 59 pageReads 0 objectsRefreshed 0 scavenges 0 tode 1 > logout tode 1 > ./daveIndex --noIndexes=date Test starting TestCase SimpleDo - Date Predicate TestCase SimpleDo - Date Predicate. Results:266539 elapsedTime 4641 marksweeps 0 objectsRead 8002074 cpuTime 4637 pageReads 26946 objectsRefreshed 0 scavenges 0 tode 1 > ./daveIndex --noIndexes=date Test starting TestCase SimpleDo - Date Predicate TestCase SimpleDo - Date Predicate. Results:266539 elapsedTime 658 marksweeps 0 objectsRead 0 cpuTime 658 pageReads 0 objectsRefreshed 0 scavenges 0 dkh 2/7/2017 17:40 ------------------ []1. Ready to run existing tests again: 237 run, 237 passes, 0 expected defects, 0 failures, 0 errors, 0 unexpected passes [2. run Marten's stuff and see if we're still going fast ... - resolve the streaming stuff once and for all ... sent mail to Marten making case that do block-based queries are going to perform as good as the stream-based queries without all of the work ... will defer - 5 predicate query (warm) is 20% fasterthat 2/3/17 timings. cold gem is 10% faster. 3. Check out Marten's bug ... ---> 4. makeNsc queries 5. basic* indexes needed --- yes - revisit the encoding schemes - encodedIndex as an option ... large result sets no value to using encoding ... small result sets lots of value 6. SimpleDomainTest test case based on a large collection with a mulit-level path and lots of single element lookups for random values ... to validate Basic* use cases 7. option for allowing nil, without incurring cost of per element comparisons (string and float only)... perhaps just fine tune the implementation so that per element comparisons can be skipped without while allowing nils ... 8. PathSorter and friends ... is there a better way? 9. make sure that optimizedComparison uses the basic btree exclusion algoritm ... need constraints to be enforced to make optimized comparison to work ... tests! tests! tests! dkh 2/13/2017 11:57 ------------------- installBtree, testBtree scripts written .... need updateBtree which simply loads the Btree project. dkh 1/24/2017 17:08 -------------------- []1. BtreePlusLeafKeyValueNode have value for indexes on first path term ... XX2. Consider a new tree of evaluator that use BtreeQuerySpec to run evaluation when no index is available. []3. Customize implementation of BtreePlusReadStream implementation to use leaf nodes only ... - now running 4x faster than brute force ... odd that brute force seemed to slow down - simulating homogenousEquality ... []4. Optimize non-indexed GsQuery evaluation - added GsPreparedRangeQueryPredicate that spec things up considerably ... - additional optimizations would require optimizedComparison on a predicate by predicate basis (as can be done with indexes) --- but not practical expecially considering that if they want speed, add an index ... 5. streams on identity indexes 6. profile all queries ... 7. new prims for make nsc and intersect/union without bag semantics... 8. optimizedComparisons []- use the actual Smalltalk selectors for comparison, rather thant the _idxForCompare variants, - we are competing for speed with the original article, so it's not a question of correctness. []- GsPrepared*Predicate (GsPreparedRangeQueryPredicate) ... caching of values for execution inside of tight loops. - some variant of GsBtreeComparisonQuerySpec? - there are path traversal primitives as well ... []- strictly homogenous attribute in index (index option?) - _canCompare.... or _canStrictlyCompare .... - Symbols and Strings are not strictly homogenous - Date, DateAndTime ... all of the Basic index classes - NAN and Floars are not strictly homogenous - compareForSort and compreForCompare can give different answers - class attribute as well hasHomogenousEquality answers false if _idxForSort and _idxForCompare are not implemented the same - _canStrictlyCompare on instance side explicitly excludes the non-homogenous cases - makeNsc optimizations that grab root object and put directly in the makensc object - test on each result element only needed for unicode and float indexes even without optimized comparisons XX9. useInstanceVariableAccessor query option ... message send instead of structural lookup (if performance advantage) []10. Basic btree subclasses 11. Rc variant of BtreePlusRoot ... I think []12. enhanced streaming capabilities (multi-predicate, non-indexed) await Marten's response before submitting feature request. []13. _idxNativeOptimizedComparison ... many objects (basic classes except CharacterCollection and Float) can naturally use optimizedComparisons without requiring an explicit option ... class side method some way of telling no index queries to use optimized comparisons ... 14. Set-valued, enumerated, selector and optional path term support