Can I avoid ReversedRangeIndexReadStream >> atEnd ?

Mariano,

The costly code is in RangeIndexReadStream>>_atEndForStream: (when aBool is true) and that code can be safely skipped if the values in the index will not include nils, or if your index is on a CharacterCollection and you do not mix Strings and Symbols or if your index is on a Float and the values do not include NaNs.

In 3.4 we do this automatically when you create a btree plus index with optmiizedComparison index option ... the 3.4 stream operations have been optimized in other ways as well --- 3.4 is due to be released very soon ...

Dale

On 10/25/2017 07:01 AM, Mariano Martinez Peck via Glass wrote:

Hi Dale,

I am doing a query like this (as per your recommendation):

nearestQueryFor: aDate on: aCollection

| query |

query := GsQuery fromFormula: self nearestQuery  on: aCollection. 

query bind: 'targetDate' to: aDate.

^ query

nearestQuery 

^ nearestQuery ifNil: [ 

nearestQuery := GsQueryPredicate fromQueryStatement:  'each.date <= targetDate'.

nearestQuery

].

And when I use it, I do this:

query := self nearestQueryFor: dateToCompare on: (self indexingDict at: securityUniqueId).

stream := query reversedReadStream.

            ^ stream atEnd

                ifTrue: [ nil ]

                ifFalse: [ stream next ]

I am profiling some code that uses this a lot and I see that quite some time is spent in the #atEnd. 

I attach a part of the profiling output to see what I mean. 

Do you see something obvious I can speed up?

Thanks in advance, 

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

Mariano,

It is possible that there is a bug in reverse read stream that is masked by fact that the comparison code in the loop is returning true at the right time  ... while working on the btree plus implementation I did find some bugs the reversed read stream implementation that might explain this behavior ... the btree plus reverse read stream implementation is very different because in the btree plus implementation the leaf nodes are doubly linked lists and the previous/next leaks are used ...

I would suggest that you wait for 3.4 to be released and  use btree plus indexes ... we have an Alpha5 release that is pretty close to the final release that you could take for a spin --- you should definitely see better performance with the btree plus indexes for your use case ...

Dale

On 10/25/2017 12:14 PM, Mariano Martinez Peck wrote:

On Wed, Oct 25, 2017 at 3:07 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Mariano,

The costly code is in RangeIndexReadStream>>_atEndForStream: (when aBool is true) and that code can be safely skipped if the values in the index will not include nils, or if your index is on a CharacterCollection and you do not mix Strings and Symbols or if your index is on a Float and the values do not include NaNs.

Hi Dale,

In my case, the collection is of instances of FaSecurityClosingPriceRecord and as you can see in my original email the index is on a date type of instVar. 

If you see code below:

             query := self

                nearestQueryFor: dateToCompare

                on: (self indexingDict at: securityUniqueId).

              stream := query reversedReadStream.

              ^ (stream _atEndForStream: false)

                ifTrue: [ nil ]

                ifFalse: [ stream getNext ] 

 

I can assure you that:

1) Values of the indexes are always dates.

2) I have no FaSecurityClosingPriceRecord instance with nil date

3) I have no nil in the collection passed by to `nearestQueryFor: dateToCompare  on:`

Yet....I am getting different results wether I pass `false` or `true` to #_atEndForStream:   :(

And even worst, it looks like the correct one is when I pass false...

Maybe I am hitting an scenario where `true` is needed that is not in the ones you listed? 

I can confirm at least that the speedup when using `false` was huge, so I would love to find a way to skip that!

Thanks in advance, 

In 3.4 we do this automatically when you create a btree plus index with optmiizedComparison index option ... the 3.4 stream operations have been optimized in other ways as well --- 3.4 is due to be released very soon ...

Dale

On 10/25/2017 07:01 AM, Mariano Martinez Peck via Glass wrote:

Hi Dale,

I am doing a query like this (as per your recommendation):

nearestQueryFor: aDate on: aCollection

| query |

query := GsQuery fromFormula: self nearestQuery  on: aCollection. 

query bind: 'targetDate' to: aDate.

^ query

nearestQuery 

^ nearestQuery ifNil: [ 

nearestQuery := GsQueryPredicate fromQueryStatement:  'each.date <= targetDate'.

nearestQuery

].

And when I use it, I do this:

query := self nearestQueryFor: dateToCompare on: (self indexingDict at: securityUniqueId).

stream := query reversedReadStream.

            ^ stream atEnd

                ifTrue: [ nil ]

                ifFalse: [ stream next ]

I am profiling some code that uses this a lot and I see that quite some time is spent in the #atEnd. 

I attach a part of the profiling output to see what I mean. 

Do you see something obvious I can speed up?

Thanks in advance, 

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--

Mariano
http://marianopeck.wordpress.com

Mariano,

The btree plus indexes (with optimizedComparison option - recommended) do not allow nil values ... also there are new classes being used for Streams (see BtreePlusReadStream) and a number of other other places --- in converting to the new set of classes I did a fair amount of optimization and beefed up test coverage considerably ... with that said I would be interested in seeing any performance comparisons that you might have for your use case ... I think that there is still room for improvement, but there are quite a few different dimensions that can be looked at with tradeoffs for different schemes, so  I would prefer to focus on performance issues for real life use cases, then spend a lot of time and effort optimizing a use case that is not being used ...

Dale

On 12/8/17 3:16 PM, Mariano Martinez Peck wrote:

Hi Dale,

I am reviving this old thread. So...with the legacy indexes I had something like this:

stream := query reversedReadStream.

"We are using #_atEndForStream: and #getNext because of performance reasons. For more details see email 'Can I avoid ReversedRangeIndexReadStream >> atEnd ?'

In GemStone 3.4 we may be able to pass a false below... "

            ^ (stream _atEndForStream: false)

                ifTrue: [ nil ]

                ifFalse: [ stream getNext ]

As I was sure there was no nils in the collection and that way, it was much faster (else, #atEnd was too slow).

I just upgraded to use the new btreeplus indexes but I am not sure if I should do something in particular for above speedup. I am using now the default BtreePlusGsIndexReadStream >> atEnd. Is that correct or there could be a fastest path when I am sure there is no nils?

Thanks in advance,

On Thu, Oct 26, 2017 at 9:56 AM, Mariano Martinez Peck <[hidden email]> wrote:

On Wed, Oct 25, 2017 at 5:27 PM, Dale Henrichs <[hidden email]> wrote:

Mariano,

It is possible that there is a bug in reverse read stream that is masked by fact that the comparison code in the loop is returning true at the right time  ... while working on the btree plus implementation I did find some bugs the reversed read stream implementation that might explain this behavior ... the btree plus reverse read stream implementation is very different because in the btree plus implementation the leaf nodes are doubly linked lists and the previous/next leaks are used ...

I would suggest that you wait for 3.4 to be released and  use btree plus indexes ... we have an Alpha5 release that is pretty close to the final release that you could take for a spin --- you should definitely see better performance with the btree plus indexes for your use case ...

OK, thanks Dale for the answer. I guess I will wait then. Thanks for the offer of Alpha5 but I guess I would rather wait a bit until final release. 

Cheers,

 

Dale

On 10/25/2017 12:14 PM, Mariano Martinez Peck wrote:

On Wed, Oct 25, 2017 at 3:07 PM, Dale Henrichs via Glass <[hidden email]> wrote:

Mariano,

The costly code is in RangeIndexReadStream>>_atEndForStream: (when aBool is true) and that code can be safely skipped if the values in the index will not include nils, or if your index is on a CharacterCollection and you do not mix Strings and Symbols or if your index is on a Float and the values do not include NaNs.

Hi Dale,

In my case, the collection is of instances of FaSecurityClosingPriceRecord and as you can see in my original email the index is on a date type of instVar. 

If you see code below:

             query := self

                nearestQueryFor: dateToCompare

                on: (self indexingDict at: securityUniqueId).

              stream := query reversedReadStream.

              ^ (stream _atEndForStream: false)

                ifTrue: [ nil ]

                ifFalse: [ stream getNext ] 

 

I can assure you that:

1) Values of the indexes are always dates.

2) I have no FaSecurityClosingPriceRecord instance with nil date

3) I have no nil in the collection passed by to `nearestQueryFor: dateToCompare  on:`

Yet....I am getting different results wether I pass `false` or `true` to #_atEndForStream:   :(

And even worst, it looks like the correct one is when I pass false...

Maybe I am hitting an scenario where `true` is needed that is not in the ones you listed? 

I can confirm at least that the speedup when using `false` was huge, so I would love to find a way to skip that!

Thanks in advance, 

In 3.4 we do this automatically when you create a btree plus index with optmiizedComparison index option ... the 3.4 stream operations have been optimized in other ways as well --- 3.4 is due to be released very soon ...

Dale

On 10/25/2017 07:01 AM, Mariano Martinez Peck via Glass wrote:

Hi Dale,

I am doing a query like this (as per your recommendation):

nearestQueryFor: aDate on: aCollection

| query |

query := GsQuery fromFormula: self nearestQuery  on: aCollection. 

query bind: 'targetDate' to: aDate.

^ query

nearestQuery 

^ nearestQuery ifNil: [ 

nearestQuery := GsQueryPredicate fromQueryStatement:  'each.date <= targetDate'.

nearestQuery

].

And when I use it, I do this:

query := self nearestQueryFor: dateToCompare on: (self indexingDict at: securityUniqueId).

stream := query reversedReadStream.

            ^ stream atEnd

                ifTrue: [ nil ]

                ifFalse: [ stream next ]

I am profiling some code that uses this a lot and I see that quite some time is spent in the #atEnd. 

I attach a part of the profiling output to see what I mean. 

Do you see something obvious I can speed up?

Thanks in advance, 

--

Mariano
http://marianopeck.wordpress.com

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

_______________________________________________
Glass mailing list
[hidden email]
http://lists.gemtalksystems.com/mailman/listinfo/glass

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com

--

Mariano
http://marianopeck.wordpress.com