Smalltalk › Instantiations

VA7.52: AIX: Analyzing Gargabe Collection

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

16 messages Options

shanna.bastkowski

VA7.52: AIX: Analyzing Gargabe Collection

Hi there,

is there any way to monitor the VM's gargabe collection?
When I run my Smalltlalk program under AIX (instead of Windows) it takes about 60 times longer to make its calculations.

I already tried tuning the memory by changing some of the command line parameters mentioned here: http://www.instantiations.com/docs/FAQ/wwhelp/wwhimpl/js/html/wwhelp.htm#href=va03001.html

Unfortunately, I could not achieve a significant better performance. The program fetches data from the database and writes into it. My assumption is that the VM is busy with garbage collection because a lot of new objects are created. Is there any way to get some output to see when and how much gargabe collection is performed?

Thanks.

Best regards,

Shanna

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.

Wayne Johnston

Re: VA7.52: AIX: Analyzing Gargabe Collection

This is not the detail you ask for, but it may prove your theory. We periodically output some statistics via:

reportGcStatsOn: aStream

| a |

a := EsbTimer vmStats.

a isNil

ifFalse: [

aStream

tab;

nextPutAll: 'numberOfScavenges';

tab;

nextPutAll: a numberOfScavenges printString;

cr;

tab;

nextPutAll: 'numberOfGlobalGCs';

tab;

nextPutAll: a numberOfGlobalGCs printString;

cr;

tab;

nextPutAll: 'millisecondScavengeTime';

tab;

nextPutAll: a millisecondScavengeTime printString;

cr;

tab;

nextPutAll: 'millisecondGlobalGCTime';

tab;

nextPutAll: a millisecondGlobalGCTime printString]

Noschvie

Re: VA7.52: AIX: Analyzing Gargabe Collection

In reply to this post by shanna.bastkowski

What do you mean with "database" ? Maybe the database connection is the bottleneck, have you checked the timings of the fetch and write operations?

Norbert

Davide Grandi-2

Re: VA7.52: AIX: Analyzing Gargabe Collection

Among the "usual suspects" : dynamic SQL via ODBC ? Maybe there's (still) a cache on VA side (beside the one on db). I remember a switch between DB2 and SQL Server : after setting the sql cache on VA side ... ... performances boost up. Then we de-dynamic-ize the SQL queries ('WHERE' clauses were expanded in query text : introduced '?' parameters and pre-preparing queries). Just my 2 Euro cents. Ciao, Davide

On 18/09/2018 09:55, Norbert Schlemmer wrote:

What do you mean with "database" ? Maybe the database connection is the bottleneck, have you checked the timings of the fetch and write operations?

Norbert

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.

shanna.bastkowski

Re: VA7.52: AIX: Analyzing Gargabe Collection

In reply to this post by Wayne Johnston

Hi Wayne,

perfect, I'll try to do that. Thanks!

On Monday, September 17, 2018 at 6:21:47 PM UTC+2, Wayne Johnston wrote:

This is not the detail you ask for, but it may prove your theory. We periodically output some statistics via:

reportGcStatsOn: aStream
| a |
a := EsbTimer vmStats.
a isNil
ifFalse: [
aStream
tab;
nextPutAll: 'numberOfScavenges';
tab;
nextPutAll: a numberOfScavenges printString;
cr;
tab;
nextPutAll: 'numberOfGlobalGCs';
tab;
nextPutAll: a numberOfGlobalGCs printString;
cr;
tab;
nextPutAll: 'millisecondScavengeTime';
tab;
nextPutAll: a millisecondScavengeTime printString;
cr;
tab;
nextPutAll: 'millisecondGlobalGCTime';
tab;
nextPutAll: a millisecondGlobalGCTime printString]

shanna.bastkowski

Re: VA7.52: AIX: Analyzing Gargabe Collection

In reply to this post by Noschvie

Hi Norbert,

I am using DB2. I do not think that this is the bottleneck though since the same code runs under Windows, using the same DB2, much faster. This is why I assumed that the bad performance must be somehow OS-dependent.

On Tuesday, September 18, 2018 at 9:55:26 AM UTC+2, Norbert Schlemmer wrote:

What do you mean with "database" ? Maybe the database connection is the bottleneck, have you checked the timings of the fetch and write operations?
Norbert

shanna.bastkowski

Re: VA7.52: AIX: Analyzing Gargabe Collection

In reply to this post by Davide Grandi-2

Hi Davide,

I don't have much experience in databases (nor Smalltalk :D) and am using the build-in database parts in order to fetch data (DB2).

Can you give a hint how to set the SQL cache??
Thanks.

On Tuesday, September 18, 2018 at 10:12:37 AM UTC+2, Davide Grandi wrote:

Among the "usual suspects" : dynamic SQL via ODBC ? Maybe there's (still) a cache on VA side (beside the one on db). I remember a switch between DB2 and SQL Server : after setting the sql cache on VA side ... ... performances boost up. Then we de-dynamic-ize the SQL queries ('WHERE' clauses were expanded in query text : introduced '?' parameters and pre-preparing queries). Just my 2 Euro cents. Ciao, Davide

On 18/09/2018 09:55, Norbert Schlemmer wrote:

What do you mean with "database" ? Maybe the database connection is the bottleneck, have you checked the timings of the fetch and write operations?

Norbert

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="Lm-x9kr3FwAJ">va-smalltalk...@googlegroups.com.
To post to this group, send email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="Lm-x9kr3FwAJ">va-sma...@....
Visit this group at <a onmousedown="this.href='https://groups.google.com/group/va-smalltalk';return true;" onclick="this.href='https://groups.google.com/group/va-smalltalk';return true;" href="https://groups.google.com/group/va-smalltalk" target="_blank" rel="nofollow">https://groups.google.com/group/va-smalltalk.
For more options, visit <a onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;" href="https://groups.google.com/d/optout" target="_blank" rel="nofollow">https://groups.google.com/d/optout.

Noschvie

Re: VA7.52: AIX: Analyzing Gargabe Collection

How many fetch and write operations are done ? Just to get a feeling for the dimension.

Davide Grandi-2

Re: VA7.52: AIX: Analyzing Gargabe Collection

In reply to this post by shanna.bastkowski

The default abt.ini ? Back to home I could find the original edit. (you could, of course, follow the execution of a SQL SELECT in order to catch that cache, if still esxists but it may be a lengthy work). Best regards, Davide Grandi

On 18/09/2018 12:20, [hidden email] wrote:

Hi Davide,

I don't have much experience in databases (nor Smalltalk :D) and am using the build-in database parts in order to fetch data (DB2).

Can you give a hint how to set the SQL cache??
Thanks.

On Tuesday, September 18, 2018 at 10:12:37 AM UTC+2, Davide Grandi wrote:

Among the "usual suspects" : dynamic SQL via ODBC ? Maybe there's (still) a cache on VA side (beside the one on db). I remember a switch between DB2 and SQL Server : after setting the sql cache on VA side ... ... performances boost up. Then we de-dynamic-ize the SQL queries ('WHERE' clauses were expanded in query text : introduced '?' parameters and pre-preparing queries). Just my 2 Euro cents. Ciao, Davide

On 18/09/2018 09:55, Norbert Schlemmer wrote:

What do you mean with "database" ? Maybe the database connection is the bottleneck, have you checked the timings of the fetch and write operations?

Norbert

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="Lm-x9kr3FwAJ" moz-do-not-send="true">va-smalltalk...@googlegroups.com.
To post to this group, send email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="Lm-x9kr3FwAJ" moz-do-not-send="true">va-sma...@....
Visit this group at <a onmousedown="this.href='https://groups.google.com/group/va-smalltalk';return true;" onclick="this.href='https://groups.google.com/group/va-smalltalk';return true;" href="https://groups.google.com/group/va-smalltalk" target="_blank" rel="nofollow" moz-do-not-send="true">https://groups.google.com/group/va-smalltalk.
For more options, visit <a onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;" href="https://groups.google.com/d/optout" target="_blank" rel="nofollow" moz-do-not-send="true">https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.

Davide Grandi-2

Re: VA7.52: AIX: Analyzing Gargabe Collection

Found in a veeeeeery old VA doc.-- VisualAge keeps the last 10 database queries in a cache. This means that queries after the first query should process faster. To improve performance further, you can increase the number of queries that are cached by changing the PreparedStatementCacheSize value in the ABT.INI file. -- May be it still apply, maybe not. Sorry ... Best regards, Davide Grandi

On 18/09/2018 16:53, Davide Grandi wrote:

The default abt.ini ? Back to home I could find the original edit. (you could, of course, follow the execution of a SQL SELECT in order to catch that cache, if still esxists but it may be a lengthy work). Best regards, Davide Grandi

On 18/09/2018 12:20, [hidden email] wrote:

Hi Davide,

I don't have much experience in databases (nor Smalltalk :D) and am using the build-in database parts in order to fetch data (DB2).

Can you give a hint how to set the SQL cache??
Thanks.

On Tuesday, September 18, 2018 at 10:12:37 AM UTC+2, Davide Grandi wrote:

Among the "usual suspects" : dynamic SQL via ODBC ? Maybe there's (still) a cache on VA side (beside the one on db). I remember a switch between DB2 and SQL Server : after setting the sql cache on VA side ... ... performances boost up. Then we de-dynamic-ize the SQL queries ('WHERE' clauses were expanded in query text : introduced '?' parameters and pre-preparing queries). Just my 2 Euro cents. Ciao, Davide

On 18/09/2018 09:55, Norbert Schlemmer wrote:

What do you mean with "database" ? Maybe the database connection is the bottleneck, have you checked the timings of the fetch and write operations?

Norbert

--

You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To post to this group, send email to [hidden email].
Visit this group at https://groups.google.com/group/va-smalltalk.
For more options, visit https://groups.google.com/d/optout.

Davide Grandi-2

Re: VA7.52: AIX: Analyzing Gargabe Collection

This trick was still in 9.0 :
--
www.instantiations.com/docs/90/db/dbdfqr.html
--

Good luck,

Davide Grandi

On 18/09/2018 17:06, Davide Grandi wrote:

> Found in a veeeeeery old VA doc.
> --
> VisualAge keeps the last 10 database queries
> in a cache. This means that queries after the
> first query should process faster. To improve
> performance further, you can increase the
> number of queries that are cached by
> changing the PreparedStatementCacheSize
> value in the ABT.INI file.
> --
>
> May be it still apply, maybe not. Sorry ...
>
> Best regards,
>
>      Davide Grandi
>
> On 18/09/2018 16:53, Davide Grandi wrote:
>> The default abt.ini ?
>> Back to home I could find the original edit.
>> (you could, of course, follow the execution of a SQL SELECT in order
>> to catch that cache, if still esxists
>> but it may be a lengthy work).
>>
>> Best regards,
>>
>>     Davide Grandi
>>
>> On 18/09/2018 12:20, [hidden email] wrote:
>>> Hi Davide,
>>>
>>> I don't have much experience in databases (nor Smalltalk :D) and am
>>> using the build-in database parts in order to fetch data (DB2).
>>> Can you give a hint how to set the SQL cache??
>>> Thanks.
>>>
>>> On Tuesday, September 18, 2018 at 10:12:37 AM UTC+2, Davide Grandi wrote:
>>>
>>> Among the "usual suspects" : dynamic SQL via ODBC ?
>>> Maybe there's (still) a cache on VA side (beside the one on db).
>>>
>>> I remember a switch between DB2 and SQL Server : after setting
>>> the sql cache on VA side ...
>>> ... performances boost up.
>>> Then we de-dynamic-ize the SQL queries ('WHERE' clauses were
>>> expanded in query text : introduced '?' parameters and
>>> pre-preparing queries).
>>>
>>> Just my 2 Euro cents.
>>>
>>> Ciao,
>>>
>>>     Davide
>>>
>>> On 18/09/2018 09:55, Norbert Schlemmer wrote:
>>>> What do you mean with "database" ? Maybe the database connection
>>>> is the bottleneck, have you checked the timings of the fetch and
>>>> write operations?
>>>> Norbert
>>>> --
>>>
>>> You received this message because you are subscribed to the Google
>>> Groups "VA Smalltalk" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to [hidden email]
>>> <mailto:[hidden email]>.
>>> To post to this group, send email to [hidden email]
>>> <mailto:[hidden email]>.
>>> Visit this group at https://groups.google.com/group/va-smalltalk.
>>> For more options, visit https://groups.google.com/d/optout.

shanna.bastkowski

Re: VA7.52: AIX: Analyzing Gargabe Collection

In reply to this post by shanna.bastkowski

Thanks for your replies.

After further analyzing by evaluating the execution time of specific blocks and placing more log messages in the code I could find the the line of code which takes too long.

anArray detect: [ :each | (each isValidAt: anAbtTimestamp) and: [each isValidFor: anInteger] ] ifNone: [].

This line takes about 770ms to execute. It is placed in a loop which runs 262 times, so it takes about 200ms to run.

This loop runs 11 times and therefore takes about 37 minutes. This loops runs twice at the two places where the program did not seem to do anything.
In the methods isValidAt: and isValidFor: simple comparisons like anAbtTimestamp >= anotherAbtTimestamp are made. It does not fetch or write any data, nor is creating a lot of new objects. The method isValidAt: takes about 10ms to execute and will run 77 times since the array has 77 entries (fits to the 770ms from the detect-block).

Even if I found the place which causes the performance issue and I could probably rewrite it in a more efficient way I still have no clue why the exact same code runs MUCH faster under Windows.
Any ideas?

Louis LaBrunda

Re: VA7.52: AIX: Analyzing Gargabe Collection

Hi Shanna,

Just a guess off the top of my head, maybe there is some optimization on Windows like inlining going on that isn't AIX. If that is the case, I'm not sure what you can do about it.

Lou

On Wednesday, September 19, 2018 at 12:06:38 PM UTC-4, [hidden email] wrote:

Thanks for your replies.

After further analyzing by evaluating the execution time of specific blocks and placing more log messages in the code I could find the the line of code which takes too long.

aSortedCollection detect: [ :each | (each isValidAt: anAbtTimestamp) and: [each isValidFor: anInteger] ] ifNone: [].

This line takes about 770ms to execute. It is placed in a loop which runs 262 times, so it takes about 200s to run.
This loop runs 11 times and therefore takes about 37 minutes. This loops runs twice at the two places where the program did not seem to do anything.
In the methods isValidAt: and isValidFor: simple comparisons like anAbtTimestamp >= anotherAbtTimestamp are made. It does not fetch or write any data, nor is creating a lot of new objects. The method isValidAt: takes about 10ms to execute and will run 77 times since the sorted collection has 77 entries (fits to the 770ms from the detect-block).

Even if I found the place which causes the performance issue and I could probably rewrite it in a more efficient way I still have no clue why the exact same code runs MUCH faster under Windows.
Any ideas?

shanna.bastkowski

Re: VA7.52: AIX: Analyzing Gargabe Collection

In reply to this post by Davide Grandi-2

Ciao Davide,

after solving the performance problems by rewriting the critital area I found another performance leak.

(self subpartNamed: 'Insert') executeQueryAsTransaction.

This line takes about 700ms to execute and is executed about 2800 times in the loop.

Do you know under which heading I have to configure PreparedStatementCacheSize in abt.ini?

On Tuesday, September 18, 2018 at 5:09:32 PM UTC+2, Davide Grandi wrote:

This trick was still in 9.0 :
--
<a onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.instantiations.com%2Fdocs%2F90%2Fdb%2Fdbdfqr.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEPfxpx0dlq_Pe4QaTzM8WdgQepwA';return true;" onclick="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fwww.instantiations.com%2Fdocs%2F90%2Fdb%2Fdbdfqr.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEPfxpx0dlq_Pe4QaTzM8WdgQepwA';return true;" href="http://www.instantiations.com/docs/90/db/dbdfqr.html" target="_blank" rel="nofollow">www.instantiations.com/docs/90/db/dbdfqr.html
--

Good luck,

Davide Grandi

On 18/09/2018 17:06, Davide Grandi wrote:

> Found in a veeeeeery old VA doc.
> --
> VisualAge keeps the last 10 database queries
> in a cache. This means that queries after the
> first query should process faster. To improve
> performance further, you can increase the
> number of queries that are cached by
> changing the PreparedStatementCacheSize
> value in the ABT.INI file.
> --
>
> May be it still apply, maybe not. Sorry ...
>
> Best regards,
>
>      Davide Grandi
>
> On 18/09/2018 16:53, Davide Grandi wrote:
>> The default abt.ini ?
>> Back to home I could find the original edit.
>> (you could, of course, follow the execution of a SQL SELECT in order
>> to catch that cache, if still esxists
>> but it may be a lengthy work).
>>
>> Best regards,
>>
>>     Davide Grandi
>>
>> On 18/09/2018 12:20, <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="OuskQwsOGAAJ">shanna.b...@... wrote:
>>> Hi Davide,
>>>
>>> I don't have much experience in databases (nor Smalltalk :D) and am
>>> using the build-in database parts in order to fetch data (DB2).
>>> Can you give a hint how to set the SQL cache??
>>> Thanks.
>>>
>>> On Tuesday, September 18, 2018 at 10:12:37 AM UTC+2, Davide Grandi wrote:
>>>
>>> Among the "usual suspects" : dynamic SQL via ODBC ?
>>> Maybe there's (still) a cache on VA side (beside the one on db).
>>>
>>> I remember a switch between DB2 and SQL Server : after setting
>>> the sql cache on VA side ...
>>> ... performances boost up.
>>> Then we de-dynamic-ize the SQL queries ('WHERE' clauses were
>>> expanded in query text : introduced '?' parameters and
>>> pre-preparing queries).
>>>
>>> Just my 2 Euro cents.
>>>
>>> Ciao,
>>>
>>>     Davide
>>>
>>> On 18/09/2018 09:55, Norbert Schlemmer wrote:
>>>> What do you mean with "database" ? Maybe the database connection
>>>> is the bottleneck, have you checked the timings of the fetch and
>>>> write operations?
>>>> Norbert
>>>> --
>>>
>>> You received this message because you are subscribed to the Google
>>> Groups "VA Smalltalk" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="OuskQwsOGAAJ">va-smalltalk...@googlegroups.com
>>> <mailto:<a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="OuskQwsOGAAJ">va-smalltalk+unsubscribe@...>.
>>> To post to this group, send email to <a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="OuskQwsOGAAJ">va-sma...@...
>>> <mailto:<a onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;" href="javascript:" target="_blank" rel="nofollow" gdf-obfuscated-mailto="OuskQwsOGAAJ">va-sma...@googlegroups.com>.
>>> Visit this group at <a onmousedown="this.href='https://groups.google.com/group/va-smalltalk';return true;" onclick="this.href='https://groups.google.com/group/va-smalltalk';return true;" href="https://groups.google.com/group/va-smalltalk" target="_blank" rel="nofollow">https://groups.google.com/group/va-smalltalk.
>>> For more options, visit <a onmousedown="this.href='https://groups.google.com/d/optout';return true;" onclick="this.href='https://groups.google.com/d/optout';return true;" href="https://groups.google.com/d/optout" target="_blank" rel="nofollow">https://groups.google.com/d/optout.

Noschvie

Re: VA7.52: AIX: Analyzing Gargabe Collection

Hi Shanna

DB2 supports transactions, and you have a single transaction for each insert operation. But you can use one transaction for all your insert operations.

https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/intro/src/tpc/db2z_commitandrollbackoftransactions.html

use #executeQuery for each insert operation and #commitTransaction at the end after the loop

Norbert

shanna.bastkowski

Re: VA7.52: AIX: Analyzing Gargabe Collection

Hi Norbert,

thank you, this helps a lot!

On Tuesday, September 25, 2018 at 11:18:14 AM UTC+2, Norbert Schlemmer wrote:

Hi Shanna

DB2 supports transactions, and you have a single transaction for each insert operation. But you can use one transaction for all your insert operations.
<a onmousedown="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.ibm.com%2Fsupport%2Fknowledgecenter%2Fen%2FSSEPEK_10.0.0%2Fintro%2Fsrc%2Ftpc%2Fdb2z_commitandrollbackoftransactions.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEkjd0kqvoUMyE-Si5hO6tmSklwxw';return true;" onclick="this.href='https://www.google.com/url?q\x3dhttps%3A%2F%2Fwww.ibm.com%2Fsupport%2Fknowledgecenter%2Fen%2FSSEPEK_10.0.0%2Fintro%2Fsrc%2Ftpc%2Fdb2z_commitandrollbackoftransactions.html\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNEkjd0kqvoUMyE-Si5hO6tmSklwxw';return true;" href="https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/intro/src/tpc/db2z_commitandrollbackoftransactions.html" target="_blank" rel="nofollow">https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/intro/src/tpc/db2z_commitandrollbackoftransactions.html

use #executeQuery for each insert operation and #commitTransaction at the end after the loop

Norbert