discussion on oldSpaceSize and newSpaceSize - speed implications

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

On Monday, November 30, 2020 at 9:32:36 PM UTC-8 [hidden email] wrote:

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

A couple of points, based on empirical evidence.

The defaults for the memory management parameters (and pretty well all the others) were established in the early 90s for what is now considered fairly limited hardware capabilities. I don't think any of them have changed since VA 3.x and I'm guessing not since OTI's original product.

When I worked at Geico on their original primary sales application, I did bump up some of the memory management parameters for a noticeable improvement in performance. I don't think I bumped them up too much. Perhaps new space to 10 MB or so. I don't think you want new space too large, otherwise your new space swapping performance goes down, and it's possibly the most critical memory space.

It sounds like you have good metering tools in your application, so I suggest you conduct incremental experiments. Try 16 MB and see how the stats compare to the current settings. If they improve, try 32 MB and see what happens, repeat. You may even want to start smaller, lest you miss the sweet spot entirely.

You can also register something to be informed of GC events. See Process class>>#addGCEventHandler: and the corresponding remove partner. It's called from Process class>>#gcEvent:. This would allow you to capture frequency counts of scavenges versus global garbage collections.

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

Richard,

thanks a lot for your thoughts. One thing I thought about after writing my post was that maybe I first need to make the increase steps bigger, so that whenever a user needs more memory, they get either way too much or, in case of users who really need a lot, don't have to allocate too often. So today we're testing with "enough" newSpace and an allocationIncrement of 8 MBytes.

Let's see how far we get with this.

Thanks especially for confirming the default numbers stem from a time when a megabyte was beyond what a "high end" PC had available or could address...

I am not so sure I understand your point about swapping. Are you talking about the classical memory swapping of the host's O/S? in this case, we don't have to worry. The server machines have plenty of memory. Our images now start with a memory footprint of 620 MB, the server has 64GB RAM, so I guess this can be ignored in this specific situation.

My biggest concern here is that our new settings cause negative side effects for our typical users who never come close to these memory requirements...

Joachim

Richard Sargent schrieb am Dienstag, 1. Dezember 2020 um 23:51:45 UTC+1:

On Monday, November 30, 2020 at 9:32:36 PM UTC-8 [hidden email] wrote:

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

A couple of points, based on empirical evidence.

The defaults for the memory management parameters (and pretty well all the others) were established in the early 90s for what is now considered fairly limited hardware capabilities. I don't think any of them have changed since VA 3.x and I'm guessing not since OTI's original product.

When I worked at Geico on their original primary sales application, I did bump up some of the memory management parameters for a noticeable improvement in performance. I don't think I bumped them up too much. Perhaps new space to 10 MB or so. I don't think you want new space too large, otherwise your new space swapping performance goes down, and it's possibly the most critical memory space.

It sounds like you have good metering tools in your application, so I suggest you conduct incremental experiments. Try 16 MB and see how the stats compare to the current settings. If they improve, try 32 MB and see what happens, repeat. You may even want to start smaller, lest you miss the sweet spot entirely.

You can also register something to be informed of GC events. See Process class>>#addGCEventHandler: and the corresponding remove partner. It's called from Process class>>#gcEvent:. This would allow you to capture frequency counts of scavenges versus global garbage collections.

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

It has been a few decades since I looked at VA GC memory settings. However as mention they are too small (perhaps)

Things to consider for a flip/swap/scavenge.

How long things will take to perform the copy of the living objects between the two semi spaces. Is is a concern? If you need to service a socket interrupt how long can you delay for?

The impact on the old space to new space remember table. I don’t know how that is implemented for growth or for scanning performance in VA.

How do weak objects get cleaned up, is it tied to a scavenge? 

Beware of making new space a multiple of your larger short lived objects, I’ve seen cases where 4mb happily holds 40 100k objects but all those 20 byte objects spill into old space, adjusting to 4,099k lessened early tenuring.

A repeatable test case is alway helpful

Measure business logic thruput, and if needed latency.

Sent from ProtonMail Mobile

On Tue, Dec 1, 2020 at 23:53, Richard Sargent <[hidden email]> wrote:

On Tuesday, December 1, 2020 at 11:36:05 PM UTC-8 [hidden email] wrote:

Richard,

thanks a lot for your thoughts. One thing I thought about after writing my post was that maybe I first need to make the increase steps bigger, so that whenever a user needs more memory, they get either way too much or, in case of users who really need a lot, don't have to allocate too often. So today we're testing with "enough" newSpace and an allocationIncrement of 8 MBytes.

Let's see how far we get with this.

Thanks especially for confirming the default numbers stem from a time when a megabyte was beyond what a "high end" PC had available or could address...

I am not so sure I understand your point about swapping. Are you talking about the classical memory swapping of the host's O/S? in this case, we don't have to worry. The server machines have plenty of memory. Our images now start with a memory footprint of 620 MB, the server has 64GB RAM, so I guess this can be ignored in this specific situation.

Sorry. I was referring to the two halves of new space. When one half fills, the still referenced objects need to be copied to the other half and then things proceed with it filling. The bigger the halves, the longer it takes to do that process. This is what I meant by "swapping".

Arguably, with a big new space, the swap from half to half would be less frequent, but would take longer, for some unclear notion of longer. That would affect all your users versus the few who have really large new space requirements from time to time.

I don't know how your system is architected. If it were a web server with users randomly distributed among multiple images for each request, you could possibly target requests known to be large to (a small number of) images with large new spaces and the rest of the requests to a larger pool of not so large new space images. The ability to decide that is uncertain and perhaps impractical.

My biggest concern here is that our new settings cause negative side effects for our typical users who never come close to these memory requirements...

Finding the optimum is likely trial and error, with more error than trial. :-)

Joachim

Richard Sargent schrieb am Dienstag, 1. Dezember 2020 um 23:51:45 UTC+1:

On Monday, November 30, 2020 at 9:32:36 PM UTC-8 [hidden email] wrote:

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

A couple of points, based on empirical evidence.

The defaults for the memory management parameters (and pretty well all the others) were established in the early 90s for what is now considered fairly limited hardware capabilities. I don't think any of them have changed since VA 3.x and I'm guessing not since OTI's original product.

When I worked at Geico on their original primary sales application, I did bump up some of the memory management parameters for a noticeable improvement in performance. I don't think I bumped them up too much. Perhaps new space to 10 MB or so. I don't think you want new space too large, otherwise your new space swapping performance goes down, and it's possibly the most critical memory space.

It sounds like you have good metering tools in your application, so I suggest you conduct incremental experiments. Try 16 MB and see how the stats compare to the current settings. If they improve, try 32 MB and see what happens, repeat. You may even want to start smaller, lest you miss the sweet spot entirely.

You can also register something to be informed of GC events. See Process class>>#addGCEventHandler: and the corresponding remove partner. It's called from Process class>>#gcEvent:. This would allow you to capture frequency counts of scavenges versus global garbage collections.

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/62ebc3f4-17d6-48d8-b08e-88bec3e16694n%40googlegroups.com.

Hallo

donot be afraid of increasing newspace...., we use 512MB , and it is also possible to change that on demand

I would try to start at 64MB and measure the results with EsbSampler (you can see the scavenges and gc there )

[hidden email] schrieb am Mittwoch, 2. Dezember 2020 um 09:14:17 UTC+1:

It has been a few decades since I looked at VA GC memory settings. However as mention they are too small (perhaps)

Things to consider for a flip/swap/scavenge.

How long things will take to perform the copy of the living objects between the two semi spaces. Is is a concern? If you need to service a socket interrupt how long can you delay for?

The impact on the old space to new space remember table. I don’t know how that is implemented for growth or for scanning performance in VA.

How do weak objects get cleaned up, is it tied to a scavenge? 

Beware of making new space a multiple of your larger short lived objects, I’ve seen cases where 4mb happily holds 40 100k objects but all those 20 byte objects spill into old space, adjusting to 4,099k lessened early tenuring.

A repeatable test case is alway helpful

Measure business logic thruput, and if needed latency.

Sent from ProtonMail Mobile

On Tue, Dec 1, 2020 at 23:53, Richard Sargent <[hidden email]> wrote:

On Tuesday, December 1, 2020 at 11:36:05 PM UTC-8 [hidden email] wrote:

Richard,

thanks a lot for your thoughts. One thing I thought about after writing my post was that maybe I first need to make the increase steps bigger, so that whenever a user needs more memory, they get either way too much or, in case of users who really need a lot, don't have to allocate too often. So today we're testing with "enough" newSpace and an allocationIncrement of 8 MBytes.

Let's see how far we get with this.

Thanks especially for confirming the default numbers stem from a time when a megabyte was beyond what a "high end" PC had available or could address...

I am not so sure I understand your point about swapping. Are you talking about the classical memory swapping of the host's O/S? in this case, we don't have to worry. The server machines have plenty of memory. Our images now start with a memory footprint of 620 MB, the server has 64GB RAM, so I guess this can be ignored in this specific situation.

Sorry. I was referring to the two halves of new space. When one half fills, the still referenced objects need to be copied to the other half and then things proceed with it filling. The bigger the halves, the longer it takes to do that process. This is what I meant by "swapping".

Arguably, with a big new space, the swap from half to half would be less frequent, but would take longer, for some unclear notion of longer. That would affect all your users versus the few who have really large new space requirements from time to time.

I don't know how your system is architected. If it were a web server with users randomly distributed among multiple images for each request, you could possibly target requests known to be large to (a small number of) images with large new spaces and the rest of the requests to a larger pool of not so large new space images. The ability to decide that is uncertain and perhaps impractical.

My biggest concern here is that our new settings cause negative side effects for our typical users who never come close to these memory requirements...

Finding the optimum is likely trial and error, with more error than trial. :-)

Joachim

Richard Sargent schrieb am Dienstag, 1. Dezember 2020 um 23:51:45 UTC+1:

On Monday, November 30, 2020 at 9:32:36 PM UTC-8 [hidden email] wrote:

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

A couple of points, based on empirical evidence.

The defaults for the memory management parameters (and pretty well all the others) were established in the early 90s for what is now considered fairly limited hardware capabilities. I don't think any of them have changed since VA 3.x and I'm guessing not since OTI's original product.

When I worked at Geico on their original primary sales application, I did bump up some of the memory management parameters for a noticeable improvement in performance. I don't think I bumped them up too much. Perhaps new space to 10 MB or so. I don't think you want new space too large, otherwise your new space swapping performance goes down, and it's possibly the most critical memory space.

It sounds like you have good metering tools in your application, so I suggest you conduct incremental experiments. Try 16 MB and see how the stats compare to the current settings. If they improve, try 32 MB and see what happens, repeat. You may even want to start smaller, lest you miss the sweet spot entirely.

You can also register something to be informed of GC events. See Process class>>#addGCEventHandler: and the corresponding remove partner. It's called from Process class>>#gcEvent:. This would allow you to capture frequency counts of scavenges versus global garbage collections.

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/62ebc3f4-17d6-48d8-b08e-88bec3e16694n%40googlegroups.com.

Wolfgang,

thanks for your tip. I even went so far to go to 256 MB newSpace following your advice, in addition to increasing the allocationIncrement tp 8 MB.

The images start big now, but I could get rid of a few GC cycles during this operation.

So far I can tell it imporves the situation for the affected "heavy users" by postponing the point where the system gets slow by quite some repetitions of the affected action. And it doesn't hurt our "normal" users, which was my biggest concern. So I think we found a combination that is much better that the defaults for our very individual situation. We now have an oldSpace of 128MB, a newSpace of 256 MB (which leaves some air at the top) and an allocationIncrement of 8MB.

So we could push the wall a few minutes ahead, but still hit it ;-)

The good news: we also found some other problem in our code / object model that made the object allocation in such massive numbers necessary. With that change to our program, we could allow our user to work for almost two hours without any (noticable) effect. She started with commit times of ~350 ms, and after almost two hours we were still somehwere between 1700 and 2500 ms.

So maybe the whole GC tweaking stuff wasn't even necessary. The biggest problem was in our code, not GC or scavenges etc, even though tewaking it could put some paint on the stain ;-) Thanks to everybody who helped and provided input! As always, this list is a place to learn! Thanks guys for enabling this.

Joachim

wobotzen schrieb am Donnerstag, 3. Dezember 2020 um 09:44:47 UTC+1:

Hallo

donot be afraid of increasing newspace...., we use 512MB , and it is also possible to change that on demand

I would try to start at 64MB and measure the results with EsbSampler (you can see the scavenges and gc there )

[hidden email] schrieb am Mittwoch, 2. Dezember 2020 um 09:14:17 UTC+1:

It has been a few decades since I looked at VA GC memory settings. However as mention they are too small (perhaps)

Things to consider for a flip/swap/scavenge.

How long things will take to perform the copy of the living objects between the two semi spaces. Is is a concern? If you need to service a socket interrupt how long can you delay for?

The impact on the old space to new space remember table. I don’t know how that is implemented for growth or for scanning performance in VA.

How do weak objects get cleaned up, is it tied to a scavenge? 

Beware of making new space a multiple of your larger short lived objects, I’ve seen cases where 4mb happily holds 40 100k objects but all those 20 byte objects spill into old space, adjusting to 4,099k lessened early tenuring.

A repeatable test case is alway helpful

Measure business logic thruput, and if needed latency.

Sent from ProtonMail Mobile

On Tue, Dec 1, 2020 at 23:53, Richard Sargent <[hidden email]> wrote:

On Tuesday, December 1, 2020 at 11:36:05 PM UTC-8 [hidden email] wrote:

Richard,

thanks a lot for your thoughts. One thing I thought about after writing my post was that maybe I first need to make the increase steps bigger, so that whenever a user needs more memory, they get either way too much or, in case of users who really need a lot, don't have to allocate too often. So today we're testing with "enough" newSpace and an allocationIncrement of 8 MBytes.

Let's see how far we get with this.

Thanks especially for confirming the default numbers stem from a time when a megabyte was beyond what a "high end" PC had available or could address...

I am not so sure I understand your point about swapping. Are you talking about the classical memory swapping of the host's O/S? in this case, we don't have to worry. The server machines have plenty of memory. Our images now start with a memory footprint of 620 MB, the server has 64GB RAM, so I guess this can be ignored in this specific situation.

Sorry. I was referring to the two halves of new space. When one half fills, the still referenced objects need to be copied to the other half and then things proceed with it filling. The bigger the halves, the longer it takes to do that process. This is what I meant by "swapping".

Arguably, with a big new space, the swap from half to half would be less frequent, but would take longer, for some unclear notion of longer. That would affect all your users versus the few who have really large new space requirements from time to time.

I don't know how your system is architected. If it were a web server with users randomly distributed among multiple images for each request, you could possibly target requests known to be large to (a small number of) images with large new spaces and the rest of the requests to a larger pool of not so large new space images. The ability to decide that is uncertain and perhaps impractical.

My biggest concern here is that our new settings cause negative side effects for our typical users who never come close to these memory requirements...

Finding the optimum is likely trial and error, with more error than trial. :-)

Joachim

Richard Sargent schrieb am Dienstag, 1. Dezember 2020 um 23:51:45 UTC+1:

On Monday, November 30, 2020 at 9:32:36 PM UTC-8 [hidden email] wrote:

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

A couple of points, based on empirical evidence.

The defaults for the memory management parameters (and pretty well all the others) were established in the early 90s for what is now considered fairly limited hardware capabilities. I don't think any of them have changed since VA 3.x and I'm guessing not since OTI's original product.

When I worked at Geico on their original primary sales application, I did bump up some of the memory management parameters for a noticeable improvement in performance. I don't think I bumped them up too much. Perhaps new space to 10 MB or so. I don't think you want new space too large, otherwise your new space swapping performance goes down, and it's possibly the most critical memory space.

It sounds like you have good metering tools in your application, so I suggest you conduct incremental experiments. Try 16 MB and see how the stats compare to the current settings. If they improve, try 32 MB and see what happens, repeat. You may even want to start smaller, lest you miss the sweet spot entirely.

You can also register something to be informed of GC events. See Process class>>#addGCEventHandler: and the corresponding remove partner. It's called from Process class>>#gcEvent:. This would allow you to capture frequency counts of scavenges versus global garbage collections.

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/62ebc3f4-17d6-48d8-b08e-88bec3e16694n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/9cbb20fc-25e3-453a-82b7-0022c8179ff3n%40googlegroups.com.

Greetings all,

Just wanted to throw an out-of-the-box tool that may detect this kind of issue in which you generate a huge unnecessary amount of objects. This tool is intended for a development image, but it may still be useful. 

1) Load feature "z.ST: Performance Workbench"

2) Evaluate:  "EsMemoryUseReport openMemorySnapShow". That opens a table widget with classes and amount of instances and refreshes every second. See attached example. 

Best,

On Thu, Dec 3, 2020 at 8:35 AM Joachim Tuchel <[hidden email]> wrote:

Wolfgang,

thanks for your tip. I even went so far to go to 256 MB newSpace following your advice, in addition to increasing the allocationIncrement tp 8 MB.

The images start big now, but I could get rid of a few GC cycles during this operation.

So far I can tell it imporves the situation for the affected "heavy users" by postponing the point where the system gets slow by quite some repetitions of the affected action. And it doesn't hurt our "normal" users, which was my biggest concern. So I think we found a combination that is much better that the defaults for our very individual situation. We now have an oldSpace of 128MB, a newSpace of 256 MB (which leaves some air at the top) and an allocationIncrement of 8MB.

So we could push the wall a few minutes ahead, but still hit it ;-)

The good news: we also found some other problem in our code / object model that made the object allocation in such massive numbers necessary. With that change to our program, we could allow our user to work for almost two hours without any (noticable) effect. She started with commit times of ~350 ms, and after almost two hours we were still somehwere between 1700 and 2500 ms.

So maybe the whole GC tweaking stuff wasn't even necessary. The biggest problem was in our code, not GC or scavenges etc, even though tewaking it could put some paint on the stain ;-) Thanks to everybody who helped and provided input! As always, this list is a place to learn! Thanks guys for enabling this.

Joachim

wobotzen schrieb am Donnerstag, 3. Dezember 2020 um 09:44:47 UTC+1:

Hallo

donot be afraid of increasing newspace...., we use 512MB , and it is also possible to change that on demand

I would try to start at 64MB and measure the results with EsbSampler (you can see the scavenges and gc there )

[hidden email] schrieb am Mittwoch, 2. Dezember 2020 um 09:14:17 UTC+1:

It has been a few decades since I looked at VA GC memory settings. However as mention they are too small (perhaps)

Things to consider for a flip/swap/scavenge.

How long things will take to perform the copy of the living objects between the two semi spaces. Is is a concern? If you need to service a socket interrupt how long can you delay for?

The impact on the old space to new space remember table. I don’t know how that is implemented for growth or for scanning performance in VA.

How do weak objects get cleaned up, is it tied to a scavenge? 

Beware of making new space a multiple of your larger short lived objects, I’ve seen cases where 4mb happily holds 40 100k objects but all those 20 byte objects spill into old space, adjusting to 4,099k lessened early tenuring.

A repeatable test case is alway helpful

Measure business logic thruput, and if needed latency.

Sent from ProtonMail Mobile

On Tue, Dec 1, 2020 at 23:53, Richard Sargent <[hidden email]> wrote:

On Tuesday, December 1, 2020 at 11:36:05 PM UTC-8 [hidden email] wrote:

Richard,

thanks a lot for your thoughts. One thing I thought about after writing my post was that maybe I first need to make the increase steps bigger, so that whenever a user needs more memory, they get either way too much or, in case of users who really need a lot, don't have to allocate too often. So today we're testing with "enough" newSpace and an allocationIncrement of 8 MBytes.

Let's see how far we get with this.

Thanks especially for confirming the default numbers stem from a time when a megabyte was beyond what a "high end" PC had available or could address...

I am not so sure I understand your point about swapping. Are you talking about the classical memory swapping of the host's O/S? in this case, we don't have to worry. The server machines have plenty of memory. Our images now start with a memory footprint of 620 MB, the server has 64GB RAM, so I guess this can be ignored in this specific situation.

Sorry. I was referring to the two halves of new space. When one half fills, the still referenced objects need to be copied to the other half and then things proceed with it filling. The bigger the halves, the longer it takes to do that process. This is what I meant by "swapping".

Arguably, with a big new space, the swap from half to half would be less frequent, but would take longer, for some unclear notion of longer. That would affect all your users versus the few who have really large new space requirements from time to time.

I don't know how your system is architected. If it were a web server with users randomly distributed among multiple images for each request, you could possibly target requests known to be large to (a small number of) images with large new spaces and the rest of the requests to a larger pool of not so large new space images. The ability to decide that is uncertain and perhaps impractical.

My biggest concern here is that our new settings cause negative side effects for our typical users who never come close to these memory requirements...

Finding the optimum is likely trial and error, with more error than trial. :-)

Joachim

Richard Sargent schrieb am Dienstag, 1. Dezember 2020 um 23:51:45 UTC+1:

On Monday, November 30, 2020 at 9:32:36 PM UTC-8 [hidden email] wrote:

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

A couple of points, based on empirical evidence.

The defaults for the memory management parameters (and pretty well all the others) were established in the early 90s for what is now considered fairly limited hardware capabilities. I don't think any of them have changed since VA 3.x and I'm guessing not since OTI's original product.

When I worked at Geico on their original primary sales application, I did bump up some of the memory management parameters for a noticeable improvement in performance. I don't think I bumped them up too much. Perhaps new space to 10 MB or so. I don't think you want new space too large, otherwise your new space swapping performance goes down, and it's possibly the most critical memory space.

It sounds like you have good metering tools in your application, so I suggest you conduct incremental experiments. Try 16 MB and see how the stats compare to the current settings. If they improve, try 32 MB and see what happens, repeat. You may even want to start smaller, lest you miss the sweet spot entirely.

You can also register something to be informed of GC events. See Process class>>#addGCEventHandler: and the corresponding remove partner. It's called from Process class>>#gcEvent:. This would allow you to capture frequency counts of scavenges versus global garbage collections.

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/62ebc3f4-17d6-48d8-b08e-88bec3e16694n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/9cbb20fc-25e3-453a-82b7-0022c8179ff3n%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/

Hi Mariano,

thanks, this looks extremely helpful. I am afraid I will have to add it tom my personal toolbox...
Do you happen to also have something that helps finding out whether and/or when objects moves from newSpace to oldSpace (get tenured is the right term, I think)?

BTW: just for the record: the objects aren't created unnecessarily. They are just kept in a GlorpSession's #registeredObjects although most of them will never be updated or deleted. They are search results for the GUI or are needed for some calculations and plausibility checks. The science here is to find a more or less automated way of telling whether or not they will potentially by updated or not and thus should be managed by the current unit of work or not. So completely unrelated to GC and such and to be solved on the application side of things rather than VAST.

Joachim

[hidden email] schrieb am Donnerstag, 3. Dezember 2020 um 13:00:55 UTC+1:

Greetings all,

Just wanted to throw an out-of-the-box tool that may detect this kind of issue in which you generate a huge unnecessary amount of objects. This tool is intended for a development image, but it may still be useful. 

1) Load feature "z.ST: Performance Workbench"

2) Evaluate:  "EsMemoryUseReport openMemorySnapShow". That opens a table widget with classes and amount of instances and refreshes every second. See attached example. 

Best,

On Thu, Dec 3, 2020 at 8:35 AM Joachim Tuchel <[hidden email]> wrote:

Wolfgang,

thanks for your tip. I even went so far to go to 256 MB newSpace following your advice, in addition to increasing the allocationIncrement tp 8 MB.

The images start big now, but I could get rid of a few GC cycles during this operation.

So far I can tell it imporves the situation for the affected "heavy users" by postponing the point where the system gets slow by quite some repetitions of the affected action. And it doesn't hurt our "normal" users, which was my biggest concern. So I think we found a combination that is much better that the defaults for our very individual situation. We now have an oldSpace of 128MB, a newSpace of 256 MB (which leaves some air at the top) and an allocationIncrement of 8MB.

So we could push the wall a few minutes ahead, but still hit it ;-)

The good news: we also found some other problem in our code / object model that made the object allocation in such massive numbers necessary. With that change to our program, we could allow our user to work for almost two hours without any (noticable) effect. She started with commit times of ~350 ms, and after almost two hours we were still somehwere between 1700 and 2500 ms.

So maybe the whole GC tweaking stuff wasn't even necessary. The biggest problem was in our code, not GC or scavenges etc, even though tewaking it could put some paint on the stain ;-) Thanks to everybody who helped and provided input! As always, this list is a place to learn! Thanks guys for enabling this.

Joachim

wobotzen schrieb am Donnerstag, 3. Dezember 2020 um 09:44:47 UTC+1:

Hallo

donot be afraid of increasing newspace...., we use 512MB , and it is also possible to change that on demand

I would try to start at 64MB and measure the results with EsbSampler (you can see the scavenges and gc there )

[hidden email] schrieb am Mittwoch, 2. Dezember 2020 um 09:14:17 UTC+1:

It has been a few decades since I looked at VA GC memory settings. However as mention they are too small (perhaps)

Things to consider for a flip/swap/scavenge.

How long things will take to perform the copy of the living objects between the two semi spaces. Is is a concern? If you need to service a socket interrupt how long can you delay for?

The impact on the old space to new space remember table. I don’t know how that is implemented for growth or for scanning performance in VA.

How do weak objects get cleaned up, is it tied to a scavenge? 

Beware of making new space a multiple of your larger short lived objects, I’ve seen cases where 4mb happily holds 40 100k objects but all those 20 byte objects spill into old space, adjusting to 4,099k lessened early tenuring.

A repeatable test case is alway helpful

Measure business logic thruput, and if needed latency.

Sent from ProtonMail Mobile

On Tue, Dec 1, 2020 at 23:53, Richard Sargent <[hidden email]> wrote:

On Tuesday, December 1, 2020 at 11:36:05 PM UTC-8 [hidden email] wrote:

Richard,

thanks a lot for your thoughts. One thing I thought about after writing my post was that maybe I first need to make the increase steps bigger, so that whenever a user needs more memory, they get either way too much or, in case of users who really need a lot, don't have to allocate too often. So today we're testing with "enough" newSpace and an allocationIncrement of 8 MBytes.

Let's see how far we get with this.

Thanks especially for confirming the default numbers stem from a time when a megabyte was beyond what a "high end" PC had available or could address...

I am not so sure I understand your point about swapping. Are you talking about the classical memory swapping of the host's O/S? in this case, we don't have to worry. The server machines have plenty of memory. Our images now start with a memory footprint of 620 MB, the server has 64GB RAM, so I guess this can be ignored in this specific situation.

Sorry. I was referring to the two halves of new space. When one half fills, the still referenced objects need to be copied to the other half and then things proceed with it filling. The bigger the halves, the longer it takes to do that process. This is what I meant by "swapping".

Arguably, with a big new space, the swap from half to half would be less frequent, but would take longer, for some unclear notion of longer. That would affect all your users versus the few who have really large new space requirements from time to time.

I don't know how your system is architected. If it were a web server with users randomly distributed among multiple images for each request, you could possibly target requests known to be large to (a small number of) images with large new spaces and the rest of the requests to a larger pool of not so large new space images. The ability to decide that is uncertain and perhaps impractical.

My biggest concern here is that our new settings cause negative side effects for our typical users who never come close to these memory requirements...

Finding the optimum is likely trial and error, with more error than trial. :-)

Joachim

Richard Sargent schrieb am Dienstag, 1. Dezember 2020 um 23:51:45 UTC+1:

On Monday, November 30, 2020 at 9:32:36 PM UTC-8 [hidden email] wrote:

Hi there,

we do have problems with slow transactions in Glorp on VAST 9.2 once the number of objects in memory (or better: in a UnitOfWork) gets relatively high, somewhere between 15000 and 20000. What we've found out so far is that the lag shows up shortly before any SQL is generated, during a phase where Glorp creates a RowMap (in-memory copy of a row to be written to the DB) fo revery object that might have to be saved by an INSERT or UPDATE statement. These RowMaps are then used to compare them against a copy of the row that once was read from the DB. So depending on the scenario, most of these row maps will just be compared against another one and be garbage just a few milliseconds later.
I only mention this in order to explain why a commit of a transaction may allocate and create lots of garbarge, it is not relevant to the actual problem. The short form is: we do have users which suddenly create 40MB or more of objects just for the sake of ignoring 39MB of them just a second later.

That is not a problem per se, we think we found out this only makes the image lag for literally two minues when there are a few gc's during that time. Which I think makes sense, because VAST needs to make room for lots of new objects fast.

Interestingly, this only affects users that have a large number of objects in a UnitOfWork. Most users never get into any troubles and commit times are somewhere around 300-800 msecs.

So here I am, wondering if playingwith newSpaceSize would make things better. In my naive picture, more NewSpace means less scavenging and less garbage collection. I also think/hope that moving bits around during scavenging is not a real show stopper, even if we increase newSpace to 64 MB or bigger. My theory is that it's the garbage collection process (analysis of needed/unneeded buffers) that takes time, not the moving around.

So I would guess that increasing newSpace will help by reducing the number of gc's in such a situation.

The scenario above sounds to me like there is little use in tweaking oldSpace, because most of my objects won't be old enough to be moved to OldSpace.

So what I am looking for is some discussion on GarbageCollection and these values in the ini (or command line) for mortals. What are the implications of increasing one parameter and what are the effects of one parameter to another.

What scares me a little is that most of the parameter comments in the .ini that ships with VAST are talking about numbers like 2 MB or even 256K. What if I turn the knob from 2 MB to 64MB, that's an increase by 32... The gods of VAST may have good reasons to keep these values small even with a 64bits VM, where we are speaking of theoretical image sizes of 2 GB, 4GB or even more....

A couple of points, based on empirical evidence.

The defaults for the memory management parameters (and pretty well all the others) were established in the early 90s for what is now considered fairly limited hardware capabilities. I don't think any of them have changed since VA 3.x and I'm guessing not since OTI's original product.

When I worked at Geico on their original primary sales application, I did bump up some of the memory management parameters for a noticeable improvement in performance. I don't think I bumped them up too much. Perhaps new space to 10 MB or so. I don't think you want new space too large, otherwise your new space swapping performance goes down, and it's possibly the most critical memory space.

It sounds like you have good metering tools in your application, so I suggest you conduct incremental experiments. Try 16 MB and see how the stats compare to the current settings. If they improve, try 32 MB and see what happens, repeat. You may even want to start smaller, lest you miss the sweet spot entirely.

You can also register something to be informed of GC events. See Process class>>#addGCEventHandler: and the corresponding remove partner. It's called from Process class>>#gcEvent:. This would allow you to capture frequency counts of scavenges versus global garbage collections.

Is there any reading material you can recommend? I am not so much interested in understanding the merits of one GC algorithm over another, or the mathematical models of things, just an explanation deep enough to understand what may or may not make sense.

Concrete questions I have right now:

• Does it make sense to increase newSpaceSize to 64MB without touching any other paremeter?
  (To my understanding, a newSpace of 64MB is only big enough for a maximim of somehwat near 32MB of objects - scavenging splits this space to half)
• So if I know some transactions need 70MB, this still means at least 3 GC's , so should I better go north of 128MB because this specific activity needs all objects of these objects, but only once - and what negative implicatios might this have (RAM is not an issue on our servers these days),
  
• Are there parameters that might have a positive influence in our situation? Like the limits of how much free minimum space a GC must have after it's done?
• What implications - other than space requirements - does a huge newSpace have? Does it slow down every GC by large, just for one special situation?
• 
  

I know it would be best to avoid garbage in general. It's just not easy if you need a solution to a problem NOW...

Any comments, ideas, reading tips?

Joachim

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/62ebc3f4-17d6-48d8-b08e-88bec3e16694n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "VA Smalltalk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].

To view this discussion on the web visit https://groups.google.com/d/msgid/va-smalltalk/9cbb20fc-25e3-453a-82b7-0022c8179ff3n%40googlegroups.com.

--

Mariano Martinez Peck
Software Engineer, Instantiations Inc.

Email: [hidden email]

Twitter: https://twitter.com/MartinezPeck

LinkedIn: www.linkedin.com/in/mariano-martinez-peck

Blog: https://marianopeck.wordpress.com/