Store in VW7.7 -- memory++++

We have been using Store (with VW7.6) and have our memory
upper-limit set to 300,000,000, with base image size of 130,000,000 - no
problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

When does the memory spike occur? Does the size of the image go back  
down after a stave, close, restart?

David.

~~
On Feb 17, 2010, at 2:47 PM, Dennis Smith wrote:

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

You may find additional information by using the package "Memory
Monitor" from the public store repository.  It's an improved version of
John Brant's and Don Roberts' Memory Monitor.  Among other things, it
can write CSV logs for easier analysis.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of David Caster
Sent: Wednesday, February 17, 2010 3:10 PM
To: VWNC,
Subject: Re: [vwnc] Store in VW7.7 -- memory++++

When does the memory spike occur? Does the size of the image go back
down after a stave, close, restart?

David.

~~
On Feb 17, 2010, at 2:47 PM, Dennis Smith wrote:

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dennis:

> Is this due to GLORP?

It could be, but unlikely... A Store/Glorp Session object caches objects it touches (in order to preserve identity among other reasons). However, Session objects are used when accessing the DB, and then thrown away once their job is done (for instance during a Load or Publish)... There is no global Session (although we have thought about it).

As far as 7.7 goes though, unless there is uncollected garbage, these sessions aren't used anywhere else in Store.

                                And So It Goes
                                     Sames
______________________________________________________________________

Samuel S. Shuster [|]
VisualWorks Engineering, Store Project
Smalltalk Enables Success -- What Are YOU Using?





_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Dennis, Steven Kelly wrote this a bit ago.  Maybe it's related?  I have
no idea myself, I am not an expert in these matters.

=============
Publishing to Store has become an order of magnitude slower for me in
7.7 compared to earlier versions. Publishing the base differences
(reconciled with 7.6) was taking several hours. The garbage collect
cursor was showing all the time, but actual memory usage was staying
constant (we set the growth upper bound to 64MB; I tried upping it to
128MB but that seemed just to slow things down further).

The garbage was being created in
ODBCLargeObjectBuffer>>bindInputExternal, where it breaks the buffer of
SQL arguments into chunks. In the line:

self at: 1 put: (valueStream nextAvailable:  (bufferSession
getDataChunkSize min: cbValueMax)).

both "bufferSession getDataChunkSize" and cbValueMax were 1MB. Each time
this line was sent the system thus allocated a new 1MB ByteArray in
#nextAvailable. Since that won't fit in Eden or LargeSpace (by default
<300KB), it went straight to OldSpace, which quickly filled to the
growth upper bound. When the next allocation failed it triggered a full
compacting GC of all but PermSpace, which takes something of the order
of a second. Saving the image with new startup sizes to make LargeSpace
big enough would probably have helped, but the real problem was that the
actual content of valueStream was almost always much less than the
ByteArray allocated.

I thus made a quick hack of adding "min: bufferValue size", i.e. no
bigger than the total collection from which valueStream is being read.
This seemed to solve the problem.

The only downside is that now I have my hacky fix published as part of
the 7.7 base... Anyone have hints as to the Glorp Store invocation to
make my 7.7 version of ODBCEXDI point to the 7.6 version of that
method?!

Also, it's odd that things slowed down now, if that method is unchanged;
presumably something else changed elsewhere in the code, since our
environment is unchanged from 7.6 (Win XP to SQLServer 2000 over a
100Mbps LAN).

Steve
=============

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Dennis Smith
Sent: Wednesday, February 17, 2010 2:48 PM
To: VWNC,
Subject: [vwnc] Store in VW7.7 -- memory++++

We have been using Store (with VW7.6) and have our memory upper-limit
set to 300,000,000, with base image size of 130,000,000 - no problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

It's hard to know without more information. Do you see this on publishing? Loading? Both? What are the characteristics of the things you're working with at the time. Is this, for example, only when loading a very large bundle into a clean image? Or does it happen on smaller incremental loads and/or publishes? What database are you using. The issue Steven Kelly raised does sound like a promising candidate for large memory use against at least some ODBC databases. We've been looking at that, and have a proposed fix underway, so it may help.

At 05:47 PM 2010-02-17, Dennis Smith wrote:

We have been using Store (with VW7.6) and have our memory
upper-limit set to 300,000,000, with base image size of 130,000,000 - no
problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              sip:[hidden email]
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Thanks for ALL the replies -- let me answer most of the questions in one post ...

1. I seldom ever just load, so I don't know if its there on load, it is certainly there on publish and on reconcile which is most common activity
2. I have two bundles, each containing many packages not sure if that is a factor or not
3. When I first saw these, I was able to open (my own) memory policy window while the process-display with out-of-space was open -- this showed me at the max and a full GC did nothing to change that.  I changed the memory from 350Mb to 450Mb -- it ran out again, and this time I sent to 800Mb and it finished.  When finished a GC got it all back and I was down to my norman 130Mb
   
4. we are using Postgres
5. re Steven Kelly's slowness not -- yes it is slower -- I had not noted that because being a good engineer I changed from WinXP to Win7 and from VW7.6 to VW7.7 at the same time :) - moving to Win7 was not my choice - and my Win7 network access is slow (10x or more) - so I was not sure

I will monitor this over the next few weeks -- for now, when we do Store activity I will auto-raise the memory max


On 2/17/2010 7:46 PM, Alan Knight wrote:

It's hard to know without more information. Do you see this on publishing? Loading? Both? What are the characteristics of the things you're working with at the time. Is this, for example, only when loading a very large bundle into a clean image? Or does it happen on smaller incremental loads and/or publishes? What database are you using. The issue Steven Kelly raised does sound like a promising candidate for large memory use against at least some ODBC databases. We've been looking at that, and have a proposed fix underway, so it may help.

At 05:47 PM 2010-02-17, Dennis Smith wrote:

We have been using Store (with VW7.6) and have our memory
upper-limit set to 300,000,000, with base image size of 130,000,000 - no
problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a class="moz-txt-link-freetext" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

--

Alan Knight [|], Engineering Manager, Cincom Smalltalk

[hidden email]

[hidden email]

http://www.cincom.com/smalltalk

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
  

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a class="moz-txt-link-freetext" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

So here's something to try. In the method #maximumSizeToGroupWriteFor:, replace the constant 1000 with 1. That will make the publish slower (more network round trips), but use less memory at once. Postgresql in particular can accumulate a good bit of memory in strings while it does these operations. I doubt the ODBC issues is relevant. The Postgresql driver works completely differently.

So, I'm assuming you're seeing these as memory spikes during the actual operations, rather than as long-term memory consumption that persists once the operation is finished. Is that right?

I can imagine that reconcile of a large bundle might use a good bit of memory. It's going to read in the definition of the whole bundle and compare it against the image, so while that's in progress, there could be a lot of stuff in use. It might be possible to have it go package by package. That is, unless it already does it that way. I would expect publish to use less memory, because it should only need to be looking at the packages that have changed.

At 07:40 AM 2010-02-18, Dennis Smith wrote:

Thanks for ALL the replies -- let me answer most of the questions in one post ...

1. I seldom ever just load, so I don't know if its there on load, it is certainly there on publish and on reconcile which is most common activity
2. I have two bundles, each containing many packages not sure if that is a factor or not
3. When I first saw these, I was able to open (my own) memory policy window while the process-display with out-of-space was open -- this showed me at the max and a full GC did nothing to change that.  I changed the memory from 350Mb to 450Mb -- it ran out again, and this time I sent to 800Mb and it finished.  When finished a GC got it all back and I was down to my norman 130Mb
4. we are using Postgres
5. re Steven Kelly's slowness not -- yes it is slower -- I had not noted that because being a good engineer I changed from WinXP to Win7 and from VW7.6 to VW7.7 at the same time :) - moving to Win7 was not my choice - and my Win7 network access is slow (10x or more) - so I was not sure

I will monitor this over the next few weeks -- for now, when we do Store activity I will auto-raise the memory max


On 2/17/2010 7:46 PM, Alan Knight wrote:

It's hard to know without more information. Do you see this on publishing? Loading? Both? What are the characteristics of the things you're working with at the time. Is this, for example, only when loading a very large bundle into a clean image? Or does it happen on smaller incremental loads and/or publishes? What database are you using. The issue Steven Kelly raised does sound like a promising candidate for large memory use against at least some ODBC databases. We've been looking at that, and have a proposed fix underway, so it may help.

At 05:47 PM 2010-02-17, Dennis Smith wrote:

We have been using Store (with VW7.6) and have our memory
upper-limit set to 300,000,000, with base image size of 130,000,000 - no
problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a href="sip:dennis@CherniakSoftware.com"> sip:dennis@...
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

--
Alan Knight [|], Engineering Manager, Cincom Smalltalk
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

_______________________________________________
vwnc mailing list
[hidden email]

http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
 

-- 
Dennis
Smith                
   
        
         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1
416.798.0948
509-2001 Sheppard Avenue East       
[hidden email]
Toronto, ON M2J
4Z8             
<a href="sip:dennis@CherniakSoftware.com">
sip:dennis@...
Canada  
        
        
        

http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the
DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Hm -- I might rather the memory -- I have our developers yelling at me that the store publish is taking
lots longer than it used to -- me? I can't tell, I just got a Win7 "nice new PC" that accesses the network about 20x slower
than any other PC -- but others are talking about store being slower.  I will play a bit with this though just so
we can get a few answers.

On 2/18/2010 1:26 PM, Alan Knight wrote:

So here's something to try. In the method #maximumSizeToGroupWriteFor:, replace the constant 1000 with 1. That will make the publish slower (more network round trips), but use less memory at once. Postgresql in particular can accumulate a good bit of memory in strings while it does these operations. I doubt the ODBC issues is relevant. The Postgresql driver works completely differently.

So, I'm assuming you're seeing these as memory spikes during the actual operations, rather than as long-term memory consumption that persists once the operation is finished. Is that right?

I can imagine that reconcile of a large bundle might use a good bit of memory. It's going to read in the definition of the whole bundle and compare it against the image, so while that's in progress, there could be a lot of stuff in use. It might be possible to have it go package by package. That is, unless it already does it that way. I would expect publish to use less memory, because it should only need to be looking at the packages that have changed.

At 07:40 AM 2010-02-18, Dennis Smith wrote:

Thanks for ALL the replies -- let me answer most of the questions in one post ...

1. I seldom ever just load, so I don't know if its there on load, it is certainly there on publish and on reconcile which is most common activity
2. I have two bundles, each containing many packages not sure if that is a factor or not
3. When I first saw these, I was able to open (my own) memory policy window while the process-display with out-of-space was open -- this showed me at the max and a full GC did nothing to change that.  I changed the memory from 350Mb to 450Mb -- it ran out again, and this time I sent to 800Mb and it finished.  When finished a GC got it all back and I was down to my norman 130Mb
4. we are using Postgres
5. re Steven Kelly's slowness not -- yes it is slower -- I had not noted that because being a good engineer I changed from WinXP to Win7 and from VW7.6 to VW7.7 at the same time :) - moving to Win7 was not my choice - and my Win7 network access is slow (10x or more) - so I was not sure

I will monitor this over the next few weeks -- for now, when we do Store activity I will auto-raise the memory max


On 2/17/2010 7:46 PM, Alan Knight wrote:

It's hard to know without more information. Do you see this on publishing? Loading? Both? What are the characteristics of the things you're working with at the time. Is this, for example, only when loading a very large bundle into a clean image? Or does it happen on smaller incremental loads and/or publishes? What database are you using. The issue Steven Kelly raised does sound like a promising candidate for large memory use against at least some ODBC databases. We've been looking at that, and have a proposed fix underway, so it may help.

At 05:47 PM 2010-02-17, Dennis Smith wrote:

We have been using Store (with VW7.6) and have our memory
upper-limit set to 300,000,000, with base image size of 130,000,000 - no
problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a moz-do-not-send="true" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

--
Alan Knight [|], Engineering Manager, Cincom Smalltalk
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

_______________________________________________
vwnc mailing list
[hidden email]

http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
 
      

-- 
Dennis
Smith                
   
        
         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1
416.798.0948
509-2001 Sheppard Avenue East       
[hidden email]
Toronto, ON M2J
4Z8             
<a moz-do-not-send="true" href="sip:dennis@CherniakSoftware.com">
sip:dennis@...
Canada  
        
        
        

http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the
DVP

--

Alan Knight [|], Engineering Manager, Cincom Smalltalk

[hidden email]

[hidden email]

http://www.cincom.com/smalltalk

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a class="moz-txt-link-freetext" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Hm -- I might rather the memory -- I have our developers yelling at me that the store publish is taking
lots longer than it used to -- me? I can't tell, I just got a Win7 "nice new PC" that accesses the network about 20x slower
than any other PC -- but others are talking about store being slower.  I will play a bit with this though just so
we can get a few answers.

On 2/18/2010 1:26 PM, Alan Knight wrote:

So here's something to try. In the method #maximumSizeToGroupWriteFor:, replace the constant 1000 with 1. That will make the publish slower (more network round trips), but use less memory at once. Postgresql in particular can accumulate a good bit of memory in strings while it does these operations. I doubt the ODBC issues is relevant. The Postgresql driver works completely differently.

So, I'm assuming you're seeing these as memory spikes during the actual operations, rather than as long-term memory consumption that persists once the operation is finished. Is that right?

I can imagine that reconcile of a large bundle might use a good bit of memory. It's going to read in the definition of the whole bundle and compare it against the image, so while that's in progress, there could be a lot of stuff in use. It might be possible to have it go package by package. That is, unless it already does it that way. I would expect publish to use less memory, because it should only need to be looking at the packages that have changed.

At 07:40 AM 2010-02-18, Dennis Smith wrote:

Thanks for ALL the replies -- let me answer most of the questions in one post ...

1. I seldom ever just load, so I don't know if its there on load, it is certainly there on publish and on reconcile which is most common activity
2. I have two bundles, each containing many packages not sure if that is a factor or not
3. When I first saw these, I was able to open (my own) memory policy window while the process-display with out-of-space was open -- this showed me at the max and a full GC did nothing to change that.  I changed the memory from 350Mb to 450Mb -- it ran out again, and this time I sent to 800Mb and it finished.  When finished a GC got it all back and I was down to my norman 130Mb
4. we are using Postgres
5. re Steven Kelly's slowness not -- yes it is slower -- I had not noted that because being a good engineer I changed from WinXP to Win7 and from VW7.6 to VW7.7 at the same time :) - moving to Win7 was not my choice - and my Win7 network access is slow (10x or more) - so I was not sure

I will monitor this over the next few weeks -- for now, when we do Store activity I will auto-raise the memory max


On 2/17/2010 7:46 PM, Alan Knight wrote:

It's hard to know without more information. Do you see this on publishing? Loading? Both? What are the characteristics of the things you're working with at the time. Is this, for example, only when loading a very large bundle into a clean image? Or does it happen on smaller incremental loads and/or publishes? What database are you using. The issue Steven Kelly raised does sound like a promising candidate for large memory use against at least some ODBC databases. We've been looking at that, and have a proposed fix underway, so it may help.

At 05:47 PM 2010-02-17, Dennis Smith wrote:

We have been using Store (with VW7.6) and have our memory
upper-limit set to 300,000,000, with base image size of 130,000,000 - no
problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a moz-do-not-send="true" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

--
Alan Knight [|], Engineering Manager, Cincom Smalltalk
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

_______________________________________________
vwnc mailing list
[hidden email]

http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
 
      

-- 
Dennis
Smith                
   
        
         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1
416.798.0948
509-2001 Sheppard Avenue East       
[hidden email]
Toronto, ON M2J
4Z8             
<a moz-do-not-send="true" href="sip:dennis@CherniakSoftware.com">
sip:dennis@...
Canada  
        
        
        

http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the
DVP

--

Alan Knight [|], Engineering Manager, Cincom Smalltalk

[hidden email]

[hidden email]

http://www.cincom.com/smalltalk

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a class="moz-txt-link-freetext" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Nice thought re the IPv6 thanks -- didn't help :(   I still get 1Mbyte per second transfer rate, 1Gbit network.

On 2/18/2010 3:19 PM, Boris Popov, DeepCove Labs (YVR) wrote:

Touch off topic, but maybe try disabling IPv6, I'd heard rumblings of that causing issues on some machines; my own Win7 experience so far has been quite pleasant.

-Boris (via BlackBerry)

---

From: [hidden email] [hidden email]
To: Alan Knight [hidden email]
Cc: VWNC, [hidden email]
Sent: Thu Feb 18 12:10:51 2010
Subject: Re: [vwnc] Store in VW7.7 -- memory++++

Hm -- I might rather the memory -- I have our developers yelling at me that the store publish is taking
lots longer than it used to -- me? I can't tell, I just got a Win7 "nice new PC" that accesses the network about 20x slower
than any other PC -- but others are talking about store being slower.  I will play a bit with this though just so
we can get a few answers.

On 2/18/2010 1:26 PM, Alan Knight wrote:

So here's something to try. In the method #maximumSizeToGroupWriteFor:, replace the constant 1000 with 1. That will make the publish slower (more network round trips), but use less memory at once. Postgresql in particular can accumulate a good bit of memory in strings while it does these operations. I doubt the ODBC issues is relevant. The Postgresql driver works completely differently.

So, I'm assuming you're seeing these as memory spikes during the actual operations, rather than as long-term memory consumption that persists once the operation is finished. Is that right?

I can imagine that reconcile of a large bundle might use a good bit of memory. It's going to read in the definition of the whole bundle and compare it against the image, so while that's in progress, there could be a lot of stuff in use. It might be possible to have it go package by package. That is, unless it already does it that way. I would expect publish to use less memory, because it should only need to be looking at the packages that have changed.

At 07:40 AM 2010-02-18, Dennis Smith wrote:

Thanks for ALL the replies -- let me answer most of the questions in one post ...

1. I seldom ever just load, so I don't know if its there on load, it is certainly there on publish and on reconcile which is most common activity
2. I have two bundles, each containing many packages not sure if that is a factor or not
3. When I first saw these, I was able to open (my own) memory policy window while the process-display with out-of-space was open -- this showed me at the max and a full GC did nothing to change that.  I changed the memory from 350Mb to 450Mb -- it ran out again, and this time I sent to 800Mb and it finished.  When finished a GC got it all back and I was down to my norman 130Mb
4. we are using Postgres
5. re Steven Kelly's slowness not -- yes it is slower -- I had not noted that because being a good engineer I changed from WinXP to Win7 and from VW7.6 to VW7.7 at the same time :) - moving to Win7 was not my choice - and my Win7 network access is slow (10x or more) - so I was not sure

I will monitor this over the next few weeks -- for now, when we do Store activity I will auto-raise the memory max


On 2/17/2010 7:46 PM, Alan Knight wrote:

It's hard to know without more information. Do you see this on publishing? Loading? Both? What are the characteristics of the things you're working with at the time. Is this, for example, only when loading a very large bundle into a clean image? Or does it happen on smaller incremental loads and/or publishes? What database are you using. The issue Steven Kelly raised does sound like a promising candidate for large memory use against at least some ODBC databases. We've been looking at that, and have a proposed fix underway, so it may help.

At 05:47 PM 2010-02-17, Dennis Smith wrote:

We have been using Store (with VW7.6) and have our memory
upper-limit set to 300,000,000, with base image size of 130,000,000 - no
problems.

With VW7.7 I have had to go as high as 700,000,000 for memory to prevent
out-of-space issues.

Is this due to GLORP?  something else??  something we did??

--
Dennis Smith                                       +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a moz-do-not-send="true" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada                              http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

--
Alan Knight [|], Engineering Manager, Cincom Smalltalk
[hidden email]
[hidden email]
http://www.cincom.com/smalltalk

_______________________________________________
vwnc mailing list
[hidden email]

http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
 
      

-- 
Dennis
Smith                
   
        
         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1
416.798.0948
509-2001 Sheppard Avenue East       
[hidden email]
Toronto, ON M2J
4Z8             
<a moz-do-not-send="true" href="sip:dennis@CherniakSoftware.com">
sip:dennis@...
Canada  
        
        
        

http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the
DVP

--

Alan Knight [|], Engineering Manager, Cincom Smalltalk

[hidden email]

[hidden email]

http://www.cincom.com/smalltalk

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a moz-do-not-send="true"
 class="moz-txt-link-freetext" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

-- 
Dennis Smith                 		         +1 416.798.7948
Cherniak Software Development Corporation   Fax: +1 416.798.0948
509-2001 Sheppard Avenue East        [hidden email]
Toronto, ON M2J 4Z8              <a class="moz-txt-link-freetext" href="sip:dennis@CherniakSoftware.com">sip:dennis@...
Canada			         http://www.CherniakSoftware.com
Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc