Data
Locked
 
|
Hi,
Now I have registered some 500 records of data consisting of 4 integers, 5 floats and 7 strings totaling on average 30-40 characters, and it seems I cannot put it all in a collection without hitting the roof of my 960 MB RAM. The data itself shouldn't make out more than about 35-40 kB. Is there so much overhead in the system? It seems impossible, but my systems performance chart apparently indicates that I am exhausting my memory as I am trying to store this data in a SortedCollection, which I then return from a method. What is happening exactly? LEF |
|
|
Lars Finsen wrote:
> Now I have registered some 500 records of data consisting of 4 > integers, 5 floats and 7 strings totaling on average 30-40 characters, > and it seems I cannot put it all in a collection without hitting the > roof of my 960 MB RAM. The data itself shouldn't make out more than > about 35-40 kB. Is there so much overhead in the system? It seems > impossible, but my systems performance chart apparently indicates that > I am exhausting my memory as I am trying to store this data in a > SortedCollection, which I then return from a method. What is happening > exactly? Can you share the stack you see when you hit your memory limit? Could the implementation of comparison in the record class be recursive or suffer from an infinite message send loop? Thanks, Andres. |
|
|
Andres Valloud wrote: > quoting me: >> Now I have registered some 500 records of data consisting of 4 >> integers, 5 floats and 7 strings totaling on average 30-40 >> characters, and it seems I cannot put it all in a collection >> without hitting the roof of my 960 MB RAM. The data itself >> shouldn't make out more than about 35-40 kB. Is there so much >> overhead in the system? It seems impossible, but my systems >> performance chart apparently indicates that I am exhausting my >> memory as I am trying to store this data in a SortedCollection, >> which I then return from a method. What is happening exactly? > Can you share the stack you see when you hit your memory limit? > Could the implementation of comparison in the record class be > recursive or suffer from an infinite message send loop? I am afraid I have no such thing, as the system crashed without giving me a route into the debugger. (I am working with MS Windows for this project.) My solution now is to select or discard my data as I read it instead of loading it all. But I will look for any of the two possible problem sources you mention. Since the collection I have been loading the data into is a SortedCollection, I needed to implement my own comparisons for the record class, and maybe these are not watertight. During my testing earlier today the data loaded successfully from a file with a few records removed. I might find data in those records that require me to refine my comparisons perhaps... LEF |
|
|
Try loading StackOverflow parcel and running your code, this might point
out a recursion problem, Cheers! -Boris -- +1.604.689.0322 DeepCove Labs Ltd. 4th floor 595 Howe Street Vancouver, Canada V6C 2T5 http://tinyurl.com/r7uw4 [hidden email] CONFIDENTIALITY NOTICE This email is intended only for the persons named in the message header. Unless otherwise indicated, it contains information that is private and confidential. If you have received it in error, please notify the sender and delete the entire message including any attachments. Thank you. > -----Original Message----- > From: Lars Finsen [mailto:[hidden email]] > Sent: Sunday, July 15, 2007 5:13 PM > To: [hidden email] > Subject: Re: Data > > > Andres Valloud wrote: > > > quoting me: > >> Now I have registered some 500 records of data consisting of 4 > >> integers, 5 floats and 7 strings totaling on average 30-40 > >> characters, and it seems I cannot put it all in a collection > >> without hitting the roof of my 960 MB RAM. The data itself > >> shouldn't make out more than about 35-40 kB. Is there so much > >> overhead in the system? It seems impossible, but my systems > >> performance chart apparently indicates that I am exhausting my > >> memory as I am trying to store this data in a SortedCollection, > >> which I then return from a method. What is happening exactly? > > Can you share the stack you see when you hit your memory limit? > > Could the implementation of comparison in the record class be > > recursive or suffer from an infinite message send loop? > > I am afraid I have no such thing, as the system crashed without > giving me a route into the debugger. (I am working with MS Windows > for this project.) My solution now is to select or discard my data as > I read it instead of loading it all. But I will look for any of the > two possible problem sources you mention. Since the collection I have > been loading the data into is a SortedCollection, I needed to > implement my own comparisons for the record class, and maybe these > are not watertight. During my testing earlier today the data loaded > successfully from a file with a few records removed. I might find > data in those records that require me to refine my comparisons > perhaps... > > LEF |
|
|
You could also try doing a multi-allocation-profile on a smaller data
set (one small enough to not crash the system). This might show something allocating storage that you didn't expect. The profilers are available from Transcript -> Tools -> Advanced -> Profiles. David |
|
|
In reply to this post by Boris Popov, DeepCove Labs (SNN)
Den 16. jul. 2007 kl. 02.19 skrev Boris Popov: > Try loading StackOverflow parcel and running your code, this might > point out a recursion problem, I found it and loaded it and ran my code with the old method loading all the data, but it crashed as before with nothing indicating any activity or reaction from the parcel. Is it supposed to write a report somewhere, or what? LEF |
|
|
In reply to this post by Lars Finsen
Hi Lars,
Can you check the Memory Policy settings of your image. You can find these in VisualLauncher>System>Settings And then go to System>MemoryPolicy Perhaps these values are set to very low values. There is also a 'help' button explaining the different settings. -----Original Message----- From: Lars Finsen [mailto:[hidden email]] Sent: maandag 16 juli 2007 12:26 To: [hidden email] Subject: Re: Data Den 16. jul. 2007 kl. 02.19 skrev Boris Popov: > Try loading StackOverflow parcel and running your code, this might > point out a recursion problem, I found it and loaded it and ran my code with the old method loading all the data, but it crashed as before with nothing indicating any activity or reaction from the parcel. Is it supposed to write a report somewhere, or what? LEF |
|
|
Den 16. jul. 2007 kl. 12.49 skrev Mark Plas: > Can you check the Memory Policy settings of your image. Well, it appears that Free Memory is 8 MB, which is a bit low, but not that extreme. Both the Windows and the MAC OSX applications have exactly the same figures, so I guess the installations don't check the local resources as they are set up, do they? Apparently there's many parts of the system that I'm not been acquainted with so far. Guess I should explore it a little. LEF |
|
|
In reply to this post by Lars Finsen
Hi Lars,
The 'free memory' option is not that important. It's the value of the 'memory' parameter that can be important here. Its value indicates the upper bound of memory VW is going to use. This means that if it would have (for instance) a value of '10000000' (10 million), your visualworks application will never use more than 10MB of memory, no matter how much memory your computer has. So, you're right, VW doesn't check the resources available on your machine. You have to manually supply values for its memory management. Could you check the value that's supplied there? Perhaps it's a bit low. -----Original Message----- From: Lars Finsen [mailto:[hidden email]] Sent: maandag 16 juli 2007 13:34 To: [hidden email] Subject: Re: Data Den 16. jul. 2007 kl. 12.49 skrev Mark Plas: > Can you check the Memory Policy settings of your image. Well, it appears that Free Memory is 8 MB, which is a bit low, but not that extreme. Both the Windows and the MAC OSX applications have exactly the same figures, so I guess the installations don't check the local resources as they are set up, do they? Apparently there's many parts of the system that I'm not been acquainted with so far. Guess I should explore it a little. LEF |
|
|
Den 16. jul. 2007 kl. 13.41 skrev Mark Plas: > This means that if it would have (for instance) a value of '10000000' > (10 million), your visualworks application will never use more than > 10MB of memory, no matter how much memory your computer has. So, > you're > right, VW doesn't check the resources available on your machine. You > have to manually supply values for its memory management. Yes, I see what you mean. > Could you check the value that's supplied there? Perhaps it's a bit > low. It's more than 500 MB, so that should not be the problem. BTW, curious that the list reply-to address here is set to sender instead of the list. That means you have to manually cnange the recipients address if your reply has any public interest, which it has most often I guess, since we are here to discuss Smalltalk. Does it have to be like this? LEF |
|
|
One comment -- with the data sizes you first presented -- there is no
way you should be anywhere near running out of memory -- you should easily be able to do 100x that. Does it take very long to process your data? If so, you might try interrupting the process to see things like stack size -- use CTRL+Y to interrupt. Lars Finsen wrote: > > Den 16. jul. 2007 kl. 13.41 skrev Mark Plas: > >> This means that if it would have (for instance) a value of '10000000' >> (10 million), your visualworks application will never use more than >> 10MB of memory, no matter how much memory your computer has. So, you're >> right, VW doesn't check the resources available on your machine. You >> have to manually supply values for its memory management. > > Yes, I see what you mean. > >> Could you check the value that's supplied there? Perhaps it's a bit low. > > It's more than 500 MB, so that should not be the problem. > > BTW, curious that the list reply-to address here is set to sender > instead of the list. That means you have to manually cnange the > recipients address if your reply has any public interest, which it has > most often I guess, since we are here to discuss Smalltalk. Does it > have to be like this? mushroom to lots of recipients. > > LEF > -- Dennis Smith +1 416.798.7948 Cherniak Software Development Corporation Fax: +1 416.798.0948 509-2001 Sheppard Avenue East [hidden email] Toronto, ON M2J 4Z8 sip:[hidden email] Canada http://www.CherniakSoftware.com Entrance off Yorkland Blvd south of Sheppard Ave east of the DVP |
«
Return to VisualWorks
|
1 view|%1 views
| Free forum by Nabble | Edit this page |