Getting my head around low space

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Getting my head around low space

Max Leske
Hi

I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.

Thanks for any ideas.

Cheers,
Max
_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Adrian Lienhard
The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:

[adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
       ./Squeak VM Opt [<option>...] -- [<argument>...]

Common <option>s:
  -help                 print this help message, then exit
  -memory <size>[mk]    use fixed heap size (added to image size)
  -pathenc <enc>        set encoding for pathnames (default: macintosh)
  -headless             run in headless (no window) mode (default: false)

Notes:
  <imageName> defaults to `Squeak.image'.
  If `-memory' is not specified then the heap will grow dynamically.
  <argument>s are ignored, but are processed by the Squeak image.
  The first <argument> normally names a Squeak `script' to execute.
  Precede <arguments> by `--' to use default image.


HTH,
Adrian

On Oct 9, 2010, at 18:01 , Max Leske wrote:

> Hi
>
> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>
> Thanks for any ideas.
>
> Cheers,
> Max
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Max Leske
Cool! I'll try that, thanks.

I just wonder: It says in the Notes "If `-memory' is not specified then the heap will grow dynamically.". As I understand that, the image should not run out of space before the OS does (or the address range assigned by the OS).  So can this option reserve more memory than the OS would otherwise allow?
I'll try it out and let you know what happened.

Cheers,
Max



On 09.10.2010, at 19:48, Adrian Lienhard wrote:

> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>
> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>       ./Squeak VM Opt [<option>...] -- [<argument>...]
>
> Common <option>s:
>  -help                 print this help message, then exit
>  -memory <size>[mk]    use fixed heap size (added to image size)
>  -pathenc <enc>        set encoding for pathnames (default: macintosh)
>  -headless             run in headless (no window) mode (default: false)
>
> Notes:
>  <imageName> defaults to `Squeak.image'.
>  If `-memory' is not specified then the heap will grow dynamically.
>  <argument>s are ignored, but are processed by the Squeak image.
>  The first <argument> normally names a Squeak `script' to execute.
>  Precede <arguments> by `--' to use default image.
>
>
> HTH,
> Adrian
>
> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>
>> Hi
>>
>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>
>> Thanks for any ideas.
>>
>> Cheers,
>> Max
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Levente Uzonyi-2
On Sat, 9 Oct 2010, Max Leske wrote:

> Cool! I'll try that, thanks.
>
> I just wonder: It says in the Notes "If `-memory' is not specified then the heap will grow dynamically.". As I understand that, the image should not run out of space before the OS does (or the address range assigned by the OS).  So can this option reserve more memory than the OS would otherwise allow?
> I'll try it out and let you know what happened.

-memory allocates the memory when the VM starts. Use -mmap (un un*x) if
you want the VM to dinamically grow the heap. AFAIK the windows VM doesn't
support more than 512MB.


Levente

>
> Cheers,
> Max
>
>
>
> On 09.10.2010, at 19:48, Adrian Lienhard wrote:
>
>> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>>
>> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
>> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>>       ./Squeak VM Opt [<option>...] -- [<argument>...]
>>
>> Common <option>s:
>>  -help                 print this help message, then exit
>>  -memory <size>[mk]    use fixed heap size (added to image size)
>>  -pathenc <enc>        set encoding for pathnames (default: macintosh)
>>  -headless             run in headless (no window) mode (default: false)
>>
>> Notes:
>>  <imageName> defaults to `Squeak.image'.
>>  If `-memory' is not specified then the heap will grow dynamically.
>>  <argument>s are ignored, but are processed by the Squeak image.
>>  The first <argument> normally names a Squeak `script' to execute.
>>  Precede <arguments> by `--' to use default image.
>>
>>
>> HTH,
>> Adrian
>>
>> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>>
>>> Hi
>>>
>>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>>
>>> Thanks for any ideas.
>>>
>>> Cheers,
>>> Max
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Levente Uzonyi-2
In reply to this post by Adrian Lienhard
On Sat, 9 Oct 2010, Adrian Lienhard wrote:

> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>
> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>       ./Squeak VM Opt [<option>...] -- [<argument>...]
>
> Common <option>s:
>  -help                 print this help message, then exit
>  -memory <size>[mk]    use fixed heap size (added to image size)
>  -pathenc <enc>        set encoding for pathnames (default: macintosh)
>  -headless             run in headless (no window) mode (default: false)
>
> Notes:
>  <imageName> defaults to `Squeak.image'.
>  If `-memory' is not specified then the heap will grow dynamically.
>  <argument>s are ignored, but are processed by the Squeak image.
>  The first <argument> normally names a Squeak `script' to execute.
>  Precede <arguments> by `--' to use default image.

This is ok for Mac, but the unix VM takes different arguments.


Levente

>
>
> HTH,
> Adrian
>
> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>
>> Hi
>>
>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>
>> Thanks for any ideas.
>>
>> Cheers,
>> Max
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Max Leske
I'm actually working on a mac, so that shouldn't be the problem.
So your proposition to use -mmap would be unix specific? Is the option available on mac?

Thanks,
Max


On 09.10.2010, at 21:34, Levente Uzonyi wrote:

> On Sat, 9 Oct 2010, Adrian Lienhard wrote:
>
>> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>>
>> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
>> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>>      ./Squeak VM Opt [<option>...] -- [<argument>...]
>>
>> Common <option>s:
>> -help                 print this help message, then exit
>> -memory <size>[mk]    use fixed heap size (added to image size)
>> -pathenc <enc>        set encoding for pathnames (default: macintosh)
>> -headless             run in headless (no window) mode (default: false)
>>
>> Notes:
>> <imageName> defaults to `Squeak.image'.
>> If `-memory' is not specified then the heap will grow dynamically.
>> <argument>s are ignored, but are processed by the Squeak image.
>> The first <argument> normally names a Squeak `script' to execute.
>> Precede <arguments> by `--' to use default image.
>
> This is ok for Mac, but the unix VM takes different arguments.
>
>
> Levente
>
>>
>>
>> HTH,
>> Adrian
>>
>> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>>
>>> Hi
>>>
>>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>>
>>> Thanks for any ideas.
>>>
>>> Cheers,
>>> Max
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Schwab,Wilhelm K
In reply to this post by Max Leske
Max,

You are probably way ahead of this (and keep pushing the envelope<g>), but you do know about #release, right?  A couple of weeks ago I ran some fairly intense calculations that resulted in low space conditions.  Sending #release to the participants (and implementing it correctly) fixed it.

One thing that IMHO has never worked well, or at least not as well as advertised, in Dolphin is relying on its finalization "water mark" to trigger cleanup of finalizable objects on need.  Most notably, I found it necessary to explicitly clean up ODBC statements to avoid errors/crashes.

How much memory do you need?  May I ask for what?  32 or 64 bit?

Bill



________________________________________
From: [hidden email] [[hidden email]] On Behalf Of Max Leske [[hidden email]]
Sent: Saturday, October 09, 2010 12:01 PM
To: [hidden email]
Subject: [Pharo-project] Getting my head around low space

Hi

I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.

Thanks for any ideas.

Cheers,
Max
_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Mariano Martinez Peck
In reply to this post by Max Leske


On Sat, Oct 9, 2010 at 9:58 PM, Max Leske <[hidden email]> wrote:
I'm actually working on a mac, so that shouldn't be the problem.
So your proposition to use -mmap would be unix specific? Is the option available on mac?

I think you can also modify the file Info.plist. Take the squeak app, right click, show package contents. There go to "Contents" and there you will find such file. There, you can change the property SqueakMaxHeapSize. Put 2 gb or whatever you want.

Cheers

Mariano




 

Thanks,
Max


On 09.10.2010, at 21:34, Levente Uzonyi wrote:

> On Sat, 9 Oct 2010, Adrian Lienhard wrote:
>
>> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>>
>> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
>> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>>      ./Squeak VM Opt [<option>...] -- [<argument>...]
>>
>> Common <option>s:
>> -help                 print this help message, then exit
>> -memory <size>[mk]    use fixed heap size (added to image size)
>> -pathenc <enc>        set encoding for pathnames (default: macintosh)
>> -headless             run in headless (no window) mode (default: false)
>>
>> Notes:
>> <imageName> defaults to `Squeak.image'.
>> If `-memory' is not specified then the heap will grow dynamically.
>> <argument>s are ignored, but are processed by the Squeak image.
>> The first <argument> normally names a Squeak `script' to execute.
>> Precede <arguments> by `--' to use default image.
>
> This is ok for Mac, but the unix VM takes different arguments.
>
>
> Levente
>
>>
>>
>> HTH,
>> Adrian
>>
>> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>>
>>> Hi
>>>
>>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>>
>>> Thanks for any ideas.
>>>
>>> Cheers,
>>> Max
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Levente Uzonyi-2
In reply to this post by Max Leske
On Sat, 9 Oct 2010, Max Leske wrote:

> I'm actually working on a mac, so that shouldn't be the problem.
> So your proposition to use -mmap would be unix specific? Is the option available on mac?

I don't have a Mac, but you can easily check it by starting an image with
"-mmap 64m", then allocating a 64MB array in the image. If it fails, then
the switch works, otherwise not.


Levente

>
> Thanks,
> Max
>
>
> On 09.10.2010, at 21:34, Levente Uzonyi wrote:
>
>> On Sat, 9 Oct 2010, Adrian Lienhard wrote:
>>
>>> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>>>
>>> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
>>> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>>>      ./Squeak VM Opt [<option>...] -- [<argument>...]
>>>
>>> Common <option>s:
>>> -help                 print this help message, then exit
>>> -memory <size>[mk]    use fixed heap size (added to image size)
>>> -pathenc <enc>        set encoding for pathnames (default: macintosh)
>>> -headless             run in headless (no window) mode (default: false)
>>>
>>> Notes:
>>> <imageName> defaults to `Squeak.image'.
>>> If `-memory' is not specified then the heap will grow dynamically.
>>> <argument>s are ignored, but are processed by the Squeak image.
>>> The first <argument> normally names a Squeak `script' to execute.
>>> Precede <arguments> by `--' to use default image.
>>
>> This is ok for Mac, but the unix VM takes different arguments.
>>
>>
>> Levente
>>
>>>
>>>
>>> HTH,
>>> Adrian
>>>
>>> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>>>
>>>> Hi
>>>>
>>>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>>>
>>>> Thanks for any ideas.
>>>>
>>>> Cheers,
>>>> Max
>>>> _______________________________________________
>>>> Pharo-project mailing list
>>>> [hidden email]
>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>

_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

johnmci
It's fairly clear:  " the Mac VM has an option -memory"
no -mmap

use it or set the info.plist setting of  SqueakMaxHeapSize.
You can in theory set values up to 4GB on a 4.2.x VM or 5.7.x
Might make lights dim and core dump on a 5.8.x VM, please try...
Obviously you have to actually use the 4GB of memory.

However I suspect asking for anything close to or over 5.8GB on a 4.2.x and 5.8.x VM won't fly.
Should work with a 5.7.x VM


On 2010-10-09, at 8:14 PM, Levente Uzonyi wrote:

> On Sat, 9 Oct 2010, Max Leske wrote:
>
>> I'm actually working on a mac, so that shouldn't be the problem.
>> So your proposition to use -mmap would be unix specific? Is the option available on mac?
>
> I don't have a Mac, but you can easily check it by starting an image with "-mmap 64m", then allocating a 64MB array in the image. If it fails, then the switch works, otherwise not.
>
>
> Levente
>
>>
>> Thanks,
>> Max
>>
>>
>> On 09.10.2010, at 21:34, Levente Uzonyi wrote:
>>
>>> On Sat, 9 Oct 2010, Adrian Lienhard wrote:
>>>
>>>> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>>>>
>>>> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
>>>> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>>>>     ./Squeak VM Opt [<option>...] -- [<argument>...]
>>>>
>>>> Common <option>s:
>>>> -help                 print this help message, then exit
>>>> -memory <size>[mk]    use fixed heap size (added to image size)
>>>> -pathenc <enc>        set encoding for pathnames (default: macintosh)
>>>> -headless             run in headless (no window) mode (default: false)
>>>>
>>>> Notes:
>>>> <imageName> defaults to `Squeak.image'.
>>>> If `-memory' is not specified then the heap will grow dynamically.
>>>> <argument>s are ignored, but are processed by the Squeak image.
>>>> The first <argument> normally names a Squeak `script' to execute.
>>>> Precede <arguments> by `--' to use default image.
>>>
>>> This is ok for Mac, but the unix VM takes different arguments.
>>>
>>>
>>> Levente
>>>
>>>>
>>>>
>>>> HTH,
>>>> Adrian
>>>>
>>>> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>>>>
>>>>> Thanks for any ideas.
>>>>>
>>>>> Cheers,
>>>>> Max
>>>>> _______________________________________________
>>>>> Pharo-project mailing list
>>>>> [hidden email]
>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>
>>>>
>>>> _______________________________________________
>>>> Pharo-project mailing list
>>>> [hidden email]
>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================





_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Max Leske
In reply to this post by Schwab,Wilhelm K
Hey Bill

No, I hadn't thought about that. But I guess It wouldn't help me much anyway since what effectively eats memory is the data I store in the image; I reuse my objects as much as possible. Basically, I'm storing thousands of very short strings in ByteArrays (files in a memory filesystem, using Colin Putney's Filesystem framework). Writing the same data to disk gives me around 500 MB because of the massive overhead per file (there's around 50 bytes per file and each file allocates 4 KB). I guess it would be less in the image but let's just say I'll need around 1 GB probably.

Max


On 09.10.2010, at 23:14, Schwab,Wilhelm K wrote:

> Max,
>
> You are probably way ahead of this (and keep pushing the envelope<g>), but you do know about #release, right?  A couple of weeks ago I ran some fairly intense calculations that resulted in low space conditions.  Sending #release to the participants (and implementing it correctly) fixed it.
>
> One thing that IMHO has never worked well, or at least not as well as advertised, in Dolphin is relying on its finalization "water mark" to trigger cleanup of finalizable objects on need.  Most notably, I found it necessary to explicitly clean up ODBC statements to avoid errors/crashes.
>
> How much memory do you need?  May I ask for what?  32 or 64 bit?
>
> Bill
>
>
>
> ________________________________________
> From: [hidden email] [[hidden email]] On Behalf Of Max Leske [[hidden email]]
> Sent: Saturday, October 09, 2010 12:01 PM
> To: [hidden email]
> Subject: [Pharo-project] Getting my head around low space
>
> Hi
>
> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>
> Thanks for any ideas.
>
> Cheers,
> Max
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Max Leske
In reply to this post by johnmci
Thanks everyone for the suggestions! I'll give the -memory option a try, using 1 GB. I'll report back.

Max


On 10.10.2010, at 06:51, John M McIntosh wrote:

> It's fairly clear:  " the Mac VM has an option -memory"
> no -mmap
>
> use it or set the info.plist setting of  SqueakMaxHeapSize.
> You can in theory set values up to 4GB on a 4.2.x VM or 5.7.x
> Might make lights dim and core dump on a 5.8.x VM, please try...
> Obviously you have to actually use the 4GB of memory.
>
> However I suspect asking for anything close to or over 5.8GB on a 4.2.x and 5.8.x VM won't fly.
> Should work with a 5.7.x VM
>
>
> On 2010-10-09, at 8:14 PM, Levente Uzonyi wrote:
>
>> On Sat, 9 Oct 2010, Max Leske wrote:
>>
>>> I'm actually working on a mac, so that shouldn't be the problem.
>>> So your proposition to use -mmap would be unix specific? Is the option available on mac?
>>
>> I don't have a Mac, but you can easily check it by starting an image with "-mmap 64m", then allocating a 64MB array in the image. If it fails, then the switch works, otherwise not.
>>
>>
>> Levente
>>
>>>
>>> Thanks,
>>> Max
>>>
>>>
>>> On 09.10.2010, at 21:34, Levente Uzonyi wrote:
>>>
>>>> On Sat, 9 Oct 2010, Adrian Lienhard wrote:
>>>>
>>>>> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>>>>>
>>>>> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
>>>>> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>>>>>    ./Squeak VM Opt [<option>...] -- [<argument>...]
>>>>>
>>>>> Common <option>s:
>>>>> -help                 print this help message, then exit
>>>>> -memory <size>[mk]    use fixed heap size (added to image size)
>>>>> -pathenc <enc>        set encoding for pathnames (default: macintosh)
>>>>> -headless             run in headless (no window) mode (default: false)
>>>>>
>>>>> Notes:
>>>>> <imageName> defaults to `Squeak.image'.
>>>>> If `-memory' is not specified then the heap will grow dynamically.
>>>>> <argument>s are ignored, but are processed by the Squeak image.
>>>>> The first <argument> normally names a Squeak `script' to execute.
>>>>> Precede <arguments> by `--' to use default image.
>>>>
>>>> This is ok for Mac, but the unix VM takes different arguments.
>>>>
>>>>
>>>> Levente
>>>>
>>>>>
>>>>>
>>>>> HTH,
>>>>> Adrian
>>>>>
>>>>> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>>>>>
>>>>>> Thanks for any ideas.
>>>>>>
>>>>>> Cheers,
>>>>>> Max
>>>>>> _______________________________________________
>>>>>> Pharo-project mailing list
>>>>>> [hidden email]
>>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pharo-project mailing list
>>>>> [hidden email]
>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pharo-project mailing list
>>>> [hidden email]
>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Max Leske
In reply to this post by johnmci
I had a first failure with 1 GB. I used the 4.2.5b1U VM. Setting the memory to anything above 1 GB would lead to an error message. I'm now trying the 5.7b1 VM with 3.5 GB.

Max


On 10.10.2010, at 06:51, John M McIntosh wrote:

> It's fairly clear:  " the Mac VM has an option -memory"
> no -mmap
>
> use it or set the info.plist setting of  SqueakMaxHeapSize.
> You can in theory set values up to 4GB on a 4.2.x VM or 5.7.x
> Might make lights dim and core dump on a 5.8.x VM, please try...
> Obviously you have to actually use the 4GB of memory.
>
> However I suspect asking for anything close to or over 5.8GB on a 4.2.x and 5.8.x VM won't fly.
> Should work with a 5.7.x VM
>
>
> On 2010-10-09, at 8:14 PM, Levente Uzonyi wrote:
>
>> On Sat, 9 Oct 2010, Max Leske wrote:
>>
>>> I'm actually working on a mac, so that shouldn't be the problem.
>>> So your proposition to use -mmap would be unix specific? Is the option available on mac?
>>
>> I don't have a Mac, but you can easily check it by starting an image with "-mmap 64m", then allocating a 64MB array in the image. If it fails, then the switch works, otherwise not.
>>
>>
>> Levente
>>
>>>
>>> Thanks,
>>> Max
>>>
>>>
>>> On 09.10.2010, at 21:34, Levente Uzonyi wrote:
>>>
>>>> On Sat, 9 Oct 2010, Adrian Lienhard wrote:
>>>>
>>>>> The unix VM and apparently also the Mac VM has an option -memory that allows you to set a heap size:
>>>>>
>>>>> [adrian:/Applications/Squeak 4.2.3beta1U.app/Contents/MacOS] ./Squeak\ VM\ Opt -help
>>>>> Usage: ./Squeak VM Opt [<option>...] [<imageName> [<argument>...]]
>>>>>   ./Squeak VM Opt [<option>...] -- [<argument>...]
>>>>>
>>>>> Common <option>s:
>>>>> -help                 print this help message, then exit
>>>>> -memory <size>[mk]    use fixed heap size (added to image size)
>>>>> -pathenc <enc>        set encoding for pathnames (default: macintosh)
>>>>> -headless             run in headless (no window) mode (default: false)
>>>>>
>>>>> Notes:
>>>>> <imageName> defaults to `Squeak.image'.
>>>>> If `-memory' is not specified then the heap will grow dynamically.
>>>>> <argument>s are ignored, but are processed by the Squeak image.
>>>>> The first <argument> normally names a Squeak `script' to execute.
>>>>> Precede <arguments> by `--' to use default image.
>>>>
>>>> This is ok for Mac, but the unix VM takes different arguments.
>>>>
>>>>
>>>> Levente
>>>>
>>>>>
>>>>>
>>>>> HTH,
>>>>> Adrian
>>>>>
>>>>> On Oct 9, 2010, at 18:01 , Max Leske wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I'm having trouble with low space warnings in Pharo. Is there a source on how much memory the image can get from the OS? Is there maybe a way to up that limit? I am loading a lot of data into the image and the limit at the moment seems to be around 500 MB.
>>>>>>
>>>>>> Thanks for any ideas.
>>>>>>
>>>>>> Cheers,
>>>>>> Max
>>>>>> _______________________________________________
>>>>>> Pharo-project mailing list
>>>>>> [hidden email]
>>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Pharo-project mailing list
>>>>> [hidden email]
>>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>>>
>>>>
>>>> _______________________________________________
>>>> Pharo-project mailing list
>>>> [hidden email]
>>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

johnmci

On 2010-10-10, at 2:00 PM, Max Leske wrote:

> I had a first failure with 1 GB. I used the 4.2.5b1U VM. Setting the memory to anything above 1 GB would lead to an error message. I'm now trying the 5.7b1 VM with 3.5 GB.
>
> Max

So what is the error message?


--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================





_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project

smime.p7s (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space

Max Leske
"There is not enough memory to give squeak the amount specified by the 'memory EMBED tag.". Additionally there's a "Premature end of file" error on the image and then the VM crashes (OS X crash dialog). Issuing the same commands with 1 GB memory works fine (no problems with the image).

Interestingly I tried to run the task with 3.5 GB in a 5.7b1 VM over night and the low space warning popped up when the progress bar was have as far then when I ran into the same condition using the 4.2.5b1U VM and 1 GB... Any suggestions? I couldn't attach the VM statistics because of the lacking menu integration but if you tell me how to extract the statistics (supposing they are interesting...) I'll happily rerun the task and post the results.

Max


On 10.10.2010, at 23:47, John M McIntosh wrote:

>
> On 2010-10-10, at 2:00 PM, Max Leske wrote:
>
>> I had a first failure with 1 GB. I used the 4.2.5b1U VM. Setting the memory to anything above 1 GB would lead to an error message. I'm now trying the 5.7b1 VM with 3.5 GB.
>>
>> Max
>
> So what is the error message?
>
>
> --
> ===========================================================================
> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ===========================================================================
>
>
>
>
> _______________________________________________
> Pharo-project mailing list
> [hidden email]
> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project


_______________________________________________
Pharo-project mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space [closed]

Max Leske
Hey folks

As was to be expected, my memory problem resulted from a problem in my code. So, just wanted to say thanks for the help and sorry for the hubbub.
What might be of interest: I use the CogVM now for the task and I'm really excited how fast it is! I started the same calculations first in the normal VM (4.2.5) and about 5 minutes later in the CogVM. After 30 minutes or so, the CogVM had overtaken the 4.2.5 VM (judging by the progress bar at least).


Cheers,
Max


On 11.10.2010, at 09:33, Max Leske wrote:

> "There is not enough memory to give squeak the amount specified by the 'memory EMBED tag.". Additionally there's a "Premature end of file" error on the image and then the VM crashes (OS X crash dialog). Issuing the same commands with 1 GB memory works fine (no problems with the image).
>
> Interestingly I tried to run the task with 3.5 GB in a 5.7b1 VM over night and the low space warning popped up when the progress bar was have as far then when I ran into the same condition using the 4.2.5b1U VM and 1 GB... Any suggestions? I couldn't attach the VM statistics because of the lacking menu integration but if you tell me how to extract the statistics (supposing they are interesting...) I'll happily rerun the task and post the results.
>
> Max
>
>
> On 10.10.2010, at 23:47, John M McIntosh wrote:
>
>>
>> On 2010-10-10, at 2:00 PM, Max Leske wrote:
>>
>>> I had a first failure with 1 GB. I used the 4.2.5b1U VM. Setting the memory to anything above 1 GB would lead to an error message. I'm now trying the 5.7b1 VM with 3.5 GB.
>>>
>>> Max
>>
>> So what is the error message?
>>
>>
>> --
>> ===========================================================================
>> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
>> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
>> ===========================================================================
>>
>>
>>
>>
>> _______________________________________________
>> Pharo-project mailing list
>> [hidden email]
>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>


Reply | Threaded
Open this post in threaded view
|

Re: Getting my head around low space [closed]

Stéphane Ducasse
In reply to this post by Max Leske
thanks for sharing that with us.

Stef
On Nov 4, 2010, at 11:27 PM, Max Leske wrote:

> Hey folks
>
> As was to be expected, my memory problem resulted from a problem in my code. So, just wanted to say thanks for the help and sorry for the hubbub.
> What might be of interest: I use the CogVM now for the task and I'm really excited how fast it is! I started the same calculations first in the normal VM (4.2.5) and about 5 minutes later in the CogVM. After 30 minutes or so, the CogVM had overtaken the 4.2.5 VM (judging by the progress bar at least).
>
>
> Cheers,
> Max
>
>
> On 11.10.2010, at 09:33, Max Leske wrote:
>
>> "There is not enough memory to give squeak the amount specified by the 'memory EMBED tag.". Additionally there's a "Premature end of file" error on the image and then the VM crashes (OS X crash dialog). Issuing the same commands with 1 GB memory works fine (no problems with the image).
>>
>> Interestingly I tried to run the task with 3.5 GB in a 5.7b1 VM over night and the low space warning popped up when the progress bar was have as far then when I ran into the same condition using the 4.2.5b1U VM and 1 GB... Any suggestions? I couldn't attach the VM statistics because of the lacking menu integration but if you tell me how to extract the statistics (supposing they are interesting...) I'll happily rerun the task and post the results.
>>
>> Max
>>
>>
>> On 10.10.2010, at 23:47, John M McIntosh wrote:
>>
>>>
>>> On 2010-10-10, at 2:00 PM, Max Leske wrote:
>>>
>>>> I had a first failure with 1 GB. I used the 4.2.5b1U VM. Setting the memory to anything above 1 GB would lead to an error message. I'm now trying the 5.7b1 VM with 3.5 GB.
>>>>
>>>> Max
>>>
>>> So what is the error message?
>>>
>>>
>>> --
>>> ===========================================================================
>>> John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
>>> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
>>> ===========================================================================
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Pharo-project mailing list
>>> [hidden email]
>>> http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-project
>>
>
>