VM and memory on win32

Hi, I was playing around with some rather large data sets and seemed to have
run into an issue with the 3.7.1 VM, which as far as I know is the latest
and greatest for windows.  It seems that the VM allocates memory
dynamically, ignores the memory: command line switch, and memory seems
capped somewhere around 900 meg.  Once I get up to about 900 or so, I get...

Space is low
Warning! Squeak is almost out of memory!

On several occasions, soon after this warning, the VM just died, costing me
some data.  This isn't a killer, I just moved my data into a database and
worked with it from there, but I was disappointed that I couldn't simply
work with the data in memory as pure objects.  It's just temporary data, not
intended for long term storage, so I figured I'd just play in a workspace
and toss it out when done.

My issues aside, I was wondering why Squeak can't use all available memory
when necessary, I have 2gigs, why should the VM be limited to 900 or so
megs?  I was on #squeak IRC channel discussing this and someone mentioned
the crashing could be a bug related to int's vs. unsigned int's somewhere in
the VM.  So if anyone has any answers to either issue, the memory limit, or
the fatal crashing, I'd love to hear about it.

Ramon Leon a écrit :
This may be cause beceause all avaliable pointer are use.
>From what I know each object are represente by a pointer instance of SmallInteger (except
SmallInteger instance who have no pointer, it is himself) (see the The missing chapters of the blue
Book ch. 26)
So the maximum memorie size is:

(SmallInteger maxVal) / (1024 * 1024)

Wich give you about 1000MB of memorie

But this is not official is just my insight.
If you want I have recently compile a 3.8 VM base on win32-3.8b4-source.

Math

On 14-Sep-06, at 3:09 PM, Mathieu wrote:
No, that's not it at all :-)

In the 'normal' 32bit machine VM object pointers are 32 bit values  
and can point to the full 4gb address range. Unfortunately older vms  
used signed numbers for address pointers and so any time object  
memory crossed the 2gb line there were cases where we would compare  
signed values with the top bit set and branch to Poughkeepise.

The newest VM code avoids this - at least I'm fairly sure it does.  
The very newest VM code can be compiled for 64bit capable machines  
and give you implausibly large potential memory spaces. More than  
most of us can actually fit into our machines, anyway.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- Forgot to pay his brain bill.

> If you want I have recently compile a 3.8 VM base on
> win32-3.8b4-source.
>
> Math

Would love it, just point me at it.

Ramon Leon wrote:
> Hi, I was playing around with some rather large data sets and seemed to have
> run into an issue with the 3.7.1 VM, which as far as I know is the latest
> and greatest for windows.  It seems that the VM allocates memory
> dynamically, ignores the memory: command line switch, and memory seems
> capped somewhere around 900 meg.  Once I get up to about 900 or so, I get...
>
> Space is low
> Warning! Squeak is almost out of memory!

Not surprising. Memory is capped at 1GB.

> My issues aside, I was wondering why Squeak can't use all available memory
> when necessary, I have 2gigs, why should the VM be limited to 900 or so
> megs?  I was on #squeak IRC channel discussing this and someone mentioned
> the crashing could be a bug related to int's vs. unsigned int's somewhere in
> the VM.  So if anyone has any answers to either issue, the memory limit, or
> the fatal crashing, I'd love to hear about it.

The answer to memory limit is simple: We can't use more than a gig just
because Windows won't let us. If you review the memory layout used by
Windows you'll find that only 2GB are available as user space to begin
with. Inside that address space we need to map both the executable,
various portions of memory allocated by the VM directly (buffers etc),
some Windows resources plus the actual Squeak object memory. And since
Squeak requires contiguous portions of memory the chances of being able
to allocate more than 1GB are very slim.

Cheers,
   - Andreas

On 14-Sep-06, at 3:36 PM, Andreas Raab wrote:

>
> The answer to memory limit is simple: We can't use more than a gig  
> just because Windows won't let us.

Hmph, yes, I'd completely missed the problem of OS's limiting the  
memory range usable.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Drugs may lead to nowhere, but at least it's the scenic route.

On Thu, 14 Sep 2006 15:36:47 -0700, Andreas Raab <[hidden email]>
wrote:

> The answer to memory limit is simple: We can't use more than a gig just
> because Windows won't let us. If you review the memory layout used by
> Windows you'll find that only 2GB are available as user space to begin
> with. Inside that address space we need to map both the executable,
> various portions of memory allocated by the VM directly (buffers etc),
> some Windows resources plus the actual Squeak object memory. And since
> Squeak requires contiguous portions of memory the chances of being able
> to allocate more than 1GB are very slim.

One of the neat things with VisualWorks is it does not have this
limitation. I've run VW images under Solaris that were up in the 1.7 GB
range, and from what I've heard those guys (where I used to work, and am
about to start working again) are now running 3.5 GB images.

Of course, VW doesn't require contiguous memory, since old space is
segmented...

Later,
Jon

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

Thanks, just what I wanted to know.

Hi Andreas,

On 9/14/06, Andreas Raab <[hidden email]> wrote:
> The answer to memory limit is simple: We can't use more than a gig just
> because Windows won't let us. If you review the memory layout used by
> Windows you'll find that only 2GB are available as user space to begin
> with.

You can bump that to 3GB:
http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

--
Dave Hylands
Vancouver, BC, Canada
http://www.DaveHylands.com/

Jon Hylands wrote:
> One of the neat things with VisualWorks is it does not have this
> limitation. I've run VW images under Solaris that were up in the 1.7 GB
> range, and from what I've heard those guys (where I used to work, and am
> about to start working again) are now running 3.5 GB images.

On Windows? Really? That would be news to me.

> Of course, VW doesn't require contiguous memory, since old space is
> segmented...

That's not the issue. The more relevant constraint is Windows memory
layout. The need for contiguous memory only means we need to reserve
memory early in the process (which we do anyway) so I could probably
bump it up to 1.5G or so without any ill side effects. I just choose 1GB
since it seemed like a nice round number.

Cheers,
   - Andreas

On Thu, 14 Sep 2006 16:48:46 -0700, Andreas Raab <[hidden email]>
wrote:

> Jon Hylands wrote:
> > One of the neat things with VisualWorks is it does not have this
> > limitation. I've run VW images under

> > Solaris

> > that were up in the 1.7 GB
> > range, and from what I've heard those guys (where I used to work, and am
> > about to start working again) are now running 3.5 GB images.
>
> On Windows? Really? That would be news to me.

--------------------------------------------------------------
   Jon Hylands      [hidden email]      http://www.huv.com/jon

  Project: Micro Seeker (Micro Autonomous Underwater Vehicle)
           http://www.huv.com

a) I'm not sure anyone has done a full audit to confirm if you run at  
say the 2GB to 3GB boundary that squeak will run and *all*  
functionality will work as expected, just because it runs doesn't  
mean it's not cheerfully rotting bits under you...

b) There *was* work which seems to have a lack of interest on the  
64bit version of squeak, again signed address restrict one to 63bits  
of address space, yes  yes the UI is all 32bit, but someday...

c) The mac carbon VM is set to 512MB in info.plist because when I set  
it to 1GB sometimes it would load at the 2GB boundary and then squeak  
in the past would *die* really fast.


On 14-Sep-06, at 4:48 PM, Jon Hylands wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

> On 9/14/06, Andreas Raab <[hidden email]> wrote:
> > The answer to memory limit is simple: We can't use more than a gig
> > just because Windows won't let us. If you review the memory layout
> > used by Windows you'll find that only 2GB are available as
> user space
> > to begin with.
>
> You can bump that to 3GB:
> http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx

Andreas, any chance of bumping this up?

Ramon Leon wrote:
>> You can bump that to 3GB:
>> http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
>
> Andreas, any chance of bumping this up?

Well, I can't do much about the above cited patches; it is solely your
responsibility to patch your local boot.ini ;-) The one thing I could do
though, is to re-interpret -memory as meaning "reserve that much address
space, no more no less" which would allow you to use more than the
currently hardcoded 1GB and also allow some low-memory condition testing
(i.e., if you want to find out whether a certain image can run within
32MB or so). If that would be helpful, please file a bug so that I don't
forget it the next time around.

Cheers,
   - Andreas