Breaking the 4GB barrier with Pharo 6 64-bit

OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/
Image here: http://files.pharo.org/get-files/60/pharo-64.zip

[2]

| meta |
ASTCache reset.
meta := Dictionary new.
Smalltalk allClassesAndTraits do: [ :each | | classMeta methods |
  (classMeta := Dictionary new)
    at: #name put: each name asSymbol;
    at: #comment put: each comment;
    at: #definition put: each definition;
    at: #object put: each.
  methods := Dictionary new.
  classMeta at: #methods put: methods.
  each methodsDo: [ :method | | methodMeta |
    (methodMeta := Dictionary new)
      at: #name put: method selector;
      at: #source put: method sourceCode;
      at: #ast put: method ast;

      at: #args put: method argumentNames asArray;
      at: #formatted put: method ast formattedCode;
      at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]);
      at: #object put: method.

    methods at: method selector put: methodMeta ].
  meta at: each name asSymbol put: classMeta ].
meta.

--
Sven Van Caekenberghe
Proudly supporting Pharo
http://pharo.org
http://association.pharo.org
http://consortium.pharo.org

2016-11-09 15:23 GMT+01:00 [hidden email] <[hidden email]>:

Anyone having tested such 64 bit goodness on a Linux box?

Only as a professional demo for code developped up to the very last moment on Pharo 32 bits. So I didn't try to explose the available RAM on my laptop (yet).

Thierry

 

Phil

OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).

Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.

I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).

I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).

Great work, thank you. Bright future again.

Sven

PS: Yes, GC is slower; No, I did not yet try to save such a large image.

[1]

VM here: http://bintray.com/estebanlm/pharo-vm/build#files/
Image here: http://files.pharo.org/get-files/60/pharo-64.zip

[2]

| meta |
ASTCache reset.
meta := Dictionary new.
Smalltalk allClassesAndTraits do: [ :each | | classMeta methods |
  (classMeta := Dictionary new)
    at: #name put: each name asSymbol;
    at: #comment put: each comment;
    at: #definition put: each definition;
    at: #object put: each.
  methods := Dictionary new.
  classMeta at: #methods put: methods.
  each methodsDo: [ :method | | methodMeta |
    (methodMeta := Dictionary new)
      at: #name put: method selector;
      at: #source put: method sourceCode;
      at: #ast put: method ast;

      at: #args put: method argumentNames asArray;
      at: #formatted put: method ast formattedCode;
      at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]);
      at: #object put: method.

    methods at: method selector put: methodMeta ].
  meta at: each name asSymbol put: classMeta ].
meta.

--
Sven Van Caekenberghe
Proudly supporting Pharo
http://pharo.org
http://association.pharo.org
http://consortium.pharo.org

Hi Igor,

Please refrain from speaking down on people.

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

Cheers,
Tudor


> On Nov 10, 2016, at 4:11 AM, Igor Stasenko <[hidden email]> wrote:
>
> Nice progress, indeed.
> Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able
> to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally.
> But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)
>
> On 9 November 2016 at 12:06, Sven Van Caekenberghe <[hidden email]> wrote:
> OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).
>
> Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.
>
> I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).
>
> <Screen Shot 2016-11-09 at 11.25.28.png>
>
> I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).
>
> Great work, thank you. Bright future again.
>
> Sven
>
> PS: Yes, GC is slower; No, I did not yet try to save such a large image.
>
> [1]
>
> VM here: http://bintray.com/estebanlm/pharo-vm/build#files/
> Image here: http://files.pharo.org/get-files/60/pharo-64.zip
>
> [2]
>
> | meta |
> ASTCache reset.
> meta := Dictionary new.
> Smalltalk allClassesAndTraits do: [ :each | | classMeta methods |
>   (classMeta := Dictionary new)
>     at: #name put: each name asSymbol;
>     at: #comment put: each comment;
>     at: #definition put: each definition;
>     at: #object put: each.
>   methods := Dictionary new.
>   classMeta at: #methods put: methods.
>   each methodsDo: [ :method | | methodMeta |
>     (methodMeta := Dictionary new)
>       at: #name put: method selector;
>       at: #source put: method sourceCode;
>       at: #ast put: method ast;
>       at: #args put: method argumentNames asArray;
>       at: #formatted put: method ast formattedCode;
>       at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]);
>       at: #object put: method.
>     methods at: method selector put: methodMeta ].
>   meta at: each name asSymbol put: classMeta ].
> meta.
>
>
>
> --
> Sven Van Caekenberghe
> Proudly supporting Pharo
> http://pharo.org
> http://association.pharo.org
> http://consortium.pharo.org
>
>
>
>
>
>
>
> --
> Best regards,
> Igor Stasenko.

--
www.tudorgirba.com
www.feenk.com

"We can create beautiful models in a vacuum.
But, to get them effective we have to deal with the inconvenience of reality."

On 10 November 2016 at 07:27, Tudor Girba <[hidden email]> wrote:

Hi Igor,

Please refrain from speaking down on people.

Hi, Doru!
I just wanted to hear you :)

 

If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it.

Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why
you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it
in image (in most optimal cases - only the portion that user sees/operates with).
As i said to you before, you will hit this wall inevitably, no matter how much memory is available.
So, what stops you from digging in that direction?
Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to
100 MB or less.
I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks.
So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about 

doing what they need. We have Fuel for how many years now? 

So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again.
Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution.

 

Cheers,
Tudor


> On Nov 10, 2016, at 4:11 AM, Igor Stasenko <[hidden email]> wrote:
>
> Nice progress, indeed.
> Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able
> to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally.
> But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :)
>
> On 9 November 2016 at 12:06, Sven Van Caekenberghe <[hidden email]> wrote:
> OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too).
>
> Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph.
>
> I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory).
>
> <Screen Shot 2016-11-09 at 11.25.28.png>
>

> I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case).
>
> Great work, thank you. Bright future again.
>
> Sven
>
> PS: Yes, GC is slower; No, I did not yet try to save such a large image.
>
> [1]
>
> VM here: http://bintray.com/estebanlm/pharo-vm/build#files/
> Image here: http://files.pharo.org/get-files/60/pharo-64.zip
>
> [2]
>
> | meta |
> ASTCache reset.
> meta := Dictionary new.
> Smalltalk allClassesAndTraits do: [ :each | | classMeta methods |
>   (classMeta := Dictionary new)
>     at: #name put: each name asSymbol;
>     at: #comment put: each comment;
>     at: #definition put: each definition;
>     at: #object put: each.
>   methods := Dictionary new.
>   classMeta at: #methods put: methods.
>   each methodsDo: [ :method | | methodMeta |
>     (methodMeta := Dictionary new)
>       at: #name put: method selector;
>       at: #source put: method sourceCode;
>       at: #ast put: method ast;
>       at: #args put: method argumentNames asArray;
>       at: #formatted put: method ast formattedCode;
>       at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]);
>       at: #object put: method.
>     methods at: method selector put: methodMeta ].
>   meta at: each name asSymbol put: classMeta ].
> meta.
>
>
>
> --
> Sven Van Caekenberghe
> Proudly supporting Pharo
> http://pharo.org
> http://association.pharo.org
> http://consortium.pharo.org
>
>
>
>
>
>
>
> --
> Best regards,
> Igor Stasenko.

--
www.tudorgirba.com
www.feenk.com

"We can create beautiful models in a vacuum.
But, to get them effective we have to deal with the inconvenience of reality."

--

Best regards,
Igor Stasenko.