Smalltalk › Squeak › Squeak VM

New Cog VMs available

‹ Previous Topic Next Topic ›
 
classic Classic list List threaded Threaded
♦
♦
 7 messages Options
Eliot Miranda-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

New Cog VMs available

Eliot Miranda-2
 
... at http://www.mirandabanda.org/files/Cog/VM/VM.r3018/.

The main difference here is fixes to memory segment allocation on Spur.  Linux and Mac OS can now grow the heap to 2.9Gb or there abouts, and Win XP can grow to 1.8Gb.  I'm interested in reports on other Windows versions that don't have XP's 2Gb user process address space limit.

Here's my test script:
| them |
them := OrderedCollection new.
[[them addLast: (ByteArray new: 16000000).
 Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush] repeat]
on: OutOfMemory
do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk garbageCollect]].
Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush.
them := nil.
Smalltalk garbageCollect.
Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush

And the README is:
CogVM binaries as per VMMaker.oscog-eem.779/r3018.

Rewrite memory allocation on linux for Spur.  Arrange that the heap can grow
above 2Gb without any large initial alloc.

Rewrite platforms/unix/vm/sqUnixSpurMemory.c to stand alone.  Use it in place
of sqMacMemory.c with Spur on Mac OS.  Now Spur can grow the heap to 2.9Gb on
both linux (CentOS 5.3) and Mac OS X (10.6.8).

Rewrite Spur memory allocation on win32 similarly to unix.  Can now allocate
up to 1.8Gb on Windows XP (which has a 2Gb address space limit).  Add a flag
to indicate if the win32 exe is running as a console app and don't write to
the in-window console if so.

Fix some sign issues with free space tallying to allow Spur to shrink memory
and answer via primitiveVMParameter heap sizes above 2Gb.

Add longPrintInstancesOf:/longPrintInstancesWithClassIndex: for debugging.

Suggest reading a suitable README when the linux VM fails to spawn the heartbeat
thread.

Add parameter 54 on Spur to answer totalFreeOldSpace.

--
best,
Eliot
Paul DeBruicker
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: New Cog VMs available

Paul DeBruicker
Hi Eliot

On Windows 7 Ultimate 32bit, on a laptop with 3GB RAM, tested in normal & Safe Mode your test  maxed out around 1870MB using the Cog Spur v3018.


Hope this helps

Paul

here is the data from one of the runs:
62.2
78.2
94.2
110.2
126.2
142.2
158.2
174.2
190.2
206.2
222.2
238.2
254.2
270.2
286.2
302.2
318.2
334.2
350.2
366.2
382.2
398.2
414.2
430.2
446.2
462.2
478.2
494.2
510.2
526.2
542.2
558.2
574.2
590.2
606.2
622.2
638.2
654.2
670.2
686.2
702.2
718.2
734.2
750.2
766.2
782.2
798.2
814.2
830.2
846.2
862.2
878.2
894.2
910.2
926.2
942.2
958.2
974.2
990.2
1006.2
1022.2
1038.2
1054.2
1070.2
1086.2
1102.2
1118.2
1134.2
1150.2
1166.2
1182.2
1198.2
1214.2
1230.2
1246.2
1262.2
1278.2
1294.2
1310.2
1326.2
1342.2
1358.2
1374.2
1390.2
1406.2
1422.2
1438.2
1454.2
1470.2
1486.2
1502.2
1518.2
1534.2
1550.2
1566.2
1582.2
1598.2
1614.2
1630.2
1646.2
1662.2
1678.2
1694.2
1710.2
1726.2
1742.2
1758.2
1774.2
1790.2
1806.2
1822.2
1838.2
1854.2
1870.2
1886.2
974.2
62.2










Eliot Miranda-2 wrote
... at http://www.mirandabanda.org/files/Cog/VM/VM.r3018/.

The main difference here is fixes to memory segment allocation on Spur.
 Linux and Mac OS can now grow the heap to 2.9Gb or there abouts, and Win
XP can grow to 1.8Gb.  I'm interested in reports on other Windows versions
that don't have XP's 2Gb user process address space limit.

Here's my test script:
| them |
them := OrderedCollection new.
[[them addLast: (ByteArray new: 16000000).
 Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0)
maxDecimalPlaces: 1; flush] repeat]
on: OutOfMemory
do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk
garbageCollect]].
Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0)
maxDecimalPlaces: 1; flush.
them := nil.
Smalltalk garbageCollect.
Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0)
maxDecimalPlaces: 1; flush

And the README is:
CogVM binaries as per VMMaker.oscog-eem.779/r3018.

Rewrite memory allocation on linux for Spur.  Arrange that the heap can grow
above 2Gb without any large initial alloc.

Rewrite platforms/unix/vm/sqUnixSpurMemory.c to stand alone.  Use it in
place
of sqMacMemory.c with Spur on Mac OS.  Now Spur can grow the heap to 2.9Gb
on
both linux (CentOS 5.3) and Mac OS X (10.6.8).

Rewrite Spur memory allocation on win32 similarly to unix.  Can now allocate
up to 1.8Gb on Windows XP (which has a 2Gb address space limit).  Add a flag
to indicate if the win32 exe is running as a console app and don't write to
the in-window console if so.

Fix some sign issues with free space tallying to allow Spur to shrink memory
and answer via primitiveVMParameter heap sizes above 2Gb.

Add longPrintInstancesOf:/longPrintInstancesWithClassIndex: for debugging.

Suggest reading a suitable README when the linux VM fails to spawn the
heartbeat
thread.

Add parameter 54 on Spur to answer totalFreeOldSpace.

--
best,
Eliot
Ryan Macnak
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: New Cog VMs available

Ryan Macnak
In reply to this post by Eliot Miranda-2
 
With appropriate settings [1], one can get 3GB on Windows. (I haven't tested this on Cog but once upon time used it in another project.) I don't know if this extra GB is worth it to anyone; given the other platforms provide above 2GB hopefully the bugs related to signed versus unsigned comparisons are already flushed out. 

[1] http://msdn.microsoft.com/en-us/library/bb613473(v=VS.85).aspx


On Wed, Jun 25, 2014 at 4:13 PM, Eliot Miranda <[hidden email]> wrote:
 
... at http://www.mirandabanda.org/files/Cog/VM/VM.r3018/.

The main difference here is fixes to memory segment allocation on Spur.  Linux and Mac OS can now grow the heap to 2.9Gb or there abouts, and Win XP can grow to 1.8Gb.  I'm interested in reports on other Windows versions that don't have XP's 2Gb user process address space limit.

Here's my test script:
| them |
them := OrderedCollection new.
[[them addLast: (ByteArray new: 16000000).
 Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush] repeat]
on: OutOfMemory
do: [:ex| 2 to: them size by: 2 do: [:i| them at: i put: nil. Smalltalk garbageCollect]].
Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush.
them := nil.
Smalltalk garbageCollect.
Transcript cr; print: (Smalltalk vmParameterAt: 3) / (1024*1024.0) maxDecimalPlaces: 1; flush

And the README is:
CogVM binaries as per VMMaker.oscog-eem.779/r3018.

Rewrite memory allocation on linux for Spur.  Arrange that the heap can grow
above 2Gb without any large initial alloc.

Rewrite platforms/unix/vm/sqUnixSpurMemory.c to stand alone.  Use it in place
of sqMacMemory.c with Spur on Mac OS.  Now Spur can grow the heap to 2.9Gb on
both linux (CentOS 5.3) and Mac OS X (10.6.8).

Rewrite Spur memory allocation on win32 similarly to unix.  Can now allocate
up to 1.8Gb on Windows XP (which has a 2Gb address space limit).  Add a flag
to indicate if the win32 exe is running as a console app and don't write to
the in-window console if so.

Fix some sign issues with free space tallying to allow Spur to shrink memory
and answer via primitiveVMParameter heap sizes above 2Gb.

Add longPrintInstancesOf:/longPrintInstancesWithClassIndex: for debugging.

Suggest reading a suitable README when the linux VM fails to spawn the heartbeat
thread.

Add parameter 54 on Spur to answer totalFreeOldSpace.

--
best,
Eliot
Frank Shearar-3
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: New Cog VMs available

Frank Shearar-3
In reply to this post by Eliot Miranda-2

On 26 June 2014 00:13, Eliot Miranda <[hidden email]> wrote:
>
> ... at http://www.mirandabanda.org/files/Cog/VM/VM.r3018/.
>
> The main difference here is fixes to memory segment allocation on Spur.  Linux and Mac OS can now grow the heap to 2.9Gb or there abouts, and Win XP can grow to 1.8Gb.  I'm interested in reports on other Windows versions that don't have XP's 2Gb user process address space limit.

Cool! CI's duly updated, and the latest build
(http://build.squeak.org/job/SqueakTrunkOnSpur/16/) shows great
improvement. It completes the entire test suite without crashing. I do
see this:

sqAllocateMemorySegmentOfSizeAboveAllocatedSizeInto mmap: Cannot allocate memory
sqAllocateMemorySegmentOfSizeAboveAllocatedSizeInto mmap: Cannot allocate memory
sqAllocateMemorySegmentOfSizeAboveAllocatedSizeInto mmap: Cannot allocate memory
sqAllocateMemorySegmentOfSizeAboveAllocatedSizeInto mmap: Cannot allocate memory
sqAllocateMemorySegmentOfSizeAboveAllocatedSizeInto mmap: Cannot allocate memory
sqAllocateMemorySegmentOfSizeAboveAllocatedSizeInto mmap: Cannot allocate memory

and there are 45 failing tests, but that's significant improvement!

frank
Henrik Sperre Johansen
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: New Cog VMs available

Henrik Sperre Johansen
In reply to this post by Ryan Macnak
 

On 26 Jun 2014, at 8:05 , Ryan Macnak <[hidden email]> wrote:

With appropriate settings [1], one can get 3GB on Windows. (I haven't tested this on Cog but once upon time used it in another project.) I don't know if this extra GB is worth it to anyone; given the other platforms provide above 2GB hopefully the bugs related to signed versus unsigned comparisons are already flushed out. 


For that, you’d also need a binary with the Large Address Aware bit set in the COFF-header, which the released binary does not have. 
It’s basically just a checkbox that yes, I assure you my application only treats addresses as unsigned, so please hand me more than 2GB of memory.
Which is also the way to get >2GB with 32bit executables on 64-bit Windows as well, so it might be worthwhile finding the proper compiler flag.

Cheers,
Henry

PS. Here’s the transcript from Windows 7 x64 on a modified version of released binary with the only change being LAA bit set to on manually:

60.7
60.7
76.7
92.7
108.7
124.7
140.7
156.7
172.7
188.7
204.7
220.7
236.7
252.7
268.7
284.7
300.7
316.7
332.7
348.7
364.7
380.7
396.7
412.7
428.7
444.7
460.7
476.7
492.7
508.7
524.7
540.7
556.7
572.7
588.7
604.7
620.7
636.7
652.7
668.7
684.7
700.7
716.7
732.7
748.7
764.7
780.7
796.7
812.7
828.7
844.7
860.7
876.7
892.7
908.7
924.7
940.7
956.7
972.7
988.7
1004.7
1020.7
1036.7
1052.7
1068.7
1084.7
1100.7
1116.7
1132.7
1148.7
1164.7
1180.7
1196.7
1212.7
1228.7
1244.7
1260.7
1276.7
1292.7
1308.7
1324.7
1340.7
1356.7
1372.7
1388.7
1404.7
1420.7
1436.7
1452.7
1468.7
1484.7
1500.7
1516.7
1532.7
1548.7
1564.7
1580.7
1596.7
1612.7
1628.7
1644.7
1660.7
1676.7
1692.7
1708.7
1724.7
1740.7
1756.7
1772.7
1788.7
1804.7
1820.7
1836.7
1852.7
1868.7
1884.7
1900.7
1916.7
1932.7
1948.7
1964.7
1980.7
1996.7
2012.7
2028.7
2044.7
2060.7
2076.7
2092.7
2108.7
2124.7
2140.7
2156.7
2172.7
2188.7
2204.7
2220.7
2236.7
2252.7
2268.7
2284.7
2300.7
2316.7
2332.7
2348.7
2364.7
2380.7
2396.7
2412.7
2428.7
2444.7
2460.7
2476.7
2492.7
2508.7
2524.7
2540.7
2556.7
2572.7
2588.7
2604.7
2620.7
2636.7
2652.7
2668.7
2684.7
2700.7
2716.7
2732.7
2748.7
2764.7
2780.7
2796.7
2812.7
2828.7
2844.7
2860.7
2876.7
2892.7
2908.7
2924.7
2940.7
2956.7
2972.7
2988.7
3004.7
3020.7
3036.7
3052.7
3068.7
3084.7
3100.7
3116.7
3132.7
3148.7
3164.7
3180.7
3196.7
3212.7
3228.7
3244.7
3260.7
3276.7
3292.7
3308.7
3324.7
3340.7
3356.7
3372.7
3388.7
3404.7
3420.7
3436.7
3452.7
3468.7
3484.7
3500.7
3516.7
3532.7
3548.7
3564.7
3580.7
3596.7
3612.7
3628.7
3644.7
3660.7
3676.7
3692.7
3708.7
3724.7
3740.7
3756.7
3772.7
3788.7
3804.7
3820.7
3836.7
3852.7
1948.7
44.7

signature.asc (859 bytes) Download Attachment
Frank Shearar-3
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: New Cog VMs available

Frank Shearar-3
In reply to this post by Eliot Miranda-2

On 26 June 2014 00:13, Eliot Miranda <[hidden email]> wrote:
>
> ... at http://www.mirandabanda.org/files/Cog/VM/VM.r3018/.
>
> The main difference here is fixes to memory segment allocation on Spur.  Linux and Mac OS can now grow the heap to 2.9Gb or there abouts, and Win XP can grow to 1.8Gb.  I'm interested in reports on other Windows versions that don't have XP's 2Gb user process address space limit.

Windows 8.1:

44.7
60.7
76.7
92.7
108.7
124.7
140.7
156.7
172.7
188.7
204.7
220.7
236.7
252.7
268.7
284.7
300.7
316.7
332.7
348.7
364.7
380.7
396.7
412.7
428.7
444.7
460.7
476.7
492.7
508.7
524.7
540.7
556.7
572.7
588.7
604.7
620.7
636.7
652.7
668.7
684.7
700.7
716.7
732.7
748.7
764.7
780.7
796.7
812.7
828.7
844.7
860.7
876.7
892.7
908.7
924.7
940.7
956.7
972.7
988.7
1004.7
1020.7
1036.7
1052.7
1068.7
1084.7
1100.7
1116.7
1132.7
1148.7
1164.7
1180.7
1196.7
1212.7
1228.7
1244.7
1260.7
1276.7
1292.7
1308.7
1324.7
1340.7
1356.7
1372.7
1388.7
1404.7
1420.7
1436.7
1452.7
1468.7
1484.7
1500.7
1516.7
1532.7
1548.7
1564.7
1580.7
1596.7
1612.7
1628.7
1644.7
1660.7
1676.7
1692.7
1708.7
1724.7
1740.7
1756.7
1772.7
1788.7
908.7
44.7

frank
Eliot Miranda-2
Reply | Threaded
Open this post in threaded view
♦
♦
|

Re: New Cog VMs available

Eliot Miranda-2
In reply to this post by Henrik Sperre Johansen
 
Thanks Ryan, Thanks Henrik,


On Thu, Jun 26, 2014 at 1:17 AM, Henrik Johansen <[hidden email]> wrote:
 

On 26 Jun 2014, at 8:05 , Ryan Macnak <[hidden email]> wrote:

With appropriate settings [1], one can get 3GB on Windows. (I haven't tested this on Cog but once upon time used it in another project.) I don't know if this extra GB is worth it to anyone; given the other platforms provide above 2GB hopefully the bugs related to signed versus unsigned comparisons are already flushed out. 


For that, you’d also need a binary with the Large Address Aware bit set in the COFF-header, which the released binary does not have. 
It’s basically just a checkbox that yes, I assure you my application only treats addresses as unsigned, so please hand me more than 2GB of memory.
Which is also the way to get >2GB with 32bit executables on 64-bit Windows as well, so it might be worthwhile finding the proper compiler flag.

I found this:

"It depends on your toolchain.

If you are using cygwin, i don't know.
But for Mingw, a quick google for "LARGEADDRESSAWARE gnu ld"
turns up "--large-address-aware". You have to pass it down to ld through gcc by
'-Wl,--large-address-aware'.
That is if you are using mingw with gnu ld.
But beware, this is still 32-Bit, so somewhere below 4GB (3GB?) is you max..
Win64 support is still unfinished AFAIK (or in the latest and greatest CVS
snapshots + still itchy).

> Thank you in advance.

Greetings

        Jan" 

Cheers,
Henry

PS. Here’s the transcript from Windows 7 x64 on a modified version of released binary with the only change being LAA bit set to on manually:

60.7
60.7
76.7
92.7
108.7
124.7
140.7
156.7
172.7
188.7
204.7
220.7
236.7
252.7
268.7
284.7
300.7
316.7
332.7
348.7
364.7
380.7
396.7
412.7
428.7
444.7
460.7
476.7
492.7
508.7
524.7
540.7
556.7
572.7
588.7
604.7
620.7
636.7
652.7
668.7
684.7
700.7
716.7
732.7
748.7
764.7
780.7
796.7
812.7
828.7
844.7
860.7
876.7
892.7
908.7
924.7
940.7
956.7
972.7
988.7
1004.7
1020.7
1036.7
1052.7
1068.7
1084.7
1100.7
1116.7
1132.7
1148.7
1164.7
1180.7
1196.7
1212.7
1228.7
1244.7
1260.7
1276.7
1292.7
1308.7
1324.7
1340.7
1356.7
1372.7
1388.7
1404.7
1420.7
1436.7
1452.7
1468.7
1484.7
1500.7
1516.7
1532.7
1548.7
1564.7
1580.7
1596.7
1612.7
1628.7
1644.7
1660.7
1676.7
1692.7
1708.7
1724.7
1740.7
1756.7
1772.7
1788.7
1804.7
1820.7
1836.7
1852.7
1868.7
1884.7
1900.7
1916.7
1932.7
1948.7
1964.7
1980.7
1996.7
2012.7
2028.7
2044.7
2060.7
2076.7
2092.7
2108.7
2124.7
2140.7
2156.7
2172.7
2188.7
2204.7
2220.7
2236.7
2252.7
2268.7
2284.7
2300.7
2316.7
2332.7
2348.7
2364.7
2380.7
2396.7
2412.7
2428.7
2444.7
2460.7
2476.7
2492.7
2508.7
2524.7
2540.7
2556.7
2572.7
2588.7
2604.7
2620.7
2636.7
2652.7
2668.7
2684.7
2700.7
2716.7
2732.7
2748.7
2764.7
2780.7
2796.7
2812.7
2828.7
2844.7
2860.7
2876.7
2892.7
2908.7
2924.7
2940.7
2956.7
2972.7
2988.7
3004.7
3020.7
3036.7
3052.7
3068.7
3084.7
3100.7
3116.7
3132.7
3148.7
3164.7
3180.7
3196.7
3212.7
3228.7
3244.7
3260.7
3276.7
3292.7
3308.7
3324.7
3340.7
3356.7
3372.7
3388.7
3404.7
3420.7
3436.7
3452.7
3468.7
3484.7
3500.7
3516.7
3532.7
3548.7
3564.7
3580.7
3596.7
3612.7
3628.7
3644.7
3660.7
3676.7
3692.7
3708.7
3724.7
3740.7
3756.7
3772.7
3788.7
3804.7
3820.7
3836.7
3852.7
1948.7
44.7




--
best,
Eliot
Free forum by Nabble Edit this page