sweep failed to find exact end of memory
Locked
12
12
 
|
I looked at this a couple of years ago and put together a change set
which carried forward the intent of what a routine returned in Slang. Mostly to catch all the return voids. However we discovered an interesting thing, may of the vmMachine.h callbacks say oh that returns integer, however the routine does NOT return anything, a favourite one was setting the primitive failure value and expecting to have a value come back, then later Oh an if that depends on it. The VM dev community punted on the idea. However in looking at this, there were only a few places where the code actually returns a pointer value, of course then many people grab it and play with it thinking it's a signed integer. So all of that would have to be reviewed, all callers set variables to the proper types, and likely check all the plugins and VM support code. I of course would be delighted to help Tim in this multi-month effort if there is funding... Plus of course it would be good to make some Sunits to validate stuff. On Feb 21, 2007, at 11:10 AM, tim Rowledge wrote: > That's a distressingly difficult question to answer. I bet there > are some nasty hacks one could do that would 'fix' it on the basis > of forcing the C compiler to do signed-unsigned conversions but... > I never trust a C compiler. Aside from Slang code there are of > course the primitives and platform code that would be indirectly > affected, if only at the level of "I have change this decl to stop > an oodleplex of compiler warnings obscuring a real problem". > There's also a boatload of VMMaker patches needing proper > integration anyway and it would stupid not to do them as well. > > I'd love to say, "oh a week" but long experience suggests that the > truth is normally "double the digits and up the unit" so probably > 'two months' is higher in the truthiness scale. > > tim > -- > tim Rowledge; [hidden email]; http://www.rowledge.org/tim > Strange OpCodes: BH: Branch and Hang > > > -- ======================================================================== === John M. McIntosh <[hidden email]> Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== === |
|
|
In reply to this post by Avi Bryant-2
The openBSD folks introduced a feature where when an application
starts up, memory is allocated in a different place than the last time. This was to avoid having an application startup and alway have the exact same memory allocation address pattern, thus perhaps enabling an exploit, some linux systems then adopted this policy, so the image memory isn't allocated at the 640K boundary rather elsewhere... On Feb 21, 2007, at 1:45 PM, Avi Bryant wrote: > On 2/21/07, tim Rowledge <[hidden email]> wrote: >> It's not so much the amount of memory as where it's allocated. A 10Mb >> allocation that involves >2Gb values will still go kaboom (There was >> supposed to be an earth-shattering Kaboom, where was the earth- >> shattering kaboom?) >> >> Some OSs allocate from low adresses; some from high. Some do weird >> things. > > Right - I see this a lot more under Linux than under OS X, for > example. > > Avi > -- ======================================================================== === John M. McIntosh <[hidden email]> Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== === |
Re: sweep failed to find exact end of memory
|
|
In reply to this post by Avi Bryant-2
Avi Bryant wrote:
> Do you have an estimate, hopefully under 1 million euros, of what it > would take to fix this? What it takes is a reliable setup to trigger the problem, preferably directly in the simulator. This is not a big deal if you can make it happen reliably, just having enough flight hours will eventually sort it out because you learn what patterns to look for. Cheers, - Andreas |
Re: sweep failed to find exact end of memory
|
|
Andreas Raab wrote:
> Avi Bryant wrote: >> Do you have an estimate, hopefully under 1 million euros, of what it >> would take to fix this? > > What it takes is a reliable setup to trigger the problem, preferably > directly in the simulator. This is not a big deal if you can make it > happen reliably, just having enough flight hours will eventually sort it > out because you learn what patterns to look for. I've been reading this thread with interest as I can hit this wall anytime I want. I am writing my app and was wanting to know how many objects I could store inside an image. So I created an ordered collection and attempted to populate it with millions of objects. In a workspace I did | oc | oc := OrderedCollection new. 1 to: 20000000 do: [:i | oc add: i]. It never completes. It always crashes with sweep failed to find exact end of memory This is on my laptop with 1.25gb of RAM 2gb of swap i386 Ubuntu Linux. I ran similar tests with Lua, Python and Ruby. First time attempting 100m objects. Lua was beautiful. It allocated 89m objects and stopped when memory was no longer available. It was still usable and had access to all that it allocated. I could t=nil to nil the table and then run garbagecollect, and all the memory was freed. Python didn't crash but I stopped it because it went into swap hell. Ruby crashed. Squeak never made it past I think 15m on my computer. :( I don't know if this helps or not. But I would dearly love Squeak to be able to compete better being more stable and faster. But in this simple allocation of objects it didn't. :( With hope. Jimmie |
Re: sweep failed to find exact end of memory
|
|
On Thu, 22 Feb 2007 11:24:43 -0600, Jimmie Houchin <[hidden email]>
wrote: > In a workspace I did > > | oc | > oc := OrderedCollection new. > 1 to: 20000000 do: [:i | oc add: i]. > > It never completes. It always crashes with > > sweep failed to find exact end of memory > > > This is on my laptop with 1.25gb of RAM 2gb of swap i386 Ubuntu Linux. Must be a Linux thing. I can do that on my Windows laptop with no problems. 2 GB of RAM, Win XP Pro. Squeak jumps from 40 MB used up to around 155 MB used when I open an inspector on the oc. Takes about 7 seconds if I don't inspect, 17 if I do. When I close the inspector, after it GC's it goes back down to 40 MB... Later, Jon -------------------------------------------------------------- Jon Hylands [hidden email] http://www.huv.com/jon Project: Micro Raptor (Small Biped Velociraptor Robot) http://www.huv.com/blog |
Re: sweep failed to find exact end of memory
|
|
Jon Hylands wrote:
> On Thu, 22 Feb 2007 11:24:43 -0600, Jimmie Houchin <[hidden email]> > wrote: > >> In a workspace I did >> >> | oc | >> oc := OrderedCollection new. >> 1 to: 20000000 do: [:i | oc add: i]. >> >> It never completes. It always crashes with >> >> sweep failed to find exact end of memory >> >> >> This is on my laptop with 1.25gb of RAM 2gb of swap i386 Ubuntu Linux. > > Must be a Linux thing. I can do that on my Windows laptop with no problems. > > 2 GB of RAM, Win XP Pro. > > Squeak jumps from 40 MB used up to around 155 MB used when I open an > inspector on the oc. Takes about 7 seconds if I don't inspect, 17 if I do. > > When I close the inspector, after it GC's it goes back down to 40 MB... Yes, Linux does play a role. From earlier in the thread how Squeak does is totally dependent on how the OS allocates memory to Squeak. In this instance it appears that Linux exacerbates the problem in Squeak. Tim says: "It's not so much the amount of memory as where it's allocated. A 10Mb allocation that involves >2Gb values will still go kaboom (There was supposed to be an earth-shattering Kaboom, where was the earth-shattering kaboom?) Some OSs allocate from low adresses; some from high. Some do weird things." Avi replies: "Right - I see this a lot more under Linux than under OS X, for example." I would love to be running on OS X servers. But at the moment the one I have is Linux. So I have to deal with it. So I guess if you are running an OS other than Linux, possibly put in a higher value than the 20m in my example above. I started at 100m and worked down to the 20m before I gave up and started working my way up instead. Jimmie |
Re: sweep failed to find exact end of memory
|
|
On Feb 22, 2007, at 19:20 , Jimmie Houchin wrote:
> Jon Hylands wrote: >> On Thu, 22 Feb 2007 11:24:43 -0600, Jimmie Houchin >> <[hidden email]> >> wrote: >> >>> In a workspace I did >>> >>> | oc | >>> oc := OrderedCollection new. >>> 1 to: 20000000 do: [:i | oc add: i]. >>> >>> It never completes. It always crashes with >>> >>> sweep failed to find exact end of memory >>> >>> >>> This is on my laptop with 1.25gb of RAM 2gb of swap i386 Ubuntu >>> Linux. >> >> Must be a Linux thing. I can do that on my Windows laptop with no >> problems. >> >> 2 GB of RAM, Win XP Pro. >> >> Squeak jumps from 40 MB used up to around 155 MB used when I open an >> inspector on the oc. Takes about 7 seconds if I don't inspect, 17 >> if I do. >> >> When I close the inspector, after it GC's it goes back down to 40 >> MB... > > Yes, Linux does play a role. From earlier in the thread how Squeak > does > is totally dependent on how the OS allocates memory to Squeak. In this > instance it appears that Linux exacerbates the problem in Squeak. > > Tim says: > "It's not so much the amount of memory as where it's allocated. > A 10Mb allocation that involves >2Gb values will still go kaboom > (There was supposed to be an earth-shattering Kaboom, > where was the earth-shattering kaboom?) > > Some OSs allocate from low adresses; some from high. > Some do weird things." > > Avi replies: > "Right - > I see this a lot more under Linux than under OS X, for example." > > I would love to be running on OS X servers. But at the moment the > one I > have is Linux. So I have to deal with it. > > So I guess if you are running an OS other than Linux, possibly put > in a > higher value than the 20m in my example above. I started at 100m and > worked down to the 20m before I gave up and started working my way up > instead. If someone had a windows machine with more than 2 GB RAM and if the Windows VM allows to allocate more than 2 GB for an image then you should run into the same problem. Now, I do have 3 GB in my Mac so I might test this theory - however, SqueakMaxHeapSize seems to be a 32 bit signed int in the plist so I cannot set it to larger than 2GB ... - Bert - |
Re: sweep failed to find exact end of memory
|
|
In reply to this post by Jimmie Houchin-3
Hi!
>>> sweep failed to find exact end of memory >>> >>> >>> This is on my laptop with 1.25gb of RAM 2gb of swap i386 Ubuntu Linux. >> >> Must be a Linux thing. I can do that on my Windows laptop with no >> problems. >> >> 2 GB of RAM, Win XP Pro. >> >> Squeak jumps from 40 MB used up to around 155 MB used when I open an >> inspector on the oc. Takes about 7 seconds if I don't inspect, 17 if I >> do. >> >> When I close the inspector, after it GC's it goes back down to 40 MB... > > Yes, Linux does play a role. From earlier in the thread how Squeak does > is totally dependent on how the OS allocates memory to Squeak. In this > instance it appears that Linux exacerbates the problem in Squeak. And it has been reported from at least one Solaris user and IIRC an OpenBSD user. In Linux it started from kernel 2.6, can't recall which version exactly but it is closely related to how the virtual memory works in Linux of course. People tend to notice it when they load ImageSegments which the SqueakMap Package Loader does - and so they sometimes begin with wondering if SqueakMap is somehow the culprit. :) There are tons of posts on the subject of course in the archives. I have avoided the problem myself by simply using the -memory option which seems to get Squeak allocated low. regards, Göran |
|
|
On Feb 22, 2007, at 11:54 PM, Göran Krampe wrote: > There are tons of posts on the subject of course in the archives. I > have > avoided the problem myself by simply using the -memory option which > seems > to get Squeak allocated low. > > regards, Göran I thought the unix VM tried to allocation 1GB of memory, the mac carbon VM did the same (years ago) but I changed it to 512MB because of this issue which in 10.1 seems fairly frequent. However Sophie has been set up to run at 1GB and we've not *heard* of problems with this issue. Mind it is fatal, so perhaps users don't tell us, and we certainly wouldn't get the walkback logs. -- ======================================================================== === John M. McIntosh <[hidden email]> Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com ======================================================================== === |
Re: sweep failed to find exact end of memory
|
|
In reply to this post by Göran Krampe
> Hi!
> > >>> sweep failed to find exact end of memory > >>> > >>> > >>> This is on my laptop with 1.25gb of RAM 2gb of swap i386 Ubuntu Linux. > >> > >> Must be a Linux thing. I can do that on my Windows laptop with no > >> problems. > >> Hi, if it is a Linux thing maybe playing with the Highmem options under " Processor type and features ---> " in the kernel config could help. There are several options: x High Memory Support (4GB) ---> x x x x Memory split (3G/1G user/kernel split) ---> x x x x Memory model (Flat Memory) ---> x x x x[*] Allocate 3rd-level pagetables from highmem x x which all must have some impact on memory allocation. Cheers, Torsten |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |