Measuring allocations
Locked
 
|
Hi folks, Having a Spur VM, I wonder how to measure heap allocations made by a process executing a particular block, from within the image. For example, let's have following block: [ | a | 100 timesRepeat: [ a := Array new: 100 ] ]. When run, this block allocates - if I'm correct - some 100 * (<object header size> + 4*100) bytes. Note that: * Ideally, I'd like to exclude allocations made by other threads but having allocations made by all threads would be OK too. * Block can span multiple scavenges even oldspace collects. * As shown in the example above, I need to take garbage into an account. How to best measure this? Thanks, Jan |
|
|
Hi Jan, > On Jun 1, 2017, at 1:17 AM, Jan Vrany <[hidden email]> wrote: > > > Hi folks, > > Having a Spur VM, I wonder how to measure heap allocations made > by a process executing a particular block, from within the image. > For example, let's have following block: > > [ > | a | > > 100 timesRepeat: [ > a := Array new: 100 > ] > ]. > > When run, this block allocates - if I'm correct - some 100 * (<object > header size> + 4*100) bytes. Plus the block arg to timesRepeat:. And the size of an Array in 32-bits is (slots >= 255 ifTrue: [16] ifFalse: [8]) + (slots = 0 ifTrue: [8] ifFalse: [slots + 1 * 2 // 2 * 8]) i.e. 32-bit objects are rounded up to a 64-bit boundary. Spur objects have at least one slot. Slot sizes greater than 254 require an overflow header (255 marks an overflow header). Hence in 64-bits (slots >= 255 ifTrue: [16] ifFalse: [8]) + (slots = 0 ifTrue: [8] ifFalse: [slots * 8]) There's a primitive in Behavior that answers the byte size of an instance so you don't have to know the formula. The primitive is used in one of the failingBasicNew: methods to compute how much to grow memory by when a huge allocation fails. > Note that: > * Ideally, I'd like to exclude allocations made by other threads Then run at high priority. > but having allocations made by all threads would be OK too. > * Block can span multiple scavenges even oldspace collects. > * As shown in the example above, I need to take garbage into > an account. > > How to best measure this? What exactly do you want to measure? There is no allocation count, as that would slow down allocation. Time taken to allocate would be derived by sending timeTiRun to the above and subtracting the time for a null loop to run, and the time taken in GC. The time taken in GC is maintained by the vm. See vmParameterAt: and surrounding. If you open the About Squeak dialog in a Squeak image this is nicely displayed in the VM Stats tab (IIRC). > Thanks, Jan Cheers! Eliot |
|
|
On Thu, 2017-06-01 at 07:52 -0700, Eliot Miranda wrote: > > Hi Jan, > > > On Jun 1, 2017, at 1:17 AM, Jan Vrany <[hidden email]> > > wrote: > > > > > > Hi folks, > > > > Having a Spur VM, I wonder how to measure heap allocations made > > by a process executing a particular block, from within the image. > > For example, let's have following block: > > > > [ > > | a | > > > > 100 timesRepeat: [ > > a := Array new: 100 > > ] > > ]. > > > > When run, this block allocates - if I'm correct - some 100 * > > (<object > > header size> + 4*100) bytes. > > Plus the block arg to timesRepeat:. And the size of an Array in 32- > bits is > Sure, sure. The exact number is not important for me right now. > > Note that: > > * Ideally, I'd like to exclude allocations made by other threads > > Then run at high priority. Good point, thanks. I always forget about that. > > > but having allocations made by all threads would be OK too. > > * Block can span multiple scavenges even oldspace collects. > > * As shown in the example above, I need to take garbage into > > an account. > > > > How to best measure this? > > What exactly do you want to measure? I want to know how much given code allocates. For example that given piece of code in total allocated 2.5GB. Say. > There is no allocation count, as that would slow down allocation. Fair enough. > Time taken to allocate would be derived by sending timeTiRun to the > above and subtracting the time for a null loop to run, and the time > taken in GC. > > The time taken in GC is maintained by the vm. See vmParameterAt: and > surrounding. If you open the About Squeak dialog in a Squeak image > this is nicely displayed in the VM Stats tab (IIRC). It is indeed. But in this case I'm not interested in time spent in GC :-) Thanks, Jan |
|
|
You should be able do that in your image by replacing #basicNew, #basicNew: and #shallowCopy with method wrappers which would sum up the size of the allocated objects in ProcessLocalVariables before returning the new objects. Levente On Thu, 1 Jun 2017, Jan Vrany wrote: > > On Thu, 2017-06-01 at 07:52 -0700, Eliot Miranda wrote: >> >> Hi Jan, >> >> > On Jun 1, 2017, at 1:17 AM, Jan Vrany <[hidden email]> >> > wrote: >> > >> > >> > Hi folks, >> > >> > Having a Spur VM, I wonder how to measure heap allocations made >> > by a process executing a particular block, from within the image. >> > For example, let's have following block: >> > >> > [ >> > | a | >> > >> > 100 timesRepeat: [ >> > a := Array new: 100 >> > ] >> > ]. >> > >> > When run, this block allocates - if I'm correct - some 100 * >> > (<object >> > header size> + 4*100) bytes. >> >> Plus the block arg to timesRepeat:. And the size of an Array in 32- >> bits is >> > > Sure, sure. The exact number is not important for me right now. > >> > Note that: >> > * Ideally, I'd like to exclude allocations made by other threads >> >> Then run at high priority. > > Good point, thanks. I always forget about that. > >> >> > but having allocations made by all threads would be OK too. >> > * Block can span multiple scavenges even oldspace collects. >> > * As shown in the example above, I need to take garbage into >> > an account. >> > >> > How to best measure this? >> >> What exactly do you want to measure? > > I want to know how much given code allocates. For example that > given piece of code in total allocated 2.5GB. Say. > > >> There is no allocation count, as that would slow down allocation. > > Fair enough. > >> Time taken to allocate would be derived by sending timeTiRun to the >> above and subtracting the time for a null loop to run, and the time >> taken in GC. >> >> The time taken in GC is maintained by the vm. See vmParameterAt: and >> surrounding. If you open the About Squeak dialog in a Squeak image >> this is nicely displayed in the VM Stats tab (IIRC). > > It is indeed. But in this case I'm not interested in time spent in GC > :-) > > Thanks, Jan |
|
|
Hi Levente, > On Jun 2, 2017, at 3:59 AM, Levente Uzonyi <[hidden email]> wrote: > > You should be able do that in your image by replacing #basicNew, #basicNew: and #shallowCopy with method wrappers which would sum up the size of the allocated objects in ProcessLocalVariables before returning the new objects. This only catches explicit allocations. It misses allocations of blocks, indirection vectors, contexts MNU messages, and boxed floats and integers that the vm allocates as side effects of execution or the results of arithmetic primitives. The VM could easily account for the bytes allocated (at each scavenge measure size of eden; growth of old space is already accounted for). > > Levente > >> On Thu, 1 Jun 2017, Jan Vrany wrote: >> >>> On Thu, 2017-06-01 at 07:52 -0700, Eliot Miranda wrote: >>> >>> Hi Jan, >>> > On Jun 1, 2017, at 1:17 AM, Jan Vrany <[hidden email]> >>> > wrote: >>> > > > Hi folks, >>> > > Having a Spur VM, I wonder how to measure heap allocations made >>> > by a process executing a particular block, from within the image. >>> > For example, let's have following block: >>> > > [ >>> > | a | >>> > > 100 timesRepeat: [ >>> > a := Array new: 100 >>> > ] >>> > ]. >>> > > When run, this block allocates - if I'm correct - some 100 * >>> > (<object >>> > header size> + 4*100) bytes. >>> Plus the block arg to timesRepeat:. And the size of an Array in 32- >>> bits is >> >> Sure, sure. The exact number is not important for me right now. >> >>> > Note that: >>> > * Ideally, I'd like to exclude allocations made by other threads >>> Then run at high priority. >> >> Good point, thanks. I always forget about that. >>> > but having allocations made by all threads would be OK too. >>> > * Block can span multiple scavenges even oldspace collects. >>> > * As shown in the example above, I need to take garbage into >>> > an account. >>> > > How to best measure this? >>> What exactly do you want to measure? >> >> I want to know how much given code allocates. For example that >> given piece of code in total allocated 2.5GB. Say. >> >>> There is no allocation count, as that would slow down allocation. >> >> Fair enough. >>> Time taken to allocate would be derived by sending timeTiRun to the >>> above and subtracting the time for a null loop to run, and the time >>> taken in GC. >>> The time taken in GC is maintained by the vm. See vmParameterAt: and >>> surrounding. If you open the About Squeak dialog in a Squeak image >>> this is nicely displayed in the VM Stats tab (IIRC). >> >> It is indeed. But in this case I'm not interested in time spent in GC >> :-) >> Thanks, Jan |
|
|
In reply to this post by Jan Vrany
Hi Jan,
On Thu, Jun 1, 2017 at 1:17 AM, Jan Vrany <[hidden email]> wrote:
I just committed the code for Spur, so as soon as the CI has built VMs you should be able to do initialAllocation := Smalltalk vmParameterAt: 34. self doMyThing. bytesAllocated := (Smalltalk vmParameterAt: 34) - initialAllocation and/or Smalltalk vmParameterAt: 34 put: 0. self doMyThing. bytesAllocated := Smalltalk vmParameterAt: 34 It needs testing. Note that it measures bytes allocated at the lowest level I can manage so it should include everything, including hidden allocations in the GC's marked stack and remembered tables, and the class tables. I *think* the code is not confused by heap growth and shrinkage. I'd love for anyone interested to review the code (see the VMMaker.oscog-eem.2237 commit). Thanks, Jan _,,,^..^,,,_ best, Eliot |
|
|
Hi Eliot, On Wed, 2017-06-07 at 18:17 -0700, Eliot Miranda wrote: > > Hi Jan, > > On Thu, Jun 1, 2017 at 1:17 AM, Jan Vrany <[hidden email]> > wrote: > > Hi folks, > > > > Having a Spur VM, I wonder how to measure heap allocations made > > by a process executing a particular block, from within the image. > > For example, let's have following block: > > > > [ > > | a | > > > > 100 timesRepeat: [ > > a := Array new: 100 > > ] > > ]. > > > > When run, this block allocates - if I'm correct - some 100 * > > (<object > > header size> + 4*100) bytes. > > > > Note that: > > * Ideally, I'd like to exclude allocations made by other threads > > but having allocations made by all threads would be OK too. > > * Block can span multiple scavenges even oldspace collects. > > * As shown in the example above, I need to take garbage into > > an account. > > > > How to best measure this? > > I just committed the code for Spur, so as soon as the CI has built > VMs you should be able to do > > initialAllocation := Smalltalk vmParameterAt: 34. > self doMyThing. > bytesAllocated := (Smalltalk vmParameterAt: 34) > - initialAllocation > > and/or > > Smalltalk vmParameterAt: 34 put: 0. > self doMyThing. > bytesAllocated := Smalltalk vmParameterAt: 34 Nice, thanks! I just tried and it seems to work fine - it does return plausible numbers for some quick and trivial tests. > > It needs testing. Note that it measures bytes allocated at the > lowest level I can manage so it should include everything, including > hidden allocations in the GC's marked stack and remembered tables, > and the class tables. I *think* the code is not confused by heap > growth and shrinkage. I'd love for anyone interested to review the > code (see the VMMaker.oscog-eem.2237 commit). I'm sorry I don't feel experienced enough / familiar with VM internals to review the code. The general idea is OK - or at least it worked well enough for me in the past. But the devil is in the detail, as always. Anyways, thanks! Jan > > Thanks, Jan > > > > _,,,^..^,,,_ > best, Eliot |
«
Return to Squeak VM
|
1 view|%1 views
| Free forum by Nabble | Edit this page |