instantiateClass:indexableSize: upper limit?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

instantiateClass:indexableSize: upper limit?

Tom Beckmann
 
Dear VM devs,

I'm trying to allocate a Bitmap of around 16MB size in a plugin via `interpreterProxy instantiateClass: interpreterProxy classBitmap indexableSize: size`.
However, I always get NULL back. I already tried wrapping the primitive call in `retryWithGC:until:`, with the same effect. Running a normal `Bitmap new: 16000000` from a workspace works flawlessly.

I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.

Is there anything to watch out for? More precisely, I'm trying to copy the contents of an image buffer generated from a pdf via libpoppler at 144dpi to a new bitmap in squeak. The primitive and how I invoke it can be seen here: https://pastebin.com/9NCdnx2J (line 50 is the call in question)

Thanks in advance for any pointers!
Tom
Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

Bert Freudenberg
 
On Fri, Jun 23, 2017 at 10:35 AM, Tom Beckmann <[hidden email]> wrote:
 
Dear VM devs,

I'm trying to allocate a Bitmap of around 16MB size in a plugin via `interpreterProxy instantiateClass: interpreterProxy classBitmap indexableSize: size`.
However, I always get NULL back. I already tried wrapping the primitive call in `retryWithGC:until:`, with the same effect. Running a normal `Bitmap new: 16000000` from a workspace works flawlessly.

I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.

Is there anything to watch out for?

​Yes. Object allocation is better handled in the image, because you don't want to deal with garbage collection strategies in a primitive. The typical way is to pass a pre-allocated array into the primitive which fills it.

- Bert -​

Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

Eliot Miranda-2
In reply to this post by Tom Beckmann
 
Hi Tom,

On Fri, Jun 23, 2017 at 1:35 AM, Tom Beckmann <[hidden email]> wrote:
 
Dear VM devs,

I'm trying to allocate a Bitmap of around 16MB size in a plugin via `interpreterProxy instantiateClass: interpreterProxy classBitmap indexableSize: size`.
However, I always get NULL back. I already tried wrapping the primitive call in `retryWithGC:until:`, with the same effect. Running a normal `Bitmap new: 16000000` from a workspace works flawlessly.

I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.

Is there anything to watch out for? More precisely, I'm trying to copy the contents of an image buffer generated from a pdf via libpoppler at 144dpi to a new bitmap in squeak. The primitive and how I invoke it can be seen here: https://pastebin.com/9NCdnx2J (line 50 is the call in question)

Thanks in advance for any pointers!

I want to endorse what Bert said; allocating in the image is a much better approach.  But I did want to explain what you're seeing, assuming you're using Spur (Squeak 5.x, Pharo 6).

The Spur old space heap is segmented.  It grows in increments defined by vm parameter 25 (Smalltalk vmParameterAt:[put:]), and shrinks whenever free memory is above parameter 24 and GC empties one or more segments.  It will not grow memory beyond parameter 67, if that parameter is non-zero:

24 memory threshold above which to shrink object memory (read-write)
25 memory headroom when growing object memory (read-write)
67 the maximum allowed size of old space in bytes, 0 implies no internal limit (Spur only).

Parmeter 25 defaults to 16Mb (grow in increments of 16Mb) and 24 to 32 Mb (only shrink while 32Mb or more of empty segments exist).

Currently memory will /only/ grow in these circumstances
- under image control via growMemoryByAtLeast:
- when scavenging tenures sufficient objects to old space that old space must grow
- when a fullGC fails to ensure there is at least vm parameter 25's worth of free space
- when an allObjects or allInstances primitive needs more room to answer its result

The current allocator does /not/ try and grow memory in the allocation routines (which are accessed in plugins via instantiateClass:indexableSize: and in the image via new: basicNew: et al), failing instead.  This gives the image the chance to mediate GC and heap growth (see e.g. Behavior>>handleFailing[Failing]BasicNew:).

Perhaps instantiateClass:indexableSize: /should/ allow growth but I prefer it remaining the same.  You would then rewrite your primitive to
- when it fails to allocate memory via instantiateClass:indexableSize: fail with interpreterProxy primitiveFailFor: PrimErrNoMemory
- have the Smalltalk primitive failure code check the primitive error code (see senders of primitive:module:error:) and if the failure code is #'insufficient object memory' grow memory by the necessary amount and retry the primitive (a la handleFailingBasicNew:)


To the general audience, I think information like the above is key to being able to understand and exploit the system effectively, but where should it reside?  Clearly "in my head" is not satisfactory.  It belongs somewhere in the image, but it needs to be somewhere where people can find it and/or will look.  Suggestions for an "architectural information" documentation section gratefully received.  Object class>>whatIsAPrimitive might perhaps work or perhaps be overloaded.

_,,,^..^,,,_
best, Eliot
Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

timrowledge
 

> On 23-06-2017, at 9:45 AM, Eliot Miranda <[hidden email]> wrote:
>
> To the general audience, I think information like the above is key to being able to understand and exploit the system effectively, but where should it reside?  Clearly "in my head" is not satisfactory.  It belongs somewhere in the image, but it needs to be somewhere where people can find it and/or will look.  Suggestions for an "architectural information" documentation section gratefully received.  Object class>>whatIsAPrimitive might perhaps work or perhaps be overloaded.

A brief explanation in the sources as a comment for the class and/or some relevant methods *plus* a link to a swiki page with the full details. The briefest explanation might well be as little as “this is a quite complex issue; see swiki.squeak.org/85643505485"


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- A .22 caliber intellect in a .357 Magnum world.


Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

Juan Vuletich-3
 
On 6/23/2017 2:01 PM, tim Rowledge wrote:

>
>
>> On 23-06-2017, at 9:45 AM, Eliot Miranda<[hidden email]>  wrote:
>>
>> To the general audience, I think information like the above is key to being able to understand and exploit the system effectively, but where should it reside?  Clearly "in my head" is not satisfactory.  It belongs somewhere in the image, but it needs to be somewhere where people can find it and/or will look.  Suggestions for an "architectural information" documentation section gratefully received.  Object class>>whatIsAPrimitive might perhaps work or perhaps be overloaded.
> A brief explanation in the sources as a comment for the class and/or some relevant methods *plus* a link to a swiki page with the full details. The briefest explanation might well be as little as “this is a quite complex issue; see swiki.squeak.org/85643505485"
>
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Useful random insult:- A .22 caliber intellect in a .357 Magnum world.

But this information is for Smalltalk developers, not just for VM
developers. And urls might eventually break.

I think the full information on how to better use the VM belong in some
file in VM sources, maybe one or several .md files. And Smalltalk
systems using the VMs might chose to duplicate it in the image. I'd
surely do it for Cuis.

This might be too much for Object class>>whatIsAPrimitive. I think some
methods in a 'documentation' category in SystemDictionary would be
better. Or one (or several) new classes, just for this. Better if we can
share them in all the distributions: Squeak, Pharo, Newspeak and Cuis.

Thanks,

--
Juan Vuletich
www.cuis-smalltalk.org
https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev
@JuanVuletich


Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

Jakob Reschke-2
In reply to this post by timrowledge
 
There are also the HelpTopic classes. WebClient has some IIRC. Since the texts reside in those classes, they would be placed next to the code in the package.

Am 23.06.2017 21:03 schrieb "Juan Vuletich" <[hidden email]>:

On 6/23/2017 2:01 PM, tim Rowledge wrote:
>
>
>> On 23-06-2017, at 9:45 AM, Eliot Miranda<[hidden email]>  wrote:
>>
>> To the general audience, I think information like the above is key to being able to understand and exploit the system effectively, but where should it reside?  Clearly "in my head" is not satisfactory.  It belongs somewhere in the image, but it needs to be somewhere where people can find it and/or will look.  Suggestions for an "architectural information" documentation section gratefully received.  Object class>>whatIsAPrimitive might perhaps work or perhaps be overloaded.
> A brief explanation in the sources as a comment for the class and/or some relevant methods *plus* a link to a swiki page with the full details. The briefest explanation might well be as little as “this is a quite complex issue; see swiki.squeak.org/85643505485"
>
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Useful random insult:- A .22 caliber intellect in a .357 Magnum world.

But this information is for Smalltalk developers, not just for VM
developers. And urls might eventually break.

I think the full information on how to better use the VM belong in some
file in VM sources, maybe one or several .md files. And Smalltalk
systems using the VMs might chose to duplicate it in the image. I'd
surely do it for Cuis.

This might be too much for Object class>>whatIsAPrimitive. I think some
methods in a 'documentation' category in SystemDictionary would be
better. Or one (or several) new classes, just for this. Better if we can
share them in all the distributions: Squeak, Pharo, Newspeak and Cuis.

Thanks,

--
Juan Vuletich
www.cuis-smalltalk.org
https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev
@JuanVuletich


Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

David T. Lewis
In reply to this post by Eliot Miranda-2
 
On Fri, Jun 23, 2017 at 09:45:45AM -0700, Eliot Miranda wrote:

>  
> Hi Tom,
>
> On Fri, Jun 23, 2017 at 1:35 AM, Tom Beckmann <[hidden email]> wrote:
>
> >
> > Dear VM devs,
> >
> > I'm trying to allocate a Bitmap of around 16MB size in a plugin via
> > `interpreterProxy instantiateClass: interpreterProxy classBitmap
> > indexableSize: size`.
> > However, I always get NULL back. I already tried wrapping the primitive
> > call in `retryWithGC:until:`, with the same effect. Running a normal
> > `Bitmap new: 16000000` from a workspace works flawlessly.
> >
> > I'm on a Ubuntu 16.04 64bit, running a squeak 32bit built from trunk.
> >
> > Is there anything to watch out for? More precisely, I'm trying to copy the
> > contents of an image buffer generated from a pdf via libpoppler at 144dpi
> > to a new bitmap in squeak. The primitive and how I invoke it can be seen
> > here: https://pastebin.com/9NCdnx2J (line 50 is the call in question)
> >
> > Thanks in advance for any pointers!
> >
>
> I want to endorse what Bert said; allocating in the image is a much better
> approach.  But I did want to explain what you're seeing, assuming you're
> using Spur (Squeak 5.x, Pharo 6).

I also want to endorse Bert's advice here. But by way of practical advice
if you did really do need to debug your primitive:

Compile the VM with debugging enabled and compiler optimization disabled. Run
the VM under a debugger (gdb or whatever). Put a breakpoint at the primitive
function, and step through it to see where things are going wrong.

If you are dealing with a primitive that is called many times from the image,
then just make a second copy of the primitive and give it a different name (so
if you are working on #primitiveFoo, then make another #primitiveFoo2). You
can then call that second copy from the image, and debug it when you hit the
breakpoint in gdb.

You mentioned that you were working with "trunk" VM sources. If that refers
to the interpreter VM, then in the makefile example at
http://squeakvm.org/cgi-bin/viewvc.cgi/squeak/trunk/platforms/unix/cmake/Makefile.example?revision=3753&view=markup
you can use this to compile with debugging and no optimization:

  CFLAGS_PARAM="--CFLAGS='-O0 -g'"

In Spur/Cog the equivalent is in the mvm scripts. Either way, the idea is
to compile your plugin with debugging symbols enabled, and with all of the
compiler optimizations turned off. Once you have that, you can poke around
in the dgb debugger and see what is going wrong. Nine times out of ten it
will turn out to be something you never would have guessed :-)

HTH,
Dave

Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

David T. Lewis
In reply to this post by Jakob Reschke-2
 
Can we follow Juan's guideance of of putting the documentation in a 'documentation'
category in SystemDictionary, and then make that information visible from the help
browsers? That will put the information in a well-known location for all distributions,
and also make it presentable through the various help browsers.

Dave


On Fri, Jun 23, 2017 at 09:55:10PM +0200, Jakob Reschke wrote:

>  
> There are also the HelpTopic classes. WebClient has some IIRC. Since the
> texts reside in those classes, they would be placed next to the code in the
> package.
>
> Am 23.06.2017 21:03 schrieb "Juan Vuletich" <[hidden email]>:
>
> >
> > On 6/23/2017 2:01 PM, tim Rowledge wrote:
> > >
> > >
> > >> On 23-06-2017, at 9:45 AM, Eliot Miranda<[hidden email]>
> > wrote:
> > >>
> > >> To the general audience, I think information like the above is key to
> > being able to understand and exploit the system effectively, but where
> > should it reside?  Clearly "in my head" is not satisfactory.  It belongs
> > somewhere in the image, but it needs to be somewhere where people can find
> > it and/or will look.  Suggestions for an "architectural information"
> > documentation section gratefully received.  Object class>>whatIsAPrimitive
> > might perhaps work or perhaps be overloaded.
> > > A brief explanation in the sources as a comment for the class and/or
> > some relevant methods *plus* a link to a swiki page with the full details.
> > The briefest explanation might well be as little as ???this is a quite
> > complex issue; see swiki.squeak.org/85643505485"
> > >
> > >
> > > tim
> > > --
> > > tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> > > Useful random insult:- A .22 caliber intellect in a .357 Magnum world.
> >
> > But this information is for Smalltalk developers, not just for VM
> > developers. And urls might eventually break.
> >
> > I think the full information on how to better use the VM belong in some
> > file in VM sources, maybe one or several .md files. And Smalltalk
> > systems using the VMs might chose to duplicate it in the image. I'd
> > surely do it for Cuis.
> >
> > This might be too much for Object class>>whatIsAPrimitive. I think some
> > methods in a 'documentation' category in SystemDictionary would be
> > better. Or one (or several) new classes, just for this. Better if we can
> > share them in all the distributions: Squeak, Pharo, Newspeak and Cuis.
> >
> > Thanks,
> >
> > --
> > Juan Vuletich
> > www.cuis-smalltalk.org
> > https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev
> > @JuanVuletich
> >
> >
> >

Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

timrowledge
 
Extending the help system is an excellent idea. A very good first job might be to properly condense all the items currently in the Help docking bar menu into the help browser and then get rid of those menu items.

Having everything within the help browser would have the desirable side effect of making it unnecessary to fix the odd behaviour shown if you choose Help->The Squeak User Interface (as one example)  - the help browser opens on that section, which is nice, but if you should happen to click on the ‘Search Results’ list entry then your help goes away and there is no way to retrieve it (other than starting from the beginning).

A help browser is an excellent place for longer form explanations of what a part of the system does and how it does it, tying together multiple classes and code scattered around. Making sure that there are pointers to the class-specific comments (or perhaps embedding them?) within those larger help articles would be smart. Likewise having pointers from a class to any help article  that discusses it would be nice. I very much like the way some help articles have executable examples with in them - full marks to whoever has provided them. Having the window menus for the main tools able to open a help browser aimed at the doc for the tool would be nice.

The most important thing about any help system is that it is actually used, so that it becomes a worthwhile habit to look in it for information. It isn’t too hard to make a tool that is easy to add information to, nor to make it easy to find stuff within. The real trick is getting everyone to remember to add well-written articles to it.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Fractured Idiom:- PORTE-KOCHERE - Sacramental wine


Reply | Threaded
Open this post in threaded view
|

Re: instantiateClass:indexableSize: upper limit?

Tom Beckmann
In reply to this post by David T. Lewis
 
Thanks a lot for the hints and the in-depth explanation!

Is there a best practice on how to deal with my scenario in the most elegant way? Inside the primitive call I learn how much memory I need and get the pointer to it. Calculating the size ahead of time is unfortunately not possible. So in order to allocate the memory in squeak, I would have to return the pointer from the primitive, right? Do I simply place it in an integer or is there a dedicated data structure? Or do I have to go all the way and return a handle that only lives for the duration of the two primitive calls?

I second what Jakob said very much. I really like the HelpBrowser approach and its possibility to just place do-it-able code inside it. Having a chapter on the very basics of plugin building would be awesome (like e.g. explaining what all the push/popRemappableOop stuff is all about).

Thanks again!
Tom

On Sat, Jun 24, 2017 at 2:03 AM tim Rowledge <[hidden email]> wrote:

Extending the help system is an excellent idea. A very good first job might be to properly condense all the items currently in the Help docking bar menu into the help browser and then get rid of those menu items.

Having everything within the help browser would have the desirable side effect of making it unnecessary to fix the odd behaviour shown if you choose Help->The Squeak User Interface (as one example)  - the help browser opens on that section, which is nice, but if you should happen to click on the ‘Search Results’ list entry then your help goes away and there is no way to retrieve it (other than starting from the beginning).

A help browser is an excellent place for longer form explanations of what a part of the system does and how it does it, tying together multiple classes and code scattered around. Making sure that there are pointers to the class-specific comments (or perhaps embedding them?) within those larger help articles would be smart. Likewise having pointers from a class to any help article  that discusses it would be nice. I very much like the way some help articles have executable examples with in them - full marks to whoever has provided them. Having the window menus for the main tools able to open a help browser aimed at the doc for the tool would be nice.

The most important thing about any help system is that it is actually used, so that it becomes a worthwhile habit to look in it for information. It isn’t too hard to make a tool that is easy to add information to, nor to make it easy to find stuff within. The real trick is getting everyone to remember to add well-written articles to it.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Fractured Idiom:- PORTE-KOCHERE - Sacramental wine