Can I get a notification before fullGC?

  Hello,

  Is there any way to get some sort of notification right *before* a
fullGC happens so that some guys release their cache?

  Is there something I can do for this?  Thank you so much!

  Below is a little background...

  I noticed that the caching behavior of TTCFont is *still* relying on
the old (and broken) behavior of weak references.  Back then, when a
WeakArray is in the old space and the referent is in the young space,
the referent didn't get reclaimed until a fullGC.  This behavior was
quite nice; it releases the cache only when a big memory clean up is
required.  However, on recent VMs, the cache is released always go
away because the reference from the weak array is the only reference
to it.

  Andreas already had a fix for it:
http://impara.de/pipermail/tweak/2005-August/000881.html and I folded
it to the eToys image (which should go into other images), and it
seems to be working as expected.  However, a little thing (which is
not critical as his patch limits the upper limit of cache entries) is
that the cache is kept even when a fullGC happens.

-- Yoshiki

Am 12.10.2006 um 03:59 schrieb Yoshiki Ohshima:

>   Is there any way to get some sort of notification right *before* a
> fullGC happens so that some guys release their cache?

Not that I know of.

> ... releases the cache only when a big memory clean up is
> required.

You could add it to the memoryHogs list ...

- Bert -

Yoshiki Ohshima wrote:
>   Is there any way to get some sort of notification right *before* a
> fullGC happens so that some guys release their cache?

There is no such ability and it would make little sense to add one since
  the VM doesn't know if you'll ever be able to trigger a manual GC
before it crashes. In other words: When you run out of memory you need
that memory back, and *fast*.

>   Is there something I can do for this?  Thank you so much!

What makes sense for a problem like you are describing is to have an
array that is cleared by the VM when running a full GC. The VM would
simply nil out everything and if there are no other (strong) references
to the objects in this array they'd get garbage collected alongside
everything else. This would give you the precise effect that the
previous (broken) weak array behavior had.

>   Andreas already had a fix for it:
> http://impara.de/pipermail/tweak/2005-August/000881.html and I folded
> it to the eToys image (which should go into other images), and it
> seems to be working as expected.  However, a little thing (which is
> not critical as his patch limits the upper limit of cache entries) is
> that the cache is kept even when a fullGC happens.

Personally, I find this behavior advantageous. The glyph cache can be
chosen to accommodate the working set for most use cases (I find the
current one too large by far but see below) and I wouldn't really want
to recreate the glyphs after each full GC.

BTW, if you're looking at TTFs again, the one thing you might want to
consider is to fix glyph color caching. Currently, the worst thing you
can do is to change colors. Say, when using syntax highlighting with
TTFs. Each time the color changes two bad things happen: First, we
recompute the glyph from scratch which is expensive, and second we fill
up the glyph cache with the same glyphs in different colors and
therefore throwing out previous "good" entries which means more
computation when that glyph needs to be displayed. I'll bet that fixing
glyph color caching would allow you dramatically reduce the glyph cache
size and simply ignore the whole notification on full GC issue.

Cheers,
   - Andreas

  Andreas,

  Yeah, a new BB combination rule that takes 8-bit (or 4 bit) form for
grayscale, and halftone (or colormap) for color and do the alpha
blending would be convenient.  I think it's been written a few times
by different people, but never gets in the mainstream VM.

-- Yoshiki

On 11-Oct-06, at 7:45 PM, Yoshiki Ohshima wrote:

>
>   Yeah, a new BB combination rule that takes 8-bit (or 4 bit) form for
> grayscale, and halftone (or colormap) for color and do the alpha
> blending would be convenient.  I think it's been written a few times
> by different people, but never gets in the mainstream VM.

So put together a decent version and submit it!


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Oxymorons: Advanced BASIC

The alternative is fixing the lookup scheme currently used in TTCFont.
E.g., for TTCFont all you need is to have a mapping from color to 256
element weak array and use that for lookup. Something like here:

TTCFont>>formOf: char
        | f assoc code |
        char charCode > 255
                ifTrue: [^ self fallbackFont formOf: char].

        cache := colorToCacheMap at: foreGroundColor
                ifAbsentPut:[WeakArray new: 256]

        "... etc ..."

Then all you need to do is some suitable flushing (say on system
shutdown) and you're done. (if you want to be *really* clever you also
cache the last cache so that you can avoid the colorToCacheMap lookup
for the common case of non-changing colors etc)

Cheers,
   - Andreas

Yoshiki Ohshima wrote:

Am 12.10.2006 um 04:18 schrieb Bert Freudenberg:
Yes, that is exactly the scenario I had in mind way back then (2000):
I don´t know, if the stuff still works in the current Squeak version,  
however...

Kind regards
Georg

Am 12.10.2006 um 10:13 schrieb Georg Gollmann:
> I don´t know, if the stuff still works in the current Squeak  
> version, however...

BTW, I just noticed a false positive on the mantis bug tracking  
system. http://bugs.impara.de/view.php?id=5173 claims:

> In 7061 (an way previous) the MemoryHogs class var is always nil or  
> empty.
> There seems to be no way of registering an object or a class as a  
> potential Memory Hog. And no class implements #freeSomeSpace which  
> is the whole idea.

Which misses the SystemDictionary>memoryHogs method and the fact that  
this is a hook for applications, so it is normal that there are no  
implementors in the base image.

Kind regards
Georg

  Thank you Andreas and Bert,

> Personally, I find this behavior advantageous. The glyph cache can be
> chosen to accommodate the working set for most use cases (I find the
> current one too large by far but see below) and I wouldn't really want
> to recreate the glyphs after each full GC.

  BTW, a typical use case I was thinking is project transition.  The
working sets can be pretty different across projects... (So, flushing
the cache upon project transition may be a good idea.)

  In regards to the one more mapping for different colors, I'd
delegate the measurement by somebody else, at least for the "October
release".

  And, memoryHog is a bit too tight.  The purpose is rather be a
better citizen to a relatively limited system, than avoiding critical
situations.

  A special array to nil-out upon fullGC would be good.  (If we have a
flexible and dynamically modifiable VM, this approach would make more
sense.)

-- Yoshiki

The VM has a counter that is incremented on each fullGC, you could  
have a periodic task that runs that pulls that vm statistic and then  
does what ever
when the fullGC counter increments.  However when or if a fullGC will  
run is always an interesting question.

On 12-Oct-06, at 3:23 AM, Yoshiki Ohshima wrote:

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===