Smalltalk › Cincom › VisualWorks

base.im questions

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

28 messages Options

Travis Griggs-4

TAG-MemoryUsage (Was: base.im questions)

On Jan 16, 2012, at 2:12 PM, Kooyman, Les wrote:

Re: [vwnc] base.im questions

I agree with the overhead comments related to internationalization.

In the follow-on release to 7.9, whatever its name may be, I'd like to move to dynamic allocation of locale instances. This would cut down on this memory pressure.

I was curious just where the PrintFormatTokens were being held, and how much memory was in them and other things. So I wrote a little tool described here:

http://screencast.com/t/exviTDoA5N7

Travis Griggs

Objologist

I multiply all time estimates by pi, to account for running around in circles.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Kooyman, Les

Re: base.im questions

In reply to this post by Kooyman, Les

Re: [vwnc] base.im questions

Andres:

As I said to Travis when he and I discussed this yesterday, at about the time the inflation in i18n was happening, there were other increases in the image as well. There was some discussion about it here and/or on VWDEV at that time, IIRC.

A quick check of the archives might be useful in that regard to refresh our memories.

Les

From: Valloud, Andres
Sent: Mon 1/16/2012 6:37 PM
To: Kooyman, Les
Subject: Re: [vwnc] base.im questions

That's not all of it, and IMO you should point out that:

a) there's no accounting of what's in base.im,

b) there has been no correlations made between certain ARs integrated
and base.im growth, which IMO should be very interesting,

c) even if I18N is the worst offender, we've had several integrations of
code that resulted in base.im growth... so what are those and how can
they be identified?...

I'm concerned that once I18N is taken care of, we'll mysteriously find
out it only accounts for 50-75% of the growth, and the rest will be
attributed to random causes which we won't investigate because "well we
did something already".

On 1/16/2012 2:12 PM, Kooyman, Les wrote:

> I agree with the overhead comments related to internationalization.
> In the follow-on release to 7.9, whatever its name may be, I'd like to
> move to dynamic allocation of locale instances. This would cut down on
> this memory pressure.
> Les
>
> ------------------------------------------------------------------------
> *From:* [hidden email] on behalf of Travis Griggs
> *Sent:* Mon 1/16/2012 12:58 PM
> *To:* James Robertson; VWNC NC
> *Subject:* Re: [vwnc] base.im questions
>
>
> On Jan 16, 2012, at 11:40 AM, James Robertson wrote:
>
> > I'd still say you're better off not using the RTP approach; trying to
> pull unused methods and classes (as opposed to whole packages) gains you
> very little at the cost of pretty high uncertainty.
> >
> > -- start with visual.im
> > -- load your parcels
> > -- use script like this to unload:
> >
> > (Store.Registry pundleNamed: #('PackageNameHere')) ifNotNil: [:pundle |
> > pundle leafItems do: [:each | each markNotModified].
> > [pundle markNotModified; unloadFromImage] on: [Warning do: [:ex | ex
> resume: true]].
> >
> > I'm sure there's an equivalent for images w/o Store loaded, but my
> builder starts from a Store image. The larger point is this: unloading
> packages and bundles removes large amounts of stuff; unloading classes
> and methods, not so much.
>
>
> I concur with this. Build up.
>
> But I think, Carl's issue is that the foundation he's building up from
> seems a little bit of overkill. I wouldn't personally use RTP to try and
> address this problem. My hunch is that there's some rather large low
> hanging fruit that you can remove with a couple of workspace/script like
> expressions.
>
> My bet, personally, is along the same lines as Martin's comment. The
> primary addition to image size seems to have been collation and locale
> stuff. On my image for example, the following expression:
>
> ObjectMemory current numOopsNumBytesIn: PrintFormatToken allGeneralInstances
>
> indicates there are ~57000 instances chewing up 1.35 MB of memory. I'm
> guessing I don't need that many instances of PrintFormatTokens to do my
> every day job.
>
> --
> Travis Griggs
> Objologist
> Simplicity is the Ultimate Sophistication -- Leonardo da Vinci
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

jarober

Re: base.im questions

In reply to this post by Christian Haider

I'm not sure why you would think that; it produces predictable builds, whereby I can tell exactly what is (and is not) in the end runtime. Using the process we have at work, we end up with a final development image that mirrors the runtime (in terms of what app pieces are in it). That means that any runtime problem can be reproduced in the dev image (which is not always the case using the RTP).

To be honest, getting another 1 or 2MB out via using the RTP isn't worth the unpredictability, IMNSHO.

On Jan 17, 2012, at 10:47 AM, Christian Haider wrote:

> James, I always thought of your simple packaging procedure as being fine
> for toy programs and hobby applications. But not for professional
> software.
>
> It is true though: using RTP requires significant effort.
>
> My packaging process is:
> 1. parcel out all my runtime packages/bundles.
> 2. load my parcels (and other libraries) into the base.im
> 3. load RTP and set all parameters, including "parcels to unload"
> 4. run RTP with the 3-step save method
> 5. adding the compressed runtime.im to the VM with reshacker to get a
> single file executable
>
> All steps are automated by one procedure (actually methods containing
> configurations and batch writing methods) which can be triggered-off
> with one click.
> This is important, since operating the RTP manually via interface is far
> too error-prone to be reliable.
> EFFORT (one time): automate you packaging procedure.
>
> My modules have always one main bundle with one runtime bundle and
> additional stuff for development and testing. I take care of only having
> runtime relevant code in the runtime bundle. Sometimes I check the
> removed code report from the RTP to identify unused code - this is
> simple for top level code, but can be tricky for libraries.
> EFFORT (continuous): keep the runtime clean by maintaining the
> runtime/dev structure.
>
> RTP has the nice feature of declaring dynamically referenced classes and
> methods. This needs to be done for generic library code (including lots
> of VisualsWorks code - the developers seem to be not too aware of
> runtime and packaging issues).
> EFFORT (once for each VW release or when libraries are used for the
> first time): debug the packaging by trial-and-error until all missing
> method errors are resolved - this can be tricky.
>
> RTP lets you unload parcels before packaging. This works nicely for many
> parcels. But there are also bad examples - really bad is COM where no
> distinction between runtime and development is maintained, although the
> package names suggest differently. Without RTP it is not possible to
> remove anything from COM (several MB).
>
> Overall, I think that this effort is reasonable and the procedure is
> fine.
> A bit more support from Cincom would be appreciated, though.
>
> Some numbers all sizes in MB):
> Version base.im size my code size exe size
> 7.4.1 8.620 3.48 / - 5.408 / -
> 7.5 8.941 6.49 / - 5.912 / -
> 7.6 9.457 6.96 / 8.46 5.790 / 5.930
> 7.7 13.716 8.61 / 7.81 7.417 / 7.970
> 7.7.1 14.129 - / 51.1 - / 13.039
> 7.8 15.078 54.7 / 51.7 15.297 / 15.981
>
> The size of my code (of two of my products) is not very meaningful,
> since I load other code as well.
> The exe size includes the VM.
>
> The base.im gets bigger with every release (jumped whooping 4 MB with
> the locales).
> PLEASE Cincom, invest a bit in cleaning the base.im.
> I want my single digit MB executables back...
>
> Cheers,
> Christian

James Robertson
http://www.jarober.com
[hidden email]

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Carl Gundel-2

Re: TAG-MemoryUsage (Was: base.im questions)

In reply to this post by Travis Griggs-4

Oh, that's pretty slick. You're hired! :-)

Did we ever find out why that drawing code fails on the Mac? You
showed that to the Sark, right?

-Carl

On Tue, Jan 17, 2012 at 12:59 PM, Travis Griggs <[hidden email]> wrote:

> On Jan 16, 2012, at 2:12 PM, Kooyman, Les wrote:
>
> I agree with the overhead comments related to internationalization.
>
> In the follow-on release to 7.9, whatever its name may be, I'd like to move
> to dynamic allocation of locale instances. This would cut down on this
> memory pressure.
>
>
> I was curious just where the PrintFormatTokens were being held, and how much
> memory was in them and other things. So I wrote a little tool described
> here:
>
> http://screencast.com/t/exviTDoA5N7
>
> --
> Travis Griggs
> Objologist
> I multiply all time estimates by pi, to account for running around in
> circles.
>
>
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Kooyman, Les

Re: base.im questions

In reply to this post by Carl Gundel-2

Re: [vwnc] base.im questions

Probably you should consider keeping the installation methods for any locales you use. Under some circumstances you could fail on return from snapshot otherwise.

You might want to keep #install_en_US_POSIX_locale in LocaleLocalizationComponent as well.

From: [hidden email] on behalf of Carl Gundel
Sent: Tue 1/17/2012 8:09 AM
To: VWNC NC
Subject: Re: [vwnc] base.im questions

Okay, I wrote the following quick script a cleared 1.2MB from my
image. That's nice. Am I missing anything? If someone running a
German language PC starts up my image am I going to get a runtime
error?

| allLocales methods keys |
"remove locales"
allLocales := LocaleLocalizationComponent classPool at: #LocaleDictionary.
allLocales keys copy do: [ :k |
(k ~= #en_US and: [ k ~= #C ]) ifTrue: [ allLocales removeKey: k ]
].
"remove locale installation methods"
methods := LocaleLocalizationComponent class methodDictionary.
keys := methods keys asOrderedCollection select: [ :k | (k
indexOfString: 'install_') = 1 ].
keys remove: #install_en_US_Locale.
keys remove: #install_C_Locale.
keys do: [ :k | methods removeKey: k ].

-Carl

On Tue, Jan 17, 2012 at 10:16 AM, Carl Gundel <[hidden email]> wrote:

> Thanks. It's amazing there are so many of these. How do I remove
> them? Do I need a script or is there a tool for adding/removing
> these? If I do remove them, how do I know which ones I should keep?
> I know I'm going to want to keep the US related ones, and perhaps
> there are EU locales too?
>
> -Carl
>
> On Mon, Jan 16, 2012 at 3:58 PM, Travis Griggs <[hidden email]> wrote:
>>
>> On Jan 16, 2012, at 11:40 AM, James Robertson wrote:
>>
>>> I'd still say you're better off not using the RTP approach; trying to pull unused methods and classes (as opposed to whole packages) gains you very little at the cost of pretty high uncertainty.
>>>
>>> -- start with visual.im
>>> -- load your parcels
>>> -- use script like this to unload:
>>>
>>> (Store.Registry pundleNamed: #('PackageNameHere')) ifNotNil: [:pundle |
>>> pundle leafItems do: [:each | each markNotModified].
>>> [pundle markNotModified; unloadFromImage] on: [Warning do: [:ex | ex resume: true]].
>>>
>>> I'm sure there's an equivalent for images w/o Store loaded, but my builder starts from a Store image. The larger point is this: unloading packages and bundles removes large amounts of stuff; unloading classes and methods, not so much.
>>
>>
>> I concur with this. Build up.
>>
>> But I think, Carl's issue is that the foundation he's building up from seems a little bit of overkill. I wouldn't personally use RTP to try and address this problem. My hunch is that there's some rather large low hanging fruit that you can remove with a couple of workspace/script like expressions.
>>
>> My bet, personally, is along the same lines as Martin's comment. The primary addition to image size seems to have been collation and locale stuff. On my image for example, the following expression:
>>
>> ObjectMemory current numOopsNumBytesIn: PrintFormatToken allGeneralInstances
>>
>> indicates there are ~57000 instances chewing up 1.35 MB of memory. I'm guessing I don't need that many instances of PrintFormatTokens to do my every day job.
>>
>> --
>> Travis Griggs
>> Objologist
>> Simplicity is the Ultimate Sophistication -- Leonardo da Vinci
>>
>>
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Carl Gundel-2

Re: base.im questions

Thanks!

On Tue, Jan 17, 2012 at 1:36 PM, Kooyman, Les <[hidden email]> wrote:

> Probably you should consider keeping the installation methods for any
> locales you use. Under some circumstances you could fail on return from
> snapshot otherwise.
>
> You might want to keep #install_en_US_POSIX_locale in
> LocaleLocalizationComponent as well.
>
> ________________________________
> From: [hidden email] on behalf of Carl Gundel
> Sent: Tue 1/17/2012 8:09 AM
> To: VWNC NC
>
> Subject: Re: [vwnc] base.im questions
>
> Okay, I wrote the following quick script a cleared 1.2MB from my
> image. That's nice. Am I missing anything? If someone running a
> German language PC starts up my image am I going to get a runtime
> error?
>
> | allLocales methods keys |
> "remove locales"
> allLocales := LocaleLocalizationComponent classPool at: #LocaleDictionary.
> allLocales keys copy do: [ :k |
> (k ~= #en_US and: [ k ~= #C ]) ifTrue: [ allLocales removeKey: k ]
> ].
> "remove locale installation methods"
> methods := LocaleLocalizationComponent class methodDictionary.
> keys := methods keys asOrderedCollection select: [ :k | (k
> indexOfString: 'install_') = 1 ].
> keys remove: #install_en_US_Locale.
> keys remove: #install_C_Locale.
> keys do: [ :k | methods removeKey: k ].
>
> -Carl
>
> On Tue, Jan 17, 2012 at 10:16 AM, Carl Gundel <[hidden email]> wrote:
>> Thanks. It's amazing there are so many of these. How do I remove
>> them? Do I need a script or is there a tool for adding/removing
>> these? If I do remove them, how do I know which ones I should keep?
>> I know I'm going to want to keep the US related ones, and perhaps
>> there are EU locales too?
>>
>> -Carl
>>
>> On Mon, Jan 16, 2012 at 3:58 PM, Travis Griggs <[hidden email]>
>> wrote:
>>>
>>> On Jan 16, 2012, at 11:40 AM, James Robertson wrote:
>>>
>>>> I'd still say you're better off not using the RTP approach; trying to
>>>> pull unused methods and classes (as opposed to whole packages) gains you
>>>> very little at the cost of pretty high uncertainty.
>>>>
>>>> -- start with visual.im
>>>> -- load your parcels
>>>> -- use script like this to unload:
>>>>
>>>> (Store.Registry pundleNamed: #('PackageNameHere')) ifNotNil: [:pundle |
>>>> pundle leafItems do: [:each | each markNotModified].
>>>> [pundle markNotModified; unloadFromImage] on: [Warning do: [:ex |
>>>> ex resume: true]].
>>>>
>>>> I'm sure there's an equivalent for images w/o Store loaded, but my
>>>> builder starts from a Store image. The larger point is this: unloading
>>>> packages and bundles removes large amounts of stuff; unloading classes and
>>>> methods, not so much.
>>>
>>>
>>> I concur with this. Build up.
>>>
>>> But I think, Carl's issue is that the foundation he's building up from
>>> seems a little bit of overkill. I wouldn't personally use RTP to try and
>>> address this problem. My hunch is that there's some rather large low hanging
>>> fruit that you can remove with a couple of workspace/script like
>>> expressions.
>>>
>>> My bet, personally, is along the same lines as Martin's comment. The
>>> primary addition to image size seems to have been collation and locale
>>> stuff. On my image for example, the following expression:
>>>
>>> ObjectMemory current numOopsNumBytesIn: PrintFormatToken
>>> allGeneralInstances
>>>
>>> indicates there are ~57000 instances chewing up 1.35 MB of memory. I'm
>>> guessing I don't need that many instances of PrintFormatTokens to do my
>>> every day job.
>>>
>>> --
>>> Travis Griggs
>>> Objologist
>>> Simplicity is the Ultimate Sophistication -- Leonardo da Vinci
>>>
>>>
>>> _______________________________________________
>>> vwnc mailing list
>>> [hidden email]
>>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Reinout Heeck-2

Re: base.im questions

In reply to this post by Kooyman, Les

On 1/16/2012 11:12 PM, Kooyman, Les wrote:

Re: [vwnc] base.im questions

I agree with the overhead comments related to internationalization.

In the follow-on release to 7.9, whatever its name may be, I'd like to move to dynamic allocation of locale instances. This would cut down on this memory pressure.

Les

The below message suggests differently (flyweight implementation) , don't know whether it is still valid though:

-------- Original Message --------

[vwnc] [7.7.1 I] I18N PrintFormatToken needs flyweight implementation

Tue, 01 Feb 2011 12:48:30 +0100

PrintFormatToken allInstances size "30063"
PrintFormatToken allInstances asSet size "33"

(ObjectMemory current numOopsNumBytesIn: PrintFormatToken allInstances) 
last "721512"


This is static data (LocaleLocalizationComponent.LocaleDictionary), so 
here's opportunity to shave roughly 3/4 MB from our deployment image size...


Reinout
---------

--
Untitled Document

Reinout Heeck, Sr. Software Engineer

Soops - Specialists in Object Technology

Tel : +31 (0) 20 6222844
Fax : +31 (0) 20 6360827
Web: www.soops.nl

* Please consider the environment before printing this e-mail *

Dit e-mailbericht is alleen bestemd voor de geadresseerde(n). Gebruik door anderen is niet toegestaan. Indien u niet de geadresseerde(n) bent wordt u verzocht de verzender hiervan op de hoogte te stellen en het bericht te verwijderen. Door de elektronische verzending kunnen aan de inhoud van dit bericht geen rechten worden ontleend.

Soops B.V. is gevestigd te Amsterdam, Nederland, en is geregistreerd bij de Kamer van Koophandel onder nummer 33240368. Soops B.V. levert volgens de Fenit voorwaarden, gedeponeerd te Den Haag op 8 december 1994 onder nummer 1994/189.

This e-mail message is intended to be exclusively for the addressee. If you are not the intended recipient you are kindly requested not to make any use whatsoever of the contents and to notify the sender immediately by returning this e-mail message. No rights can be derived from this message.

Soops B.V. is a private limited liability company and has its seat at Amsterdam, The Netherlands and is registered with the Trade Registry of the Chamber of Commerce and Industry under number 33240368. Soops B.V. delivers according to the General Terms and Conditions of Business of Fenit, registered at The Hague, The Netherlands on December 8th, 1994, under number 1994/189.

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Kooyman, Les

Re: base.im questions

Re: [vwnc] base.im questions

Part of setting up a locale is setting up it's instance of TimestampPrintPolicy, which is where the PrintFormatTokens are held on to from. (There are also a few which comprise currency formats in NumberPrintPolicy.)

In the case of Legacy locales, all locales are resident in memory. There are, however, a very small number of locales.

In the case of CLDR locales, the LocaleLocalizationComponent instances are all in memory, but are not put together into instances of CompositeLocale until they are needed. I propose to extend that to the instances of LocaleLocalizationComponent, so that they are instantiated only on demand as well.

Once a locale is instantiated and initialized, it would be retained so as to not repeat the setup overhead. For the default or system locale, this would all happen at image startup so it is unlikely the setup of one locale would be noticed. Probably the C locale would be made always resident to ensure at least one usable locale in the case of some overall failure.

Applications that use many locales concurrently would incur a penalty for instantiation the first time a particular locale is called for.

I'll keep your earlier comments in mind when I address this.

From: [hidden email] on behalf of Reinout Heeck
Sent: Mon 1/23/2012 3:47 AM
To: [hidden email]
Subject: Re: [vwnc] base.im questions

On 1/16/2012 11:12 PM, Kooyman, Les wrote:

I agree with the overhead comments related to internationalization.

In the follow-on release to 7.9, whatever its name may be, I'd like to move to dynamic allocation of locale instances. This would cut down on this memory pressure.

Les

The below message suggests differently (flyweight implementation) , don't know whether it is still valid though:

-------- Original Message --------

[vwnc] [7.7.1 I] I18N PrintFormatToken needs flyweight implementation

Tue, 01 Feb 2011 12:48:30 +0100

PrintFormatToken allInstances size "30063"
PrintFormatToken allInstances asSet size "33"

(ObjectMemory current numOopsNumBytesIn: PrintFormatToken allInstances) 
last "721512"


This is static data (LocaleLocalizationComponent.LocaleDictionary), so 
here's opportunity to shave roughly 3/4 MB from our deployment image size...


Reinout
---------

--

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

soops_logo_120x47.gif (6K) Download Attachment