Smalltalk › Squeak › Squeak - Dev

Too much MC stuff left in image

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

13 messages Options

timrowledge

Too much MC stuff left in image

Even after use of doits such as MCFileBasedRepository flushAllCaches & MCDefinition clearInstances I have 9500 MCVersionInfo and similar numbers of other MC related objects in my image. And 990 SMAccounts? 800 SMPackage? 13000 UUIDs? That all seems a bit excessive somehow.

I know there was a conversation about chopping at least some of the MC stuff out (which I think is where the flushAllCaches came from) but it seems there is an awful lot left that could be removed as part of making a deployment image.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
How many of you believe in telekinesis? Raise my hands....

Bert Freudenberg

Re: Too much MC stuff left in image

On 14.11.2014, at 03:15, tim Rowledge <[hidden email]> wrote:

> Even after use of doits such as MCFileBasedRepository flushAllCaches & MCDefinition clearInstances I have 9500 MCVersionInfo and similar numbers of other MC related objects in my image. And 990 SMAccounts? 800 SMPackage? 13000 UUIDs? That all seems a bit excessive somehow.

SM* is SqueakMap, not Monticello.

But this sounds about the right order of magnitude: In image version 13680 that means we had 13680 previous versions, and Squeak-Version-ar.4662 tells us we removed 4662 versions so effectively there should be history for 13680-4662=9018 versions in the image. MCVersionInfo instanceCount is 9125, so that's 107 additional versions which accounts for merges (versions with multiple parents).

Not excessive at all: we chose to store history to the beginning of time in a dev environment, which is the Right Thing to do (and e.g. one reason why git won over cvs, and git even stores all source back to the beginning of time, not just meta data, which is awesome).

> I know there was a conversation about chopping at least some of the MC stuff out (which I think is where the flushAllCaches came from) but it seems there is an awful lot left that could be removed as part of making a deployment image.

For deployment, you want to get rid of the version stuff completely. That is like removing change sets - it loses information.

MCWorkingCopy allManagers do: [:wc | wc unregister]
Smalltalk garbageCollect.

MCVersionInfo instanceCount
==> 0
UUID instanceCount
==> 1

- Bert -

smime.p7s (5K) Download Attachment

Chris Muller-3

Re: Too much MC stuff left in image

In reply to this post by timrowledge

On Thu, Nov 13, 2014 at 8:15 PM, tim Rowledge <[hidden email]> wrote:
> Even after use of doits such as MCFileBasedRepository flushAllCaches & MCDefinition clearInstances I have 9500 MCVersionInfo and similar numbers of other MC related objects in my image. And 990 SMAccounts? 800 SMPackage? 13000 UUIDs? That all seems a bit excessive somehow.
>
> I know there was a conversation about chopping at least some of the MC stuff out (which I think is where the flushAllCaches came from) but it seems there is an awful lot left that could be removed as part of making a deployment image.

There is also 'flush cached versions and ancestry' on the MC
repository menu next to 'flush all caches', which goes back just 10
versions ago and trims the MCInfoProxy tree from there. Space
consumed by 95% of your MCInfoProxy instances is recovered, without
actually losing any data. The proxy will reload them dynamically if
asked (e.g., by you clicking on History button of a package).

I know Bert doesn't like it, but Smalltalk is an environment that's
supposed to support "debugging in production". There is a common
usage case in MC that causes the Proxy's to be prematurely agitated
unnecessarily, resulting in a delay while they're re-loaded, which
nobody liked (understandably). I never had time to investigate that
but if you do, that'd be really great.

The thing I don't care for about Berts solution is the hindrance to
debugging in production. Invariably it will happen by necessity.
After making a code-fix, someone might be tempted to re-add the
WorkingCopy and then save the package -- losing all the ancestry!

Bert Freudenberg

Re: Too much MC stuff left in image

On 14.11.2014, at 16:17, Chris Muller <[hidden email]> wrote:

> On Thu, Nov 13, 2014 at 8:15 PM, tim Rowledge <[hidden email]> wrote:
>> Even after use of doits such as MCFileBasedRepository flushAllCaches & MCDefinition clearInstances I have 9500 MCVersionInfo and similar numbers of other MC related objects in my image. And 990 SMAccounts? 800 SMPackage? 13000 UUIDs? That all seems a bit excessive somehow.
>>
>> I know there was a conversation about chopping at least some of the MC stuff out (which I think is where the flushAllCaches came from) but it seems there is an awful lot left that could be removed as part of making a deployment image.
>
> There is also 'flush cached versions and ancestry' on the MC
> repository menu next to 'flush all caches', which goes back just 10
> versions ago and trims the MCInfoProxy tree from there. Space
> consumed by 95% of your MCInfoProxy instances is recovered, without
> actually losing any data. The proxy will reload them dynamically if
> asked (e.g., by you clicking on History button of a package).
>
> I know Bert doesn't like it, but Smalltalk is an environment that's
> supposed to support "debugging in production". There is a common
> usage case in MC that causes the Proxy's to be prematurely agitated
> unnecessarily, resulting in a delay while they're re-loaded, which
> nobody liked (understandably). I never had time to investigate that
> but if you do, that'd be really great.
>
> The thing I don't care for about Berts solution is the hindrance to
> debugging in production. Invariably it will happen by necessity.
> After making a code-fix, someone might be tempted to re-add the
> WorkingCopy and then save the package -- losing all the ancestry!

Stripping and deployment was never discussed when you introduced this, and enabled it, by default, for all developers. That was one reason I was opposed.

For a deployed image the stubbed ancestry might be much better indeed - and for that use case I wouldn't even retain 10 levels of history, but strip it down to a single version info per package. Just the bare info needed to pull in the full ancestry. (I also would make re-instating the full version hierarchy an explicit action rather than a side effect, but we had that discussion already).

- Bert -

smime.p7s (5K) Download Attachment

Levente Uzonyi-2

Re: Too much MC stuff left in image

On Fri, 14 Nov 2014, Bert Freudenberg wrote:

> On 14.11.2014, at 16:17, Chris Muller <[hidden email]> wrote:
>
>> On Thu, Nov 13, 2014 at 8:15 PM, tim Rowledge <[hidden email]> wrote:
>>> Even after use of doits such as MCFileBasedRepository flushAllCaches & MCDefinition clearInstances I have 9500 MCVersionInfo and similar numbers of other MC related objects in my image. And 990 SMAccounts? 800 SMPackage? 13000 UUIDs? That all seems a bit excessive somehow.
>>>
>>> I know there was a conversation about chopping at least some of the MC stuff out (which I think is where the flushAllCaches came from) but it seems there is an awful lot left that could be removed as part of making a deployment image.
>>
>> There is also 'flush cached versions and ancestry' on the MC
>> repository menu next to 'flush all caches', which goes back just 10
>> versions ago and trims the MCInfoProxy tree from there. Space
>> consumed by 95% of your MCInfoProxy instances is recovered, without
>> actually losing any data. The proxy will reload them dynamically if
>> asked (e.g., by you clicking on History button of a package).
>>
>> I know Bert doesn't like it, but Smalltalk is an environment that's
>> supposed to support "debugging in production". There is a common
>> usage case in MC that causes the Proxy's to be prematurely agitated
>> unnecessarily, resulting in a delay while they're re-loaded, which
>> nobody liked (understandably). I never had time to investigate that
>> but if you do, that'd be really great.
>>
>> The thing I don't care for about Berts solution is the hindrance to
>> debugging in production. Invariably it will happen by necessity.
>> After making a code-fix, someone might be tempted to re-add the
>> WorkingCopy and then save the package -- losing all the ancestry!
>
> Stripping and deployment was never discussed when you introduced this, and enabled it, by default, for all developers. That was one reason I was opposed.
>
> For a deployed image the stubbed ancestry might be much better indeed - and for that use case I wouldn't even retain 10 levels of history, but strip it down to a single version info per package. Just the bare info needed to pull in the full ancestry. (I also would make re-instating the full version hierarchy an explicit action rather than a side effect, but we had that discussion already).

All we need is a new format for the VersionInfos, which can be read
sequentially (one by one). Instead of storing one huge smalltalk array,
we could store one array per version. Ancestors and stepChildren could
reference the versions by name and uuid, or maybe by an offset. I think
that would be so fast that we wouldn't want to keep the ancestry in memory
anymore.
All we would lose is the possiblity to create a snapshot without having
the ancestors around in a repository.
Another thing is that the snapshots would have to contain both the new and
the old format for compatibility with other smalltalks and older images.

Levente

>
> - Bert -
>
>

Chris Muller-3

Re: Too much MC stuff left in image

In reply to this post by Bert Freudenberg

> For a deployed image the stubbed ancestry might be much better indeed - and for that use case I wouldn't even retain 10 levels of history, but strip it down to a single version info per package. Just the bare info needed to pull in the full ancestry. (I also

The original plan for completing the circle was to allow the number of
levels to be specified as a preference that would not only affect
where it stubbed but also how far it bothered to look when doing
various MC operations.

Today, the only ancestry access in MCAncestry is pretty much just
allAncestorsDo: which ends up accessing the whole tree, again and
again and again, even though we almost NEVER need access all the way
back to version 1. If we changed senders of allAncestorsDo: to
"recentAncestorsDo:" (respecting the preference), we could ensure the
Proxy would almost never be hit for read operations. By defining the
size of what should be considered "recentHistory", like, something
between 10 and 100, the dev environments can be leaner too.
Production deployment script could set it to 0 to gain every last bit.

> would make re-instating the full version hierarchy an explicit action rather than a side effect, but we had that discussion already).

But then what should happen when the system got to the end of the
ancestry and reached the Proxy?

Bert Freudenberg

Re: Too much MC stuff left in image

On 14.11.2014, at 18:20, Chris Muller <[hidden email]> wrote:

>> would make re-instating the full version hierarchy an explicit action rather than a side effect, but we had that discussion already).
>
> But then what should happen when the system got to the end of the ancestry and reached the Proxy?

There wouldn't be any proxies. You should simply get an error. Plus possibly a button to "fetch all version history".

- Bert -

smime.p7s (5K) Download Attachment

Tobias Pape

Re: Too much MC stuff left in image

On 14.11.2014, at 18:27, Bert Freudenberg <[hidden email]> wrote:

> On 14.11.2014, at 18:20, Chris Muller <[hidden email]> wrote:
>
>>> would make re-instating the full version hierarchy an explicit action rather than a side effect, but we had that discussion already).
>>
>> But then what should happen when the system got to the end of the ancestry and reached the Proxy?
>
> There wouldn't be any proxies. You should simply get an error. Plus possibly a button to "fetch all version history".

+1

Best
-Tobias

signature.asc (1K) Download Attachment

timrowledge

Re: Too much MC stuff left in image

In reply to this post by Bert Freudenberg

On 14-11-2014, at 1:49 AM, Bert Freudenberg <[hidden email]> wrote:

> On 14.11.2014, at 03:15, tim Rowledge <[hidden email]> wrote:
>
>> Even after use of doits such as MCFileBasedRepository flushAllCaches & MCDefinition clearInstances I have 9500 MCVersionInfo and similar numbers of other MC related objects in my image. And 990 SMAccounts? 800 SMPackage? 13000 UUIDs? That all seems a bit excessive somehow.
>
> SM* is SqueakMap, not Monticello.

Well, yeah, I know that but it still came as a nasty surprise!

> For deployment, you want to get rid of the version stuff completely. That is like removing change sets - it loses information.
>
> MCWorkingCopy allManagers do: [:wc | wc unregister]
> Smalltalk garbageCollect.
>
> MCVersionInfo instanceCount
> ==> 0
> UUID instanceCount
> ==> 1

In my image I still had 3870 left, which caused some startlement. After taking a look around I noticed SMSqueakMap class>cleanUp: and then some senders of that. Turns out that
`Smalltalk cleanUp: true`
gets rid of a lot of stuff in a way I’d never heard of before. Along with the flushed cached versions and ancestry code Chris mentioned I can now get rid of a lot of stuff. My image is now down to 14Mb from 40, so that’s a help.

I wonder where 10Mb of Bitmaps is being used...

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: AII: Add Insult to Injury

timrowledge

Re: Too much MC stuff left in image

On 14-11-2014, at 10:44 AM, tim Rowledge <[hidden email]> wrote:
>
> I wonder where 10Mb of Bitmaps is being used…

Well, duh, a 1700@1300 Display will do that, mostly. When did screens get so frakkin huge? Why, when I was a lad we had 640@400 monochrome and liked it!

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
There are two ways to write error-free programs; only the third one works.

Chris Muller-3

Re: Too much MC stuff left in image

In reply to this post by Bert Freudenberg

On Fri, Nov 14, 2014 at 11:27 AM, Bert Freudenberg <[hidden email]> wrote:
> On 14.11.2014, at 18:20, Chris Muller <[hidden email]> wrote:
>
>>> would make re-instating the full version hierarchy an explicit action rather than a side effect, but we had that discussion already).
>>
>> But then what should happen when the system got to the end of the ancestry and reached the Proxy?
>
> There wouldn't be any proxies. You should simply get an error. Plus possibly a button to "fetch all version history".

Okay, let me phrase my question another way: When you lop-off the
ancestry, what are you going to replace it with? You can't just put
nil there because it'll treat that as the end of the ancestry.

You're gonna need SOME kind of, ahem.... "Placeholder" object so the
system will know how to react..

timrowledge

Re: Too much MC stuff left in image

In reply to this post by timrowledge

There’s so much fun to be had here for anyone wanting to play -

for example, why did I find 14 Paragraphs? Because the MVC ControlManager had a couple of windows still ‘live’. I honestly don’t have a clue what might have caused that; maybe a bug relating to exiting an mvc project a long time ago?

how about 10 EncoderForV3PlusClosures? Seemingly held on to by a supposedly weak dictionary within DebuggerMethodMap - shouldn’t repeated gcs have killed them?

Just run
`SpaceTally new printSpaceAnalysis`
sometime and pick something to beat up.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
"How many Carlos Wus does it take to change a lightbulb?”
"With an unlimited breeding licence, who needs lightbulbs?"

Bert Freudenberg

Re: Too much MC stuff left in image

In reply to this post by Chris Muller-3

On 14.11.2014, at 21:25, Chris Muller <[hidden email]> wrote:

>
>> On Fri, Nov 14, 2014 at 11:27 AM, Bert Freudenberg <[hidden email]> wrote:
>> On 14.11.2014, at 18:20, Chris Muller <[hidden email]> wrote:
>>
>>>> would make re-instating the full version hierarchy an explicit action rather than a side effect, but we had that discussion already).
>>>
>>> But then what should happen when the system got to the end of the ancestry and reached the Proxy?
>>
>> There wouldn't be any proxies. You should simply get an error. Plus possibly a button to "fetch all version history".
>
> Okay, let me phrase my question another way: When you lop-off the
> ancestry, what are you going to replace it with? You can't just put
> nil there because it'll treat that as the end of the ancestry.
>
> You're gonna need SOME kind of, ahem.... "Placeholder" object so the
> system will know how to react..

Correct. But not something pretending to be something it's clearly not.

Perhaps there shouldn't even be one level of version info, because that's not a true version info either. The working copy should instead hold onto a ... MCVersionInfoStub, perhaps? Which would just know a name, package, and uuid.

- Bert -

smime.p7s (8K) Download Attachment