Ever growing image

Le 04/03/2018 à 11:13, Esteban Lorenzano a écrit :
> In our community, Pavel is the person that has the bigger experience
> on shrinking process images. I followed his work for years and I know
> the fragility of the process. Basically each new addition/removal was
> breaking it and needing dependency tracking, etc. Now is the opposite:
> you just add what you want, and you are sure the resulting image is a)
> healthy and b) identical when running.
>

Indeed it is fragile. And given the latest image size, not sure it could
lead to satisfactory results.

--
Dr. Geo
http://drgeo.eu

Hilaire

It is up to you to give a real try.
1) Did you look for REAL at our infrastructure? Because you can take
the bootstrapped image
and change our build scripts. It took us may years to produce the
loading scripts and this is the future.
In Pharo 7 we have the possibility to have a core and several images.

2) Now I see you complaining but you can also help we never rejected a
fix that make the system
more modular. And we worked a lot to make it modular and IT IS MUCH
MORE MODULAR.
Now we do not have the energy to invest in building different image
based on the core. So if you that
need to not do it then why should we?

3) We will start a new reduction of the bootstrap core with Carolina
PhD and you see
we will not keep it for us (while we could). We want to have Pharo
running on IoT devices.

4) We are starting to remove duplicated tools: Nautilus will go out,
EyeInspector too. Old text model and widgets.
Now again I personnally do not really care if Pharo has one or two mb more.

5) even on my old iPhone 53 mb is not that large. But I want a much
smaller core and we will get there.

6) You see we introduce Athens in particular for Moose and DrGeo.
Personnally I do not use at all Athens-Cairo.
So this is funny how people react.

Stef



On Sat, Mar 3, 2018 at 3:21 PM, Hilaire <[hidden email]> wrote:

>
>
> It looks like this topic comes and goes since 20 years: new features added,
> image growth then/or it becomes hard to unload ; on Squeak then now with
> Pharo at a much higher rate.
>
> We all know about the right approach (GNU Smalltalk, CUIS Smalltalk),


I will not comment on that and suggest you to code with such systems.

but it
> is not the road chosen with Pharo and it is becoming extremely uninteresting
> for me to code on DrGeo now.

Fair. Do not do it.

> So far I fell ashame to ship end user
> application with a 50MB DrGeo image, knowing only 10% has purpose related to
> DrGeo domain. When situation improves, motivation may come back. For this
> same reason and other, I turned to Python for a planed programming course to
> mid high school students.

Good luck. I was teaching python to my son and I have to explain to
him that a set cannot
contain a set or any object, that length is not a message and plenty
of other nice things.



>
> How the image is considered in the Pharo team?  A kind of sacred place,
> where stuff are added following a few guidelines (you will not duplicate
> features but replace/imprive/whatever -- aka funny shortcut mess, you will
> make your code unloadable, etc...) or just like a forum where teams add
> feature according to projects or vision.

Hilaire if you believe that we are a bunch a morons this is up to you.
Now we do pay a lot of attention to our system but our ressources are scarce
and we try to satisfy many different concerns. And we do not get that much
help from people.
You see doing something real and of good quality takes a lot of time and energy.
And I think that we are successful. It does not mean that we cannot improve.


> Sorry to discuss these topics again, really don't want but still need to
> share a few thoughts.

I would prefer that you share some code and Pull Requests that always the same
complains. But you can.

Stef

But the building process is working and we can incrementally create
custom images.
And this is what we will use in the future but we will not build the
image that suits everybody needs.

Stef

On Sun, Mar 4, 2018 at 1:08 PM, Hilaire <[hidden email]> wrote:

Pharo Smalltalk Users mailing list wrote
> Estaban,
> I guess there are some advantages to shrink the image (as compared to
> bootstrapping from a minimal image)

Also, they are not mutually exclusive. While bootstrapping seems a clear
choice for building images. A tool could then scan an image (e.g. for
deployment) and pare it down to just what's needed. IIRC Self might have
done some work in this direction…



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

Pharo7 still has ImageCleaner. It needs some work (even for Pharo6 it would
be nice to do a pass).

Vincent (I think) did already one improvement to it in Pharo7 (simplified test
unloading).

I did find some time to do an experiment with Pharo7, and if you remove all
tests and documentation with ImageCleaner (after a small fix, will commit that),
the image shrinks fro 38 to 35 (I had expected a bit more).

*But*: just opening it and running the “Do Image Cleanup”, then save and quit,
the image is again close to 38MB, which is *very* strange!

So for size there is something strange going on. We need to have a closer look.

I have put on my TODO:
        - do a pass on ImageCleaner Pharo6 and Pharo7
        - start to find duplicated code to not load anymore in the bootstrap (e.g.
            old compiler, inspector…).

        Marcus

Yes and it will be a lot more than that.
We have a better core.
Then we will continue to make it smaller.
It means that with a bunch of BaselineOf you should be able to create
the Pharo that suits your needs.
Now we will do it for people :).
I think that copying the BaselineOf and removing some unless or not my
focus package is something that people
can afford.
And if by accident something is not modular enough we always accept
better modular code.
So it will improve but again given a specific objectives people such
hilaire should do it.

Stef


On Sun, Mar 4, 2018 at 4:12 PM, Ben Coman <[hidden email]> wrote:

Thanks Markus.

I already saw an improvement when building drgeo against latest 32bits
P7: drgeo image down to 39.9MB...

I use image cleaner plus some packages removal of my own[1], left over
of the cleaning I was doing on P3. Many of my package cleaning of P4
break on P7. There is still place for improvement. I remembered doing 
it one by on P3, keeping the safe one, then excluding the one breaking
the image.

Now if there is a guide to try bootstrap without the need to hit the
head on the door, I am interested to read it.

Hilaire


[1]
https://bazaar.launchpad.net/~drgeo-developers/drgeo/trunk/files/506/src/DrGeoII-Installer

Le 05/03/2018 à 08:39, Marcus Denker a écrit :
> I have put on my TODO:
> - do a pass on ImageCleaner Pharo6 and Pharo7
> - start to find duplicated code to not load anymore in the bootstrap (e.g.
>              old compiler, inspector…).

--
Dr. Geo
http://drgeo.eu

Marcus Denker-4 wrote
> (I had expected a bit more).

I think I definitely read somewhere that there was a system that could
analyze the messages that were actually sent and remove everything else...
fun to think about the possibilities!



-----
Cheers,
Sean
--
Sent from: http://forum.world.st/Pharo-Smalltalk-Users-f1310670.html

yes, Mariano Peck made a PhD on that (when we were exploring “how to get smaller images” possibilities).
We finally went for bootstrap, but we made an analysis on all possibilities around.

Maybe he can point to it…

Esteban

Maybe this:

https://rmod.inria.fr/web/publications/bib?query=Poli17c&display=abstract

Guillermo Polito, Luc Fabresse, Noury Bouraqadi, and Stéphane Ducasse. Run-Fail-Grow: Creating Tailored Object-Oriented Runtimes

Producing a small deployment version of an application is a challenge because static abstractions such as packages cannot anticipate the use of their parts at runtime. Thus, an application often occupies more memory than actually needed. Tailoring is one of the main solutions to this problem i.e., extracting used code units such as classes and methods of an application. However, existing tailoring techniques are mostly based on static type annotations. These techniques cannot efficiently tailor applications in all their extent (e.g., runtime object graphs and metadata) nor be used in the context of dynamically-typed languages. We propose a run-fail-grow technique to tailor applications using their runtime execution. Run-fail-grow launches (a) a reference application containing the original application to tailor and (b) a nurtured application containing only a seed with a minimal set of code units the user wants to ensure in the final application. The nurtured application is executed, failing when it founds missing objects, classes or methods. On failure, the necessary elements are installed into the nurtured application from the reference one, and the execution resumes. The nurtured application is executed until it finishes, or until the developer explicitly finishes it, for example in the case of a web application. resulting in an object memory (i.e., a heap) with only objects, classes and methods required to execute the application. To validate our approach we implemented a tool based on Virtual Machine modifications, namely Tornado. Tornado succeeds to create very small memory footprint versions of applications e.g., a simple object-oriented heap of 11kb. We show how tailoring works on application code, base and third-party libraries even supporting human interaction with user G. interfaces. These experiments show memory savings ranging from 95\% to 99\%.

>
> yes, Mariano Peck made a PhD on that (when we were exploring “how to get smaller images” possibilities).
> We finally went for bootstrap, but we made an analysis on all possibilities around.
>

Mariano was doing swapping out…

Mariano Martinez Peck, Noury Bouraqadi, Marcus Denker, Stéphane Ducasse, and Luc Fabresse. Marea: An Efficient Application-Level Object Graph Swapper.

Abstract
Abstract During the execution of object-oriented applications, several millions of objects are created, used and then collected if they are not referenced. Problems appear when objects are unused but cannot be garbage-collected because they are still referenced from other objects. This is an issue because those objects waste primary memory and applications use more primary memory than they actually need. We claim that relying on the operating system's (OS) virtual memory is not always enough since it cannot take into account the domain and structure of applications. At the same time, applications have no easy way to parametrize nor cooperate with memory management. In this paper, we present Marea, an efficient application-level object graph swapper for object-oriented programming languages. Its main goal is to offer the programmer a novel solution to handle application-level memory. Developers can instruct our system to release primary memory by swapping out unused yet referenced objects to secondary memory. Our approach has been qualitatively and quantitatively validated. Our experiments and benchmarks on real-world applications show that Marea can reduce the memory footprint between 23\% and 36\%.

Hi hilaire

I think that Pavel sent around a link with the bootstrapped image.
For you, you do not want to play changing the bootstrap. What you want
is to change and load your own customized
baselines on top of the bootstrapped image (probably the one with
compiler and Monti/metacello loaded).
I do not know where they are (running now out of my office).
But the pharo70 build is based on that.
Then with carolina we will start building tools to build smaller
bootstrap and smaller VMs for IoT.

Stef


On Mon, Mar 5, 2018 at 6:33 PM, Hilaire <[hidden email]> wrote:

A "How to" as we used to have at the linux early time will be very
helpful to get started.

When I tried on this last summer I lost myself and gave up.

Hilaire


Le 06/03/2018 à 11:55, Guillermo Polito a écrit :
> For a more detailed list, you can browse BaselineOfPharoBootstrap.
>
> If you want an image that requires packages that are not there (like
> morphic), you should install them on top and specify those as
> dependencies of your project. That is what we do with the Pharo IDE.
> Take a look at BaselineOfIDE. Again, this could be enhanced and we
> will greatfully accept any contribution, or discuss on skype about the
> details or help people to get into this.

--
Dr. Geo
http://drgeo.eu

Le 06/03/2018 à 11:55, Guillermo Polito a écrit :
>
> In any case, I only wanted to say that it is unfair to say that Pharo
> is big when "you only load your packages".
> And I feel that's not true, if you're using the graphical environment,
> you need display, bitblt, morphic, widgets.
> As soon as you need to dynamically run scripts you need the compiler
> and parsing machinery.
> As soon as you write any program you mostly need many core libraries
> (just think about collections).

It is not unfair, it is just a reality. First version of DrGeo image on
Pharo, including display, bitblt, morphic, widgets, compiler, parser was
bellow 10MB (including DrGeo code)[1].
Now the question that matter for me, can I get back toward this fair
size or close to?

I don't buy on the argument we have plenty of place, who cares. It is
wrong. It is with this kind of argument we will lead to the exhaustion
of the planet Earth resources[2].

Hilaire


[1] https://gforge.inria.fr/frs/download.php/file/28307/DrGeo.app-11.03.zip
[2] https://en.wikipedia.org/wiki/Rebound_effect_(conservation)

--
Dr. Geo
http://drgeo.eu

Le 12/03/2018 à 10:16, Guillermo Polito a écrit :
> That said, I would also like to have a smaller and more malleable
> system, in that I agree with you, but that takes time and work, and I
> see few people that is diving in the insides of Pharo to help get us
> something in that direction. Not because they don't care but because
> it is difficult and time consuming.

Which direction should I follow after building a boostrap image
following these instructions
https://github.com/pharo-project/pharo#bootstrapping-pharo-from-sources ?

--
Dr. Geo
http://drgeo.eu