MooseDefaultState

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

MooseDefaultState

CyrilFerlicot
Hi,

Can someone explain the real interest to get a dictionary for properties
and a dictionary for queries in the MooseDefaultState?

Is there a downside to merge both?

I did some tests by creating a SYNState using a single dictionary for
both properties and queries. Here is the memory impact:

MooseDefaultState
-------------------

Image without model : 263Mo
Image with model : 605Mo
Image with model and default synectique caches : 650Mo
Image with model, default caches and after using the application : 792Mo

SYNState
-------------------

Image without model : 263Mo
Image with model : 589Mo
Image with model and default synectique caches : 575Mo (There was
probably a full GC that garbaged the temporary object of the import)
Image with model, default caches and after using the application : 706Mo

We see a clear gain of memory. (I did exactly the same interactions in
both try)

We might be interested by this at Synectique. This is why I would like
to know the reasons to have two different dictionaries since there is no
class comment. If the difference will not impact our tools I'll make the
default state configurable and I will introduce this SYNState renaming
it "MooseMemoryEfficientState" if it is fine with everyone.

Thank you in advance for the infos.

--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stéphane Ducasse

On 23 Sep 2017, at 20:03, Cyril Ferlicot D. <[hidden email]> wrote:

Hi,

Can someone explain the real interest to get a dictionary for properties
and a dictionary for queries in the MooseDefaultState?

cyril can you explain to me what is for queries?
Did you check if there are smalldictionary?

Is there a downside to merge both?

I did some tests by creating a SYNState using a single dictionary for
both properties and queries. Here is the memory impact:

MooseDefaultState
-------------------

Image without model : 263Mo
Just loading the code takes that amount of space? 
Are you sure that we use all that?

Image with model : 605Mo
Image with model and default synectique caches : 650Mo
What is a synectique caches?

Image with model, default caches and after using the application : 792Mo

SYNState
-------------------

Image without model : 263Mo
Image with model : 589Mo
so this is 6 mo so 1%? 

Image with model and default synectique caches : 575Mo (There was
probably a full GC that garbaged the temporary object of the import)
Image with model, default caches and after using the application : 706Mo

We see a clear gain of memory. (I did exactly the same interactions in
both try)

We might be interested by this at Synectique. This is why I would like
to know the reasons to have two different dictionaries since there is no
class comment. If the difference will not impact our tools I'll make the
default state configurable and I will introduce this SYNState renaming
it "MooseMemoryEfficientState" if it is fine with everyone.

Thank you in advance for the infos.

Yes I would like to know. 
Do you know if the keys of “queries” clash with the keys of properties?

--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

--------------------------------------------
Stéphane Ducasse
03 59 35 87 52
Assistant: Julie Jonas 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stephan Eggermont-3
In reply to this post by CyrilFerlicot
On 23-09-17 20:03, Cyril Ferlicot D. wrote:
> Can someone explain the real interest to get a dictionary for
> properties and a dictionary for queries in the MooseDefaultState?

Properties should probably not be a dictionary.  There are way too many
MooseEntities that share the same entries

Stephan
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

CyrilFerlicot
In reply to this post by Stéphane Ducasse
Le 23/09/2017 à 21:01, Stéphane Ducasse a écrit :
>
> I was thinking more about Moose, Seaside….. 
> Because Pharo is not that large. Even if we can remove things. 
> And the bootstrap is not really related to that. 
>

From Moose we get only FAME and FAMIX, but we can restrict a little more
FAMIX. I would like to make groups depending of the language because for
now we load everything for every language. Not loading the Smalltalk
package into non smalltalk technologies would help us to reduce the time
of queries.

For Seaside I opened an issue on Seaside issue tracker some weeks ago to
be able to load seaside without the tests. It will probably be done for
Seaside 3.3.

Then I think we use pretty much everything that we load on top of Pharo.

>
> ok do you know when the queries dict is used?
> Because this is the first time I hear about it. 
>

Here is the list of usages I see:

- MooseModel use it to save bookmarks
- FAMIXContainerEntity use it in #classes
- FAMIXNamespace use it for #methods
- FAMIXPackage use it for #classes, #concernedClassesCollection,
#extendedClasses, #extensionClasses, #extensionMethods, #localMethods
- Other FAMIXXXX use it that way
- MooseAbstractGroup use it in the #allModelXXX methods and
#allWithSubTypesOf:


>
> --------------------------------------------
> Stéphane Ducasse
> http://stephane.ducasse.free.fr
> http://www.synectique.eu / http://www.pharo.org 
> 03 59 35 87 52
> Assistant: Julie Jonas 
> FAX 03 59 57 78 50
> TEL 03 59 35 86 16
> S. Ducasse - Inria
> 40, avenue Halley, 
> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
> Villeneuve d'Ascq 59650
> France
>
--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stephan Eggermont-3


On 23-09-17 21:54, Cyril Ferlicot D. wrote:
> For Seaside I opened an issue on Seaside issue tracker some weeks ago to
> be able to load seaside without the tests. It will probably be done
> for Seaside 3.3. Then I think we use pretty much everything that we
> load on top of Pharo.

Is there an issue with that? AFAIK we were always able to load Seaside
without tests.
I can't find an open issue

Stephan
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

CyrilFerlicot
Le 23/09/2017 à 22:35, stephan a écrit :
> Is there an issue with that? AFAIK we were always able to load Seaside
> without tests.
> I can't find an open issue
>

The issue is already closed :

https://github.com/SeasideSt/Seaside/issues/942

> Stephan
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev


--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stéphane Ducasse
In reply to this post by Stephan Eggermont-3
Why that. All the metrics are specific to one entity.

On 23 Sep 2017, at 21:41, stephan <[hidden email]> wrote:

On 23-09-17 20:03, Cyril Ferlicot D. wrote:
Can someone explain the real interest to get a dictionary for properties and a dictionary for queries in the MooseDefaultState?

Properties should probably not be a dictionary.  There are way too many MooseEntities that share the same entries

Stephan
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

--------------------------------------------
Stéphane Ducasse
03 59 35 87 52
Assistant: Julie Jonas 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stéphane Ducasse
In reply to this post by CyrilFerlicot

On 23 Sep 2017, at 21:54, Cyril Ferlicot D. <[hidden email]> wrote:

Le 23/09/2017 à 21:01, Stéphane Ducasse a écrit :

I was thinking more about Moose, Seaside….. 
Because Pharo is not that large. Even if we can remove things. 
And the bootstrap is not really related to that. 


From Moose we get only FAME and FAMIX, but we can restrict a little more
FAMIX. I would like to make groups depending of the language because for
now we load everything for every language. Not loading the Smalltalk
package into non smalltalk technologies would help us to reduce the time
of queries.

Indeed. 
For Seaside I opened an issue on Seaside issue tracker some weeks ago to
be able to load seaside without the tests. It will probably be done for
Seaside 3.3.

I do not get why you get then a so large image of 260 Mo?
The default Pharo moose image fully charged with a model with 120000 classes that I created from undertsand
is 93 Mo
There is something strange.


Then I think we use pretty much everything that we load on top of Pharo.


ok do you know when the queries dict is used?
Because this is the first time I hear about it. 


Here is the list of usages I see:

- MooseModel use it to save bookmarks
- FAMIXContainerEntity use it in #classes
- FAMIXNamespace use it for #methods
- FAMIXPackage use it for #classes, #concernedClassesCollection,
#extendedClasses, #extensionClasses, #extensionMethods, #localMethods
- Other FAMIXXXX use it that way
- MooseAbstractGroup use it in the #allModelXXX methods and
#allWithSubTypesOf:



There are probably two dictionary so that you can show only properties in the tools. 
and you do not need to know what are the cached queries so if you merge the two 
then it means that you should keep a way to distinguish between queries and others. 

Now about the properties may be we should encode them differently. 
I’m writing an email about how to reuse metrics computation and I will send it. 


--------------------------------------------
Stéphane Ducasse
http://stephane.ducasse.free.fr
http://www.synectique.eu / http://www.pharo.org 
03 59 35 87 52
Assistant: Julie Jonas 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France


-- 
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France

--------------------------------------------
Stéphane Ducasse
03 59 35 87 52
Assistant: Julie Jonas 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stéphane Ducasse
In reply to this post by CyrilFerlicot
When I take the default moose 60 image I get 63,3 m
So I do not understand if from moose you only use 
Famix and Fame 
and seaside
how is it possible that you end up with an image without model at 263.

Did you flush the MC caches or history?

Stef
On 23 Sep 2017, at 20:03, Cyril Ferlicot D. <[hidden email]> wrote:

Hi,

Can someone explain the real interest to get a dictionary for properties
and a dictionary for queries in the MooseDefaultState?

Is there a downside to merge both?

I did some tests by creating a SYNState using a single dictionary for
both properties and queries. Here is the memory impact:

MooseDefaultState
-------------------

Image without model : 263Mo
Image with model : 605Mo
Image with model and default synectique caches : 650Mo
Image with model, default caches and after using the application : 792Mo

SYNState
-------------------

Image without model : 263Mo
Image with model : 589Mo
Image with model and default synectique caches : 575Mo (There was
probably a full GC that garbaged the temporary object of the import)
Image with model, default caches and after using the application : 706Mo

We see a clear gain of memory. (I did exactly the same interactions in
both try)

We might be interested by this at Synectique. This is why I would like
to know the reasons to have two different dictionaries since there is no
class comment. If the difference will not impact our tools I'll make the
default state configurable and I will introduce this SYNState renaming
it "MooseMemoryEfficientState" if it is fine with everyone.

Thank you in advance for the infos.

--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

--------------------------------------------
Stéphane Ducasse
03 59 35 87 52
Assistant: Julie Jonas 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

CyrilFerlicot
Le 24/09/2017 à 09:09, Stéphane Ducasse a écrit :
> When I take the default moose 60 image I get 63,3 m
> So I do not understand if from moose you only use 
> Famix and Fame 
> and seaside
> how is it possible that you end up with an image without model at 263.
>
> Did you flush the MC caches or history?
>

No, I did not took the time to check this kind of things. It would be
really cool if there was a package with help for the deployment or a
good documentation because I don't have time to check everything by myself.

> Stef
>
> --------------------------------------------
> Stéphane Ducasse
> http://stephane.ducasse.free.fr
> http://www.synectique.eu / http://www.pharo.org 
> 03 59 35 87 52
> Assistant: Julie Jonas 
> FAX 03 59 57 78 50
> TEL 03 59 35 86 16
> S. Ducasse - Inria
> 40, avenue Halley, 
> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
> Villeneuve d'Ascq 59650
> France
>
>
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev
>

--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

CyrilFerlicot
In reply to this post by Stéphane Ducasse
Le 24/09/2017 à 09:06, Stéphane Ducasse a écrit :
>
>
> I do not get why you get then a so large image of 260 Mo?
> The default Pharo moose image fully charged with a model with 120000
> classes that I created from undertsand
> is 93 Mo
> There is something strange.
>

I'll open an issue on our bug tracker to check this.

>
> There are probably two dictionary so that you can show only properties
> in the tools.
> and you do not need to know what are the cached queries so if you merge
> the two 
> then it means that you should keep a way to distinguish between queries
> and others. 
>

I think this is not the reason because then you cannot display
properties that are not in cache if you just take the properties of the
MooseDefaultState.

In anycase if it is something like this then I can introduce a new state
since we do not care of what is a property and what is a query in the
tools. We have our own layer in top of FAMIX.

> Now about the properties may be we should encode them differently. 
> I’m writing an email about how to reuse metrics computation and I will
> send it. 
>

Ok

>
> --------------------------------------------
> Stéphane Ducasse
> http://stephane.ducasse.free.fr
> http://www.synectique.eu / http://www.pharo.org 
> 03 59 35 87 52
> Assistant: Julie Jonas 
> FAX 03 59 57 78 50
> TEL 03 59 35 86 16
> S. Ducasse - Inria
> 40, avenue Halley, 
> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
> Villeneuve d'Ascq 59650
> France
>

--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Tudor Girba-2
Hi,

The reason was to allow people to remove caches in one shot without removing basic properties. I used them a lot during development of traversals and queries.

However, we can probably remove them now and favor fast loading of a whole model instead.

Cheers,
Doru


> On Sep 24, 2017, at 10:56 AM, Cyril Ferlicot D. <[hidden email]> wrote:
>
> Le 24/09/2017 à 09:06, Stéphane Ducasse a écrit :
>>
>>
>> I do not get why you get then a so large image of 260 Mo?
>> The default Pharo moose image fully charged with a model with 120000
>> classes that I created from undertsand
>> is 93 Mo
>> There is something strange.
>>
>
> I'll open an issue on our bug tracker to check this.
>
>>
>> There are probably two dictionary so that you can show only properties
>> in the tools.
>> and you do not need to know what are the cached queries so if you merge
>> the two
>> then it means that you should keep a way to distinguish between queries
>> and others.
>>
>
> I think this is not the reason because then you cannot display
> properties that are not in cache if you just take the properties of the
> MooseDefaultState.
>
> In anycase if it is something like this then I can introduce a new state
> since we do not care of what is a property and what is a query in the
> tools. We have our own layer in top of FAMIX.
>
>> Now about the properties may be we should encode them differently.
>> I’m writing an email about how to reuse metrics computation and I will
>> send it.
>>
>
> Ok
>
>>
>> --------------------------------------------
>> Stéphane Ducasse
>> http://stephane.ducasse.free.fr
>> http://www.synectique.eu / http://www.pharo.org 
>> 03 59 35 87 52
>> Assistant: Julie Jonas
>> FAX 03 59 57 78 50
>> TEL 03 59 35 86 16
>> S. Ducasse - Inria
>> 40, avenue Halley,
>> Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
>> Villeneuve d'Ascq 59650
>> France
>>
>
>
> --
> Cyril Ferlicot
> https://ferlicot.fr
>
> http://www.synectique.eu
> 2 rue Jacques Prévert 01,
> 59650 Villeneuve d'ascq France
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

--
www.tudorgirba.com
www.feenk.com

“Software has no shape. Actually, it has no one shape. It has many."

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

CyrilFerlicot
Le 24/09/2017 à 11:54, Tudor Girba a écrit :
> Hi,
>
> The reason was to allow people to remove caches in one shot without removing basic properties. I used them a lot during development of traversals and queries.
>
> However, we can probably remove them now and favor fast loading of a whole model instead.
>
> Cheers,
> Doru
>

So I can create a MooseMemoryEfficientState that will be used by default
and then rename MooseDefaultState into MooseDevelopmentState. Then we
could let an option in the model to switch to the state we want.

If everyone agree I'll do the change.

Thank you for the infos.

> --
> www.tudorgirba.com
> www.feenk.com
>
> “Software has no shape. Actually, it has no one shape. It has many."
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev
>

--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stephan Eggermont-3
In reply to this post by Stéphane Ducasse


On 24-09-17 09:02, Stéphane Ducasse wrote:
> Why that. All the metrics are specific to one entity.
Because I tend to calculate the same metric for a lot of entries.
The lookup can be shared, avoiding the need to store the keys in every
entity.

Stephan
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

Stéphane Ducasse

On 24 Sep 2017, at 12:30, stephan <[hidden email]> wrote:



On 24-09-17 09:02, Stéphane Ducasse wrote:
Why that. All the metrics are specific to one entity.
Because I tend to calculate the same metric for a lot of entries.
The lookup can be shared, avoiding the need to store the keys in every entity.

I still do not get it. 
You need a key per entity if you want to have a metric value per entity.


Stephan
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

--------------------------------------------
Stéphane Ducasse
03 59 35 87 52
Assistant: Julie Jonas 
FAX 03 59 57 78 50
TEL 03 59 35 86 16
S. Ducasse - Inria
40, avenue Halley, 
Parc Scientifique de la Haute Borne, Bât.A, Park Plaza
Villeneuve d'Ascq 59650
France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: MooseDefaultState

CyrilFerlicot
In reply to this post by Stephan Eggermont-3
Le 24/09/2017 à 12:30, stephan a écrit :
>
>
> On 24-09-17 09:02, Stéphane Ducasse wrote:
>> Why that. All the metrics are specific to one entity.
> Because I tend to calculate the same metric for a lot of entries.
> The lookup can be shared, avoiding the need to store the keys in every
> entity.
>

If you have a solution to reduce the memory by sharing the keys without
losing in perf I am interested :)

> Stephan
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev


--
Cyril Ferlicot
https://ferlicot.fr

http://www.synectique.eu
2 rue Jacques Prévert 01,
59650 Villeneuve d'ascq France


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (836 bytes) Download Attachment