A quick analyzis of the usage of instance variables in Moose

> On Nov 7, 2016, at 8:16 AM, Guillaume Larcheveque <[hidden email]> wrote:
>
> Hi Moose users,
>
> as you may know, at Synectique we experiment lots of issues about memory usage.
> That's why i had a look at the number of unused instance variables in one of our system loaded with a big model from one of our main customer.
>
> Total instance variables: 26.852.653
> Used instance variables: 17.393.938
> Empty collections: 5.023.508
> Recoverable instance variables: 14.482.223
>
> The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
>
> As you can see, we can save a lot of memory :-)
>
> Here is my (dirty) code to get that:
>
> countUsedInstanceVariableInForSubInstances: aClass
> | usedInstNbr instNbr emptyCollNbr |
> usedInstNbr := 0.
> instNbr := 0.
> emptyCollNbr := 0.
> aClass
> allSubInstancesDo: [ :anEntity |
> instNbr := instNbr + anEntity class allInstVarNames size.
> anEntity class allInstVarNames
> doWithIndex: [ :e :i |
> (anEntity instVarAt: i)
> ifNotNil: [ :content |
> ([
> content isEmpty.
> emptyCollNbr := emptyCollNbr + 1.
> false ]
> on: MessageNotUnderstood
> do: [ false ])
> ifFalse: [ usedInstNbr := usedInstNbr + 1 ] ] ] ].
> ^ {('Used instance variables' -> usedInstNbr).
> ('empty collections' -> emptyCollNbr).
> ('Recoverable instance variables' -> (instNbr - (usedInstNbr - emptyCollNbr))).
> ('Total instance variables' -> instNbr)}
>
> --
> Guillaume Larcheveque
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

> On Mar 13, 2017, at 9:53 AM, Nicolas Anquetil <[hidden email]> wrote:
>
>
> Reviving this old mail from Guillaume.
>
> I did an analysis on a huge model (>2M entities) we have:
> - .mse file: 372.5 MB
> - .image file without the model: 86.9 MB
> - image with model: 692.8 MB (thus apparently 605,9 MB used by the model)
> Then I did a small experiment: I removed the four instance variables of FAMIXSourcedEntity (sourceAnchor, sourceLanguage, comments, declaredSourceLanguage) and replaced them by variables in the privateState
>
> for example:
>
> FAMIXSourcedEntity>>comments
>     "Returns the list of comments defined for the entity."
>    
>     <MSEProperty: #comments type: #FAMIXComment opposite: #container> <multivalued> <derived>
>     <MSEComment: 'list of comments defined in the entity'>
>    
>     ^comments
> becomes
> FAMIXSourcedEntity>>comments
>     "Returns the list of comments defined for the entity."
>    
>     <MSEProperty: #comments type: #FAMIXComment opposite: #container> <multivalued> <derived>
>     <MSEComment: 'list of comments defined in the entity'>
>     "^ comments"
>     ^ self privateState cacheAt: #comments ifAbsentPut: nil
>
> As a result, the .image file is 673.3 MB (thus apparently 586,4 MB used by the model)
> Not huge, but almost 20 MB saved, 3% of the initial memory
>
> nicolas
>
> On 10/11/2016 20:49, stepharo wrote:
>> Thanks
>>
>> A task of Pavel will be to see how slots usage can reduce that space lost.
>> For the empty collection having a kind of
>>     - sharing an #() in a classVar
>>     - creating a new collection on use could be a solution.
>>
>> Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
>>> Hi Moose users,
>>>
>>> as you may know, at Synectique we experiment lots of issues about memory usage.
>>> That's why i had a look at the number of unused instance variables in one of our system loaded with a big model from one of our main customer.
>>>
>>> Total instance variables: 26.852.653
>>> Used instance variables: 17.393.938
>>> Empty collections: 5.023.508
>>> Recoverable instance variables: 14.482.223
>>>
>>> The recoverable instance variables are those with nil or an empty collection (or MultiValueLink)
>>>
>>> As you can see, we can save a lot of memory :-)
>>>
>>> Here is my (dirty) code to get that:
>>>
>>> countUsedInstanceVariableInForSubInstances: aClass
>>> | usedInstNbr instNbr emptyCollNbr |
>>> usedInstNbr := 0.
>>> instNbr := 0.
>>> emptyCollNbr := 0.
>>> aClass
>>> allSubInstancesDo: [ :anEntity |
>>> instNbr := instNbr + anEntity class allInstVarNames size.
>>> anEntity class allInstVarNames
>>> doWithIndex: [ :e :i |
>>> (anEntity instVarAt: i)
>>> ifNotNil: [ :content |
>>> ([
>>> content isEmpty.
>>> emptyCollNbr := emptyCollNbr + 1.
>>> false ]
>>> on: MessageNotUnderstood
>>> do: [ false ])
>>> ifFalse: [ usedInstNbr := usedInstNbr + 1 ] ] ] ].
>>> ^ {('Used instance variables' -> usedInstNbr).
>>> ('empty collections' -> emptyCollNbr).
>>> ('Recoverable instance variables' -> (instNbr - (usedInstNbr - emptyCollNbr))).
>>> ('Total instance variables' -> instNbr)}
>>>
>>> --
>>> Guillaume Larcheveque
>>>
>>>
>>>
>>> _______________________________________________
>>> Moose-dev mailing list
>>>
>>> [hidden email]
>>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>>
>>
>>
>> _______________________________________________
>> Moose-dev mailing list
>>
>> [hidden email]
>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>
> --
> Nicolas Anquetil -- MCF (HDR)
> Project-Team RMod
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

> -----Message d'origine-----
> De : Moose-dev [mailto:[hidden email]] De la part de
> Alexandre Bergel
> Envoyé : lundi 13 mars 2017 23:08
> À : Moose-related development
> Cc : Philippe Valenza; [hidden email]
> Objet : [Moose-dev] Re: A quick analyzis of the usage of instance variables in
> Moose
>
> Wow!
>
> Alexandre
>
> > On Mar 13, 2017, at 9:53 AM, Nicolas Anquetil <[hidden email]>
> wrote:
> >
> >
> > Reviving this old mail from Guillaume.
> >
> > I did an analysis on a huge model (>2M entities) we have:
> > - .mse file: 372.5 MB
> > - .image file without the model: 86.9 MB
> > - image with model: 692.8 MB (thus apparently 605,9 MB used by the
> > model) Then I did a small experiment: I removed the four instance
> > variables of FAMIXSourcedEntity (sourceAnchor, sourceLanguage,
> > comments, declaredSourceLanguage) and replaced them by variables in
> > the privateState
> >
> > for example:
> >
> > FAMIXSourcedEntity>>comments
> >     "Returns the list of comments defined for the entity."
> >
> >     <MSEProperty: #comments type: #FAMIXComment opposite:
> #container> <multivalued> <derived>
> >     <MSEComment: 'list of comments defined in the entity'>
> >
> >     ^comments
> > becomes
> > FAMIXSourcedEntity>>comments
> >     "Returns the list of comments defined for the entity."
> >
> >     <MSEProperty: #comments type: #FAMIXComment opposite:
> #container> <multivalued> <derived>
> >     <MSEComment: 'list of comments defined in the entity'>
> >     "^ comments"
> >     ^ self privateState cacheAt: #comments ifAbsentPut: nil
> >
> > As a result, the .image file is 673.3 MB (thus apparently 586,4 MB
> > used by the model) Not huge, but almost 20 MB saved, 3% of the initial
> > memory
> >
> > nicolas
> >
> > On 10/11/2016 20:49, stepharo wrote:
> >> Thanks
> >>
> >> A task of Pavel will be to see how slots usage can reduce that space lost.
> >> For the empty collection having a kind of
> >>     - sharing an #() in a classVar
> >>     - creating a new collection on use could be a solution.
> >>
> >> Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
> >>> Hi Moose users,
> >>>
> >>> as you may know, at Synectique we experiment lots of issues about
> memory usage.
> >>> That's why i had a look at the number of unused instance variables in
> one of our system loaded with a big model from one of our main customer.
> >>>
> >>> Total instance variables: 26.852.653 Used instance variables:
> >>> 17.393.938 Empty collections: 5.023.508 Recoverable instance
> >>> variables: 14.482.223
> >>>
> >>> The recoverable instance variables are those with nil or an empty
> >>> collection (or MultiValueLink)
> >>>
> >>> As you can see, we can save a lot of memory :-)
> >>>
> >>> Here is my (dirty) code to get that:
> >>>
> >>> countUsedInstanceVariableInForSubInstances: aClass
> >>>   | usedInstNbr instNbr emptyCollNbr |
> >>>   usedInstNbr := 0.
> >>>   instNbr := 0.
> >>>   emptyCollNbr := 0.
> >>>   aClass
> >>>           allSubInstancesDo: [ :anEntity |
> >>>                   instNbr := instNbr + anEntity class allInstVarNames
> size.
> >>>                   anEntity class allInstVarNames
> >>>                           doWithIndex: [ :e :i |
> >>>                                   (anEntity instVarAt: i)
> >>>                                           ifNotNil: [ :content |
> >>>                                                   ([
> >>>                                                   content isEmpty.
> >>>                                                   emptyCollNbr :=
> emptyCollNbr + 1.
> >>>                                                   false ]
> >>>                                                           on:
> MessageNotUnderstood
> >>>                                                           do: [ false ])
> >>>                                                           ifFalse: [
> usedInstNbr := usedInstNbr + 1 ] ] ] ].
> >>>   ^ {('Used instance variables' -> usedInstNbr).
> >>>   ('empty collections' -> emptyCollNbr).
> >>>   ('Recoverable instance variables' -> (instNbr - (usedInstNbr -
> emptyCollNbr))).
> >>>   ('Total instance variables' -> instNbr)}
> >>>
> >>> --
> >>> Guillaume Larcheveque
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Moose-dev mailing list
> >>>
> >>> [hidden email]
> >>> https://www.list.inf.unibe.ch/listinfo/moose-dev
> >>
> >>
> >>
> >> _______________________________________________
> >> Moose-dev mailing list
> >>
> >> [hidden email]
> >> https://www.list.inf.unibe.ch/listinfo/moose-dev
> >
> > --
> > Nicolas Anquetil -- MCF (HDR)
> > Project-Team RMod
> >
> > _______________________________________________
> > Moose-dev mailing list
> > [hidden email]
> > https://www.list.inf.unibe.ch/listinfo/moose-dev
>
> --
> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
> Alexandre Bergel  http://www.bergel.eu
> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
>
>
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

> Hi,
>
> Nice!
> But here we are saving memory. What about the performances for access the variables?
> Because, when we try to minimize one, we are always changing the other.
>
> Cheers,
> Vincent
>
>> -----Message d'origine-----
>> De : Moose-dev [mailto:[hidden email]] De la part de
>> Alexandre Bergel
>> Envoyé : lundi 13 mars 2017 23:08
>> À : Moose-related development
>> Cc : Philippe Valenza; [hidden email]
>> Objet : [Moose-dev] Re: A quick analyzis of the usage of instance variables in
>> Moose
>>
>> Wow!
>>
>> Alexandre
>>
>>> On Mar 13, 2017, at 9:53 AM, Nicolas Anquetil <[hidden email]>
>> wrote:
>>>
>>> Reviving this old mail from Guillaume.
>>>
>>> I did an analysis on a huge model (>2M entities) we have:
>>> - .mse file: 372.5 MB
>>> - .image file without the model: 86.9 MB
>>> - image with model: 692.8 MB (thus apparently 605,9 MB used by the
>>> model) Then I did a small experiment: I removed the four instance
>>> variables of FAMIXSourcedEntity (sourceAnchor, sourceLanguage,
>>> comments, declaredSourceLanguage) and replaced them by variables in
>>> the privateState
>>>
>>> for example:
>>>
>>> FAMIXSourcedEntity>>comments
>>>      "Returns the list of comments defined for the entity."
>>>
>>>      <MSEProperty: #comments type: #FAMIXComment opposite:
>> #container> <multivalued> <derived>
>>>      <MSEComment: 'list of comments defined in the entity'>
>>>
>>>      ^comments
>>> becomes
>>> FAMIXSourcedEntity>>comments
>>>      "Returns the list of comments defined for the entity."
>>>
>>>      <MSEProperty: #comments type: #FAMIXComment opposite:
>> #container> <multivalued> <derived>
>>>      <MSEComment: 'list of comments defined in the entity'>
>>>      "^ comments"
>>>      ^ self privateState cacheAt: #comments ifAbsentPut: nil
>>>
>>> As a result, the .image file is 673.3 MB (thus apparently 586,4 MB
>>> used by the model) Not huge, but almost 20 MB saved, 3% of the initial
>>> memory
>>>
>>> nicolas
>>>
>>> On 10/11/2016 20:49, stepharo wrote:
>>>> Thanks
>>>>
>>>> A task of Pavel will be to see how slots usage can reduce that space lost.
>>>> For the empty collection having a kind of
>>>>      - sharing an #() in a classVar
>>>>      - creating a new collection on use could be a solution.
>>>>
>>>> Le 7/11/16 à 12:16, Guillaume Larcheveque a écrit :
>>>>> Hi Moose users,
>>>>>
>>>>> as you may know, at Synectique we experiment lots of issues about
>> memory usage.
>>>>> That's why i had a look at the number of unused instance variables in
>> one of our system loaded with a big model from one of our main customer.
>>>>> Total instance variables: 26.852.653 Used instance variables:
>>>>> 17.393.938 Empty collections: 5.023.508 Recoverable instance
>>>>> variables: 14.482.223
>>>>>
>>>>> The recoverable instance variables are those with nil or an empty
>>>>> collection (or MultiValueLink)
>>>>>
>>>>> As you can see, we can save a lot of memory :-)
>>>>>
>>>>> Here is my (dirty) code to get that:
>>>>>
>>>>> countUsedInstanceVariableInForSubInstances: aClass
>>>>>    | usedInstNbr instNbr emptyCollNbr |
>>>>>    usedInstNbr := 0.
>>>>>    instNbr := 0.
>>>>>    emptyCollNbr := 0.
>>>>>    aClass
>>>>>            allSubInstancesDo: [ :anEntity |
>>>>>                    instNbr := instNbr + anEntity class allInstVarNames
>> size.
>>>>>                    anEntity class allInstVarNames
>>>>>                            doWithIndex: [ :e :i |
>>>>>                                    (anEntity instVarAt: i)
>>>>>                                            ifNotNil: [ :content |
>>>>>                                                    ([
>>>>>                                                    content isEmpty.
>>>>>                                                    emptyCollNbr :=
>> emptyCollNbr + 1.
>>>>>                                                    false ]
>>>>>                                                            on:
>> MessageNotUnderstood
>>>>>                                                            do: [ false ])
>>>>>                                                            ifFalse: [
>> usedInstNbr := usedInstNbr + 1 ] ] ] ].
>>>>>    ^ {('Used instance variables' -> usedInstNbr).
>>>>>    ('empty collections' -> emptyCollNbr).
>>>>>    ('Recoverable instance variables' -> (instNbr - (usedInstNbr -
>> emptyCollNbr))).
>>>>>    ('Total instance variables' -> instNbr)}
>>>>>
>>>>> --
>>>>> Guillaume Larcheveque
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moose-dev mailing list
>>>>>
>>>>> [hidden email]
>>>>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>>>>
>>>>
>>>> _______________________________________________
>>>> Moose-dev mailing list
>>>>
>>>> [hidden email]
>>>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>>> --
>>> Nicolas Anquetil -- MCF (HDR)
>>> Project-Team RMod
>>>
>>> _______________________________________________
>>> Moose-dev mailing list
>>> [hidden email]
>>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>> --
>> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
>> Alexandre Bergel  http://www.bergel.eu
>> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
>>
>>
>>
>> _______________________________________________
>> Moose-dev mailing list
>> [hidden email]
>> https://www.list.inf.unibe.ch/listinfo/moose-dev
> !!!*************************************************************************************
> "Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis.
>
> This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.!!!"
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev