experience with large images?

Hi Alex,

With the team at Bordeaux, we are working on an implementation to query histories (the tool is named VPraxis).

I implement a importer for Smalltalk.

For now, I can load histories for a package, for a metacello configuration, from a repository.

A first part is build in Pharo (building snapshots from Monticello), and a second part in build with our java software and build the history file. We can export xml, sql or swipl, which allows us to build prolog queries.

Here (http://code.google.com/p/harmony/wiki/VPraxis) is a tutorial for the java VPraxis.

For Smalltalk, I am working on it.

I generated the current Mondrian history in a swipl format, available here: http://dl.dropbox.com/u/7739334/MondrianHistorySwipl.zip

An important point is that we do not into account the multiple branches (it is a current work :) ). So, when there are multiple branches, for now, we check were is the merge and we "ignore the multiple branches".

Now, for your question, we create a script: http://dl.dropbox.com/u/7739334/mondrianScript.txt

Using swi-prolog (http://www.swi-prolog.org/) , you can call, in this script, the method moreN(L, 10).

For the package Mondrian-Layout, it returns: L = [id_MOAbstractGraphLayout, id_MOCircleLayout, id_MOSugiyamaLayout].

Which are the three classes that have changed more that 10 times.

For now, I am working on packaging all the things together. 

But if you have some other requests, do not hesitate to ask me :)

Cheers,

Jannik

On Nov 24, 2011, at 14:05 , Alexandre Bergel wrote:

Having a reification in Moose of 100 versions of Mondrian for example :-)

Just answering the question 'Which classes and methods of Mondrian have changed more than 10 times since the day Mondrian was born?' cannot be easily done without a lot of memory

Alexandre


On 24 Nov 2011, at 03:27, Francois Stephany wrote:

I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is.

Alex, what is your use case (in practice!) for more than 500MB?

On 23/11/11 18:25, Igor Stasenko wrote:

It is problematic, and requires different memory management than we

currently have.

I think if you need really big data sets, then use gemstone, which is

developed to deal with that specifically.

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

---

Jannik Laval

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev

On Dec 8, 2011, at 22:48 , Usman Bhatti wrote:

On Thu, Dec 8, 2011 at 5:15 PM, jannik.laval <[hidden email]> wrote:

Hi Alex,

With the team at Bordeaux, we are working on an implementation to query histories (the tool is named VPraxis).

I implement a importer for Smalltalk.

For now, I can load histories for a package, for a metacello configuration, from a repository.

A first part is build in Pharo (building snapshots from Monticello), and a second part in build with our java software and build the history file. We can export xml, sql or swipl, which allows us to build prolog queries.

I am looking for an infrastructure in java which can be used to query java history for versions of packges, classes, authors, etc. Do you already have something related to java?

Yes,

Actually, we have an importer for Java, C and Smalltalk.

Jannik

 

Here (http://code.google.com/p/harmony/wiki/VPraxis) is a tutorial for the java VPraxis.

For Smalltalk, I am working on it.

I generated the current Mondrian history in a swipl format, available here: http://dl.dropbox.com/u/7739334/MondrianHistorySwipl.zip

An important point is that we do not into account the multiple branches (it is a current work :) ). So, when there are multiple branches, for now, we check were is the merge and we "ignore the multiple branches".

Now, for your question, we create a script: http://dl.dropbox.com/u/7739334/mondrianScript.txt

Using swi-prolog (http://www.swi-prolog.org/) , you can call, in this script, the method moreN(L, 10).

For the package Mondrian-Layout, it returns: L = [id_MOAbstractGraphLayout, id_MOCircleLayout, id_MOSugiyamaLayout].

Which are the three classes that have changed more that 10 times.

For now, I am working on packaging all the things together. 

But if you have some other requests, do not hesitate to ask me :)

Cheers,

Jannik

On Nov 24, 2011, at 14:05 , Alexandre Bergel wrote:

Having a reification in Moose of 100 versions of Mondrian for example :-)

Just answering the question 'Which classes and methods of Mondrian have changed more than 10 times since the day Mondrian was born?' cannot be easily done without a lot of memory

Alexandre


On 24 Nov 2011, at 03:27, Francois Stephany wrote:

I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is.

Alex, what is your use case (in practice!) for more than 500MB?

On 23/11/11 18:25, Igor Stasenko wrote:

It is problematic, and requires different memory management than we

currently have.

I think if you need really big data sets, then use gemstone, which is

developed to deal with that specifically.

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

---

Jannik Laval

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev

---

Jannik Laval

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev

> With the team at Bordeaux, we are working on an implementation to query histories (the tool is named VPraxis).

Which kind of query are you interested in? Is Hismo not enough?
I worked on a configuration importer. We can analyze many many versions now :-)

> I implement a importer for Smalltalk.
> For now, I can load histories for a package, for a metacello configuration, from a repository.

Have a look at the Spirit project on ss3.gemstone.com/ss
It contains a lot of scripts to load versions of Mondrian, Versionner, Spy, ...

> A first part is build in Pharo (building snapshots from Monticello), and a second part in build with our java software and build the history file. We can export xml, sql or swipl, which allows us to build prolog queries.
>
> Here (http://code.google.com/p/harmony/wiki/VPraxis) is a tutorial for the java VPraxis.
> For Smalltalk, I am working on it.
>
> I generated the current Mondrian history in a swipl format, available here: http://dl.dropbox.com/u/7739334/MondrianHistorySwipl.zip
> An important point is that we do not into account the multiple branches (it is a current work :) ). So, when there are multiple branches, for now, we check were is the merge and we "ignore the multiple branches".
>
> Now, for your question, we create a script: http://dl.dropbox.com/u/7739334/mondrianScript.txt
> Using swi-prolog (http://www.swi-prolog.org/) , you can call, in this script, the method moreN(L, 10).
>
> For the package Mondrian-Layout, it returns: L = [id_MOAbstractGraphLayout, id_MOCircleLayout, id_MOSugiyamaLayout].
> Which are the three classes that have changed more that 10 times.
>
> For now, I am working on packaging all the things together.
> But if you have some other requests, do not hesitate to ask me :)

Looks cool!

Alexandre

>
>
>
> On Nov 24, 2011, at 14:05 , Alexandre Bergel wrote:
>
>> Having a reification in Moose of 100 versions of Mondrian for example :-)
>>
>> Just answering the question 'Which classes and methods of Mondrian have changed more than 10 times since the day Mondrian was born?' cannot be easily done without a lot of memory
>>
>> Alexandre
>>
>>
>> On 24 Nov 2011, at 03:27, Francois Stephany wrote:
>>
>>> I'm wondering: how big is a dataset > 500MB ? I've no idea how big it is.
>>> Alex, what is your use case (in practice!) for more than 500MB?
>>>
>>> On 23/11/11 18:25, Igor Stasenko wrote:
>>>> It is problematic, and requires different memory management than we
>>>> currently have.
>>>> I think if you need really big data sets, then use gemstone, which is
>>>> developed to deal with that specifically.
>>>
>>
>> --
>> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
>> Alexandre Bergel  http://www.bergel.eu
>> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
>>
>>
>>
>>
>>
>>
>
> ---
> Jannik Laval
>

> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.iam.unibe.ch/mailman/listinfo/moose-dev

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev

> Alex, where can I have more information about this importer? Is that related to a script you sent to the list few days ago?

Yes. You can have a look at the class AutomaticalCaseStudies in ss3.gemstone.com/ss/SPIRIT

I started to work on an importer, but I am not satisfied, mainly because I have to effectively install the code in the image (i.e., compile all the methods).
However, the scripts essentially do what we need.

Cheers,
Alexandre

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.





_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev

On Fri, Dec 9, 2011 at 8:05 PM, Alexandre Bergel <[hidden email]> wrote:

> Alex, where can I have more information about this importer? Is that related to a script you sent to the list few days ago?

Yes. You can have a look at the class AutomaticalCaseStudies in ss3.gemstone.com/ss/SPIRIT

I started to work on an importer, but I am not satisfied, mainly because I have to effectively install the code in the image (i.e., compile all the methods).
However, the scripts essentially do what we need.

Thanks Alex! I will check.

Cheers,
Alexandre

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.





_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev

--
Andre Hora

> So, you are creating Moose models and not one Hismo model in the importer. For instance in the methods AutomaticalCaseStudies>>easy* creates several Moose models. It should be good to have the Hismo model.

Hismo models are trivial to create. We did this at the beginning. However we quickly bump into the limit of 500Mb for the image size.

We were hoping to have 100's of versions at the same time, but we couldn't :-( So, we export everything to Excel.

Cheers,
Alexandre
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.





_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev