Using Moose outside of source assessment context?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Using Moose outside of source assessment context?

Udo Schneider
All,

I’m dabbling with the idea of using Moose in a totally different questions. However I do have some questions about using it in my context. Would this ML be the right place to ask those questions?

CU,

Udo


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (498 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

abergel
Yes

Alexandre


> On Dec 26, 2016, at 11:37 AM, Udo Schneider <[hidden email]> wrote:
>
> All,
>
> I’m dabbling with the idea of using Moose in a totally different questions. However I do have some questions about using it in my context. Would this ML be the right place to ask those questions?
>
> CU,
>
> Udo
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Udo Schneider
ok - here we go :-)

First of all the context: I might (90%+) start a new job on January 1st. A big part of this is cybercrime research/investigation/forensics. The idea is to model different facts (e.g. hosts, IP addresses, groups, hashes …) and their relation as objects and use Moose to reason about them. The classical way to implement this in our industry is to use a Graph/RDF database and describe everything in terms of relations/tuples (Subject/Predicate/Object). 
However taking a look at Moose I’m not 100% sure anymore whether I’d bend Moose to much for my needs based on it's original design/origin.

Defining the (Meta-)Model: I assume I need to define my own subject model representing the entities im interested it with Fame (FM3 is just the Fame classes prefix, correct?).
So I’d end up with something like FAMIX just for my problem domain, correct?

Properties for Links/“Predicates”: The predicate linking to objects need to have properties. E.g. given a tuple hostname/resolves to (DNS)/ip address - the “resolves to” predicate needs properties like the time/date of resolving and the used DNS server. In addition multiple predicates might connect the same hostname and ip address - denoting different points in time the resolve was done.
How would I model such properties for links with Fame?

Actions on subjects: I need to describe actions on subjects. Using the example above (DNS) I’d need an action on host objects which resolves the host address and creates a link (denoting the resolve) to the (to be created) ip address.
Is there any hook in Fame to express those actions on subjects?

Continuous import: The model continues to grow (in terms of subjects and links) over time. This can happen due to actions (see above) or mass import of new data (e.g. DNS zone files).
Is this kind of “growing” model compatible with Moose or does Moose expect a static model?

Model in DB/Multiple clients: As you might imagine the model for a forensic case can grow pretty big very fast. Even for a quick investigation of data we’re easily talking about 10+Gigs of data. In addition we might need to work on the same model (although on different aspects) with multiple people at the same time. So storing everything “in-image” is a no-go IMHO. I have pretty good experience with MongoDB/Voyage and assume this could be made working. Especially because I have to define my own model anyway - thus taking care of database operations.
Are there any experiences with keeping the model in a DB and working with a local image “workbench” on it. Esp. with multiple clients?

“Lab Notebook”: This one is a bit fuzzy - sorry if the intention is unclear. During a case investigation (IMHO an investigation is also part of “the” model) different findings (e.g. data, visualisations) need to be documented. My impression from Moose is that you “play” with the data (in inspectors) until you come to an conclusion based on an assessment. Once you have the result you can decide and tackle the next problem. In my context I’d need to document not only the result but also the way taken to achieve it. Maybe something simple like accumulating the research way through all inspector panes would be enough. I’m not quite sure though.
So how do I document different “results (e.g. data/visualizations) including the “way” they were achieved?

All in all I’m pretty impressed how Moose might fit on my problem domain. Especially using it as a “workbench” to explore/investigate the data is something which is much more pleasure to work with than digging through various SQL/noSQL/Splunk sources with different tools. Not to mention that the visualizations I get are simply astounding and can be used 1:1 on the reports we generate. However ignoring my first impression I’m not sure whether the data model I need to work on is “compatible” with the expectations Moose had/was designed with. Hence the questions.

Thanks,

Udo




On 26 Dec 2016, at 11:39, Alexandre Bergel <[hidden email]> wrote:

Yes

Alexandre


On Dec 26, 2016, at 11:37 AM, Udo Schneider <[hidden email]> wrote:

All,

I’m dabbling with the idea of using Moose in a totally different questions. However I do have some questions about using it in my context. Would this ML be the right place to ask those questions?

CU,

Udo

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Offray Vladimir Luna Cárdenas-2

Hi,

Nice to see this kind of use cases for modeling here, that go beyond the realm of software. I think there is a lot of potential there for Pharo and live coding.


On 26/12/16 07:19, Udo Schneider wrote:

“Lab Notebook”: This one is a bit fuzzy - sorry if the intention is unclear. During a case investigation (IMHO an investigation is also part of “the” model) different findings (e.g. data, visualisations) need to be documented. My impression from Moose is that you “play” with the data (in inspectors) until you come to an conclusion based on an assessment. Once you have the result you can decide and tackle the next problem. In my context I’d need to document not only the result but also the way taken to achieve it. Maybe something simple like accumulating the research way through all inspector panes would be enough. I’m not quite sure though.
So how do I document different “results (e.g. data/visualizations) including the “way” they were achieved?


About the notebook part, you could use Grafoscopio [1][2], which started with what moose provided as a quick prototype, but it's being extended to bring more powerful interactive documentation and is being used to document several research processes, as you can see on [2]. All documents there has been written with Grafoscopio and exported to PDF via pandoc. Is my own bridge between the document/scripting world and the objects world, combining prose and code, which has been pretty useful when we're learning something or understanding the problem and can be used for others to follow our steps and creating interactive tutorials (like the ones we use in our Data Week workshop).

[1] http://smalltalkhub.com/#!/~Offray/Grafoscopio
[2] http://mutabit.com/grafoscopio/index.en.html

It's is my first package on Pharo/Smalltalk (before that I used only Etoys, Bots Inc and Scratch), so is full of rookie code everywhere. But the more confident I become, the more the code improves and I think that Grafoscopio could benefit greatly of having more people seeing the code, making comments or commits on it.

Let me know if it helps.

Cheers,

Offray



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Udo Schneider
Hi Offray,

that might be a big chunk of the functionality I’m searching for. I’ll take a look into it! Thank you very much!

CU,

Udo


On 26 Dec 2016, at 17:18, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:

About the notebook part, you could use Grafoscopio [1][2], which started with what moose provided as a quick prototype, but it's being extended to bring more powerful interactive documentation and is being used to document several research processes, as you can see on [2]. All documents there has been written with Grafoscopio and exported to PDF via pandoc. Is my own bridge between the document/scripting world and the objects world, combining prose and code, which has been pretty useful when we're learning something or understanding the problem and can be used for others to follow our steps and creating interactive tutorials (like the ones we use in our Data Week workshop).

[1] http://smalltalkhub.com/#!/~Offray/Grafoscopio
[2] http://mutabit.com/grafoscopio/index.en.html

It's is my first package on Pharo/Smalltalk (before that I used only Etoys, Bots Inc and Scratch), so is full of rookie code everywhere. But the more confident I become, the more the code improves and I think that Grafoscopio could benefit greatly of having more people seeing the code, making comments or commits on it.

Let me know if it helps.

Cheers,

Offray


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Offray Vladimir Luna Cárdenas-2

Thank Uko for testing it. We're getting more interested people and each one counts. For example, recently we added support for links in the notebook nodes thanks to an idea from Phillipe Back. Hopefully code quality and notebooks will improve also.

Install it from the Pharo catalog, because I'm having problems with ConfigurationOfGrafoscopio now (seems that I'm not using the proper repo).

Cheers,

Offray

On 27/12/16 06:01, Udo Schneider wrote:
Hi Offray,

that might be a big chunk of the functionality I’m searching for. I’ll take a look into it! Thank you very much!

CU,

Udo


On 26 Dec 2016, at 17:18, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:

About the notebook part, you could use Grafoscopio [1][2], which started with what moose provided as a quick prototype, but it's being extended to bring more powerful interactive documentation and is being used to document several research processes, as you can see on [2]. All documents there has been written with Grafoscopio and exported to PDF via pandoc. Is my own bridge between the document/scripting world and the objects world, combining prose and code, which has been pretty useful when we're learning something or understanding the problem and can be used for others to follow our steps and creating interactive tutorials (like the ones we use in our Data Week workshop).

[1] http://smalltalkhub.com/#!/~Offray/Grafoscopio
[2] http://mutabit.com/grafoscopio/index.en.html

It's is my first package on Pharo/Smalltalk (before that I used only Etoys, Bots Inc and Scratch), so is full of rookie code everywhere. But the more confident I become, the more the code improves and I think that Grafoscopio could benefit greatly of having more people seeing the code, making comments or commits on it.

Let me know if it helps.

Cheers,

Offray


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Udo Schneider
Hi Offray,

Install it from the Pharo catalog, because I'm having problems with ConfigurationOfGrafoscopio now (seems that I'm not using the proper repo).

Yepp - discovered this the hard way :-)

I just started to play around with it and hit some issues? Where to report them best?

Just a general question: What was the reason to go for markdown/pandoc as opposed to Pillar?


CU,

Udo


On 27 Dec 2016, at 15:41, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:

Thank Uko for testing it. We're getting more interested people and each one counts. For example, recently we added support for links in the notebook nodes thanks to an idea from Phillipe Back. Hopefully code quality and notebooks will improve also.

Install it from the Pharo catalog, because I'm having problems with ConfigurationOfGrafoscopio now (seems that I'm not using the proper repo).

Cheers,

Offray

On 27/12/16 06:01, Udo Schneider wrote:
Hi Offray,

that might be a big chunk of the functionality I’m searching for. I’ll take a look into it! Thank you very much!

CU,

Udo


On 26 Dec 2016, at 17:18, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:

About the notebook part, you could use Grafoscopio [1][2], which started with what moose provided as a quick prototype, but it's being extended to bring more powerful interactive documentation and is being used to document several research processes, as you can see on [2]. All documents there has been written with Grafoscopio and exported to PDF via pandoc. Is my own bridge between the document/scripting world and the objects world, combining prose and code, which has been pretty useful when we're learning something or understanding the problem and can be used for others to follow our steps and creating interactive tutorials (like the ones we use in our Data Week workshop).

[1] http://smalltalkhub.com/#!/~Offray/Grafoscopio
[2] http://mutabit.com/grafoscopio/index.en.html

It's is my first package on Pharo/Smalltalk (before that I used only Etoys, Bots Inc and Scratch), so is full of rookie code everywhere. But the more confident I become, the more the code improves and I think that Grafoscopio could benefit greatly of having more people seeing the code, making comments or commits on it.

Let me know if it helps.

Cheers,

Offray


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

abergel
In reply to this post by Udo Schneider
Using Fame has the nice property to be able to save a model as a .mse file, for free.
There are very little documentation about fame unfortunately.
There is this document:
http://scg.unibe.ch/archive/papers/Kuhn08cFame.pdf

Famix is made for programming language. In your case, it looks like you absolutely do not need Famix (but you probably want to look into it to have example on how to use Fame).

Let us know how it goes.

Cheers,
Alexandre


> On Dec 26, 2016, at 1:19 PM, Udo Schneider <[hidden email]> wrote:
>
> ok - here we go :-)
>
> First of all the context: I might (90%+) start a new job on January 1st. A big part of this is cybercrime research/investigation/forensics. The idea is to model different facts (e.g. hosts, IP addresses, groups, hashes …) and their relation as objects and use Moose to reason about them. The classical way to implement this in our industry is to use a Graph/RDF database and describe everything in terms of relations/tuples (Subject/Predicate/Object).
> However taking a look at Moose I’m not 100% sure anymore whether I’d bend Moose to much for my needs based on it's original design/origin.
>
> Defining the (Meta-)Model: I assume I need to define my own subject model representing the entities im interested it with Fame (FM3 is just the Fame classes prefix, correct?).
> So I’d end up with something like FAMIX just for my problem domain, correct?
>
> Properties for Links/“Predicates”: The predicate linking to objects need to have properties. E.g. given a tuple hostname/resolves to (DNS)/ip address - the “resolves to” predicate needs properties like the time/date of resolving and the used DNS server. In addition multiple predicates might connect the same hostname and ip address - denoting different points in time the resolve was done.
> How would I model such properties for links with Fame?
>
> Actions on subjects: I need to describe actions on subjects. Using the example above (DNS) I’d need an action on host objects which resolves the host address and creates a link (denoting the resolve) to the (to be created) ip address.
> Is there any hook in Fame to express those actions on subjects?
>
> Continuous import: The model continues to grow (in terms of subjects and links) over time. This can happen due to actions (see above) or mass import of new data (e.g. DNS zone files).
> Is this kind of “growing” model compatible with Moose or does Moose expect a static model?
>
> Model in DB/Multiple clients: As you might imagine the model for a forensic case can grow pretty big very fast. Even for a quick investigation of data we’re easily talking about 10+Gigs of data. In addition we might need to work on the same model (although on different aspects) with multiple people at the same time. So storing everything “in-image” is a no-go IMHO. I have pretty good experience with MongoDB/Voyage and assume this could be made working. Especially because I have to define my own model anyway - thus taking care of database operations.
> Are there any experiences with keeping the model in a DB and working with a local image “workbench” on it. Esp. with multiple clients?
>
> “Lab Notebook”: This one is a bit fuzzy - sorry if the intention is unclear. During a case investigation (IMHO an investigation is also part of “the” model) different findings (e.g. data, visualisations) need to be documented. My impression from Moose is that you “play” with the data (in inspectors) until you come to an conclusion based on an assessment. Once you have the result you can decide and tackle the next problem. In my context I’d need to document not only the result but also the way taken to achieve it. Maybe something simple like accumulating the research way through all inspector panes would be enough. I’m not quite sure though.
> So how do I document different “results (e.g. data/visualizations) including the “way” they were achieved?
>
> All in all I’m pretty impressed how Moose might fit on my problem domain. Especially using it as a “workbench” to explore/investigate the data is something which is much more pleasure to work with than digging through various SQL/noSQL/Splunk sources with different tools. Not to mention that the visualizations I get are simply astounding and can be used 1:1 on the reports we generate. However ignoring my first impression I’m not sure whether the data model I need to work on is “compatible” with the expectations Moose had/was designed with. Hence the questions.
>
> Thanks,
>
> Udo
>
>
>
>
>> On 26 Dec 2016, at 11:39, Alexandre Bergel <[hidden email]> wrote:
>>
>> Yes
>>
>> Alexandre
>>
>>
>>> On Dec 26, 2016, at 11:37 AM, Udo Schneider <[hidden email]> wrote:
>>>
>>> All,
>>>
>>> I’m dabbling with the idea of using Moose in a totally different questions. However I do have some questions about using it in my context. Would this ML be the right place to ask those questions?
>>>
>>> CU,
>>>
>>> Udo
>>>
>>> _______________________________________________
>>> Moose-dev mailing list
>>> [hidden email]
>>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>>
>> --
>> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
>> Alexandre Bergel  http://www.bergel.eu
>> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
>>
>>
>>
>> _______________________________________________
>> Moose-dev mailing list
>> [hidden email]
>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Offray Vladimir Luna Cárdenas-2
In reply to this post by Udo Schneider
Hi,


On 29/12/16 07:31, Udo Schneider wrote:
> Hi Offray,
>
>> Install it from the Pharo catalog, because I'm having problems with
>> ConfigurationOfGrafoscopio now (seems that I'm not using the proper
>> repo).
>>
> Yepp - discovered this the hard way :-)

Sorry. I will fix it soon. This are slow days for family and friends at
the end of year, but I'll let you know as soon as is fixed.

>
> I just started to play around with it and hit some issues? Where to
> report them best?

Use the fossil repo for Grafoscopio, at
http://mutabit.com/repos.fossil/grafoscopio/ticket

>
> Just a general question: What was the reason to go for markdown/pandoc
> as opposed to Pillar?
>
>

I was using pandoc before for writing my PhD thesis before (now I have
migrated to LaTeX) with the key advantage of using of @keywords for
Zotero bibliography integration and I need compatibility with my own
past :-). Also markdown is widely spread compared to Pillar, with
scholar efforts to use it, like scholmd [1][2] and a wider community of
current practitioners. Pandoc is a strong suite for working with
markdown and extending it, covering most writing needs, including
footnotes, academic bibliography support (which is poorly or not covered
at all in most markup languages), importing and exporting to several
formats (as you can see on [3] at bottom) and possibilities to hack it
with several languages via Abstract Syntax Trees using JSON exchange
language[4]. By supporting Pandoc I think I will cover my own needs for
academic writing in several scenarios and compel a wider audience of
present and future researchers, as key population for Grafoscopio.

[1] http://scholmd.org/
[2] http://scholmd.org/basic_setup/what-is-scholarly-markdown.html
[3] http://pandoc.org/
[4] http://pandoc.org/scripting.html

Cheers,

Offray


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Offray Vladimir Luna Cárdenas-2


On 30/12/16 09:09, Offray Vladimir Luna Cárdenas wrote:
> I was using pandoc before for writing my PhD thesis before
Je je, that sound like the redundancy deparment of redundancy :-P

Offray

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Udo Schneider
In reply to this post by abergel
Hi Alexandre,

thanks for the clarification. I think I have to dig in into MSE/Fame then.

My conclusion at the moment (which as both positive and negative impacts) is that I can reuse parts of Moose (e.g. FAME) but not the actual Moose UI. This also means that I can build a model with storage hooks backed in right from the beginning.

Thanks,

Udo



> On 29 Dec 2016, at 20:28, Alexandre Bergel <[hidden email]> wrote:
>
> Using Fame has the nice property to be able to save a model as a .mse file, for free.
> There are very little documentation about fame unfortunately.
> There is this document:
> http://scg.unibe.ch/archive/papers/Kuhn08cFame.pdf
>
> Famix is made for programming language. In your case, it looks like you absolutely do not need Famix (but you probably want to look into it to have example on how to use Fame).
>
> Let us know how it goes.
>
> Cheers,
> Alexandre
>
>
>> On Dec 26, 2016, at 1:19 PM, Udo Schneider <[hidden email]> wrote:
>>
>> ok - here we go :-)
>>
>> First of all the context: I might (90%+) start a new job on January 1st. A big part of this is cybercrime research/investigation/forensics. The idea is to model different facts (e.g. hosts, IP addresses, groups, hashes …) and their relation as objects and use Moose to reason about them. The classical way to implement this in our industry is to use a Graph/RDF database and describe everything in terms of relations/tuples (Subject/Predicate/Object).
>> However taking a look at Moose I’m not 100% sure anymore whether I’d bend Moose to much for my needs based on it's original design/origin.
>>
>> Defining the (Meta-)Model: I assume I need to define my own subject model representing the entities im interested it with Fame (FM3 is just the Fame classes prefix, correct?).
>> So I’d end up with something like FAMIX just for my problem domain, correct?
>>
>> Properties for Links/“Predicates”: The predicate linking to objects need to have properties. E.g. given a tuple hostname/resolves to (DNS)/ip address - the “resolves to” predicate needs properties like the time/date of resolving and the used DNS server. In addition multiple predicates might connect the same hostname and ip address - denoting different points in time the resolve was done.
>> How would I model such properties for links with Fame?
>>
>> Actions on subjects: I need to describe actions on subjects. Using the example above (DNS) I’d need an action on host objects which resolves the host address and creates a link (denoting the resolve) to the (to be created) ip address.
>> Is there any hook in Fame to express those actions on subjects?
>>
>> Continuous import: The model continues to grow (in terms of subjects and links) over time. This can happen due to actions (see above) or mass import of new data (e.g. DNS zone files).
>> Is this kind of “growing” model compatible with Moose or does Moose expect a static model?
>>
>> Model in DB/Multiple clients: As you might imagine the model for a forensic case can grow pretty big very fast. Even for a quick investigation of data we’re easily talking about 10+Gigs of data. In addition we might need to work on the same model (although on different aspects) with multiple people at the same time. So storing everything “in-image” is a no-go IMHO. I have pretty good experience with MongoDB/Voyage and assume this could be made working. Especially because I have to define my own model anyway - thus taking care of database operations.
>> Are there any experiences with keeping the model in a DB and working with a local image “workbench” on it. Esp. with multiple clients?
>>
>> “Lab Notebook”: This one is a bit fuzzy - sorry if the intention is unclear. During a case investigation (IMHO an investigation is also part of “the” model) different findings (e.g. data, visualisations) need to be documented. My impression from Moose is that you “play” with the data (in inspectors) until you come to an conclusion based on an assessment. Once you have the result you can decide and tackle the next problem. In my context I’d need to document not only the result but also the way taken to achieve it. Maybe something simple like accumulating the research way through all inspector panes would be enough. I’m not quite sure though.
>> So how do I document different “results (e.g. data/visualizations) including the “way” they were achieved?
>>
>> All in all I’m pretty impressed how Moose might fit on my problem domain. Especially using it as a “workbench” to explore/investigate the data is something which is much more pleasure to work with than digging through various SQL/noSQL/Splunk sources with different tools. Not to mention that the visualizations I get are simply astounding and can be used 1:1 on the reports we generate. However ignoring my first impression I’m not sure whether the data model I need to work on is “compatible” with the expectations Moose had/was designed with. Hence the questions.
>>
>> Thanks,
>>
>> Udo
>>
>>
>>
>>
>>> On 26 Dec 2016, at 11:39, Alexandre Bergel <[hidden email]> wrote:
>>>
>>> Yes
>>>
>>> Alexandre
>>>
>>>
>>>> On Dec 26, 2016, at 11:37 AM, Udo Schneider <[hidden email]> wrote:
>>>>
>>>> All,
>>>>
>>>> I’m dabbling with the idea of using Moose in a totally different questions. However I do have some questions about using it in my context. Would this ML be the right place to ask those questions?
>>>>
>>>> CU,
>>>>
>>>> Udo
>>>>
>>>> _______________________________________________
>>>> Moose-dev mailing list
>>>> [hidden email]
>>>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>>>
>>> --
>>> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
>>> Alexandre Bergel  http://www.bergel.eu
>>> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
>>>
>>>
>>>
>>> _______________________________________________
>>> Moose-dev mailing list
>>> [hidden email]
>>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>>
>> _______________________________________________
>> Moose-dev mailing list
>> [hidden email]
>> https://www.list.inf.unibe.ch/listinfo/moose-dev
>
> --
> _,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
> Alexandre Bergel  http://www.bergel.eu
> ^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.
>
>
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Udo Schneider
In reply to this post by Offray Vladimir Luna Cárdenas-2
Hi Offray,

I just started to play around with it and hit some issues? Where to report them best?

Use the fossil repo for Grafoscopio, at http://mutabit.com/repos.fossil/grafoscopio/ticket
Will do.

And thanks for the clarification on MD/pandoc. I have not yet decided which route to follow. On the one Hand I’m very comfortable using Pillar - esp. because most of it is “in-image”. On the other hand I can imagine my coworkers already know MD - so this might be the easier route.

Best Regards,

Udo


On 30 Dec 2016, at 15:09, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:

Hi,


On 29/12/16 07:31, Udo Schneider wrote:
Hi Offray,

Install it from the Pharo catalog, because I'm having problems with ConfigurationOfGrafoscopio now (seems that I'm not using the proper repo).

Yepp - discovered this the hard way :-)

Sorry. I will fix it soon. This are slow days for family and friends at the end of year, but I'll let you know as soon as is fixed.


I just started to play around with it and hit some issues? Where to report them best?

Use the fossil repo for Grafoscopio, at http://mutabit.com/repos.fossil/grafoscopio/ticket


Just a general question: What was the reason to go for markdown/pandoc as opposed to Pillar?



I was using pandoc before for writing my PhD thesis before (now I have migrated to LaTeX) with the key advantage of using of @keywords for Zotero bibliography integration and I need compatibility with my own past :-). Also markdown is widely spread compared to Pillar, with scholar efforts to use it, like scholmd [1][2] and a wider community of current practitioners. Pandoc is a strong suite for working with markdown and extending it, covering most writing needs, including footnotes, academic bibliography support (which is poorly or not covered at all in most markup languages), importing and exporting to several formats (as you can see on [3] at bottom) and possibilities to hack it with several languages via Abstract Syntax Trees using JSON exchange language[4]. By supporting Pandoc I think I will cover my own needs for academic writing in several scenarios and compel a wider audience of present and future researchers, as key population for Grafoscopio.

[1] http://scholmd.org/
[2] http://scholmd.org/basic_setup/what-is-scholarly-markdown.html
[3] http://pandoc.org/
[4] http://pandoc.org/scripting.html

Cheers,

Offray


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Stephan Eggermont-3
In reply to this post by Udo Schneider



On 26/12/16 13:19, Udo Schneider wrote:
>  The classical way to
> implement this in our industry is to use a Graph/RDF database and
> describe everything in terms of relations/tuples
> (Subject/Predicate/Object).

The more classical way to provide real-time data-analysis is using
gemstone technology http://www.gemstone.com/customers/DISA

> Defining the (Meta-)Model: I assume I need to define my own subject
> model representing the entities im interested it with Fame (FM3 is
> just the Fame classes prefix, correct?). *So I’d end up with
> something like FAMIX just for my problem domain, correct?*

You could, and there might not be much added value in doing
that instead of just creating a domain model directly in smalltalk.

> Continuous import: The model continues to grow (in terms of subjects
> and links) over time. This can happen due to actions (see above) or
> mass import of new data (e.g. DNS zone files). *Is this kind of
> “growing” model compatible with Moose or does Moose expect a static
> model?*

Moose expects a dynamic model, but one that is not so large.
With a 64-bit image the size is less of a problem than the
processing speed. You might want to distribute the processing
over many images, and create different mappings of data sources
to images to optimize your queries.

> Model in DB/Multiple clients: As you might imagine the model for a
> forensic case can grow pretty big very fast. Even for a quick
> investigation of data we’re easily talking about 10+Gigs of data.

By default log files are extremely redundant. Compressing them by
using a decent domain model, introducing value objects to reduce
data size can be useful.

> In
> addition we might need to work on the same model (although on
> different aspects) with multiple people at the same time. So storing
> everything “in-image” is a no-go IMHO. I have pretty good experience
> with MongoDB/Voyage and assume this could be made working. Especially
> because I have to define my own model anyway - thus taking care of
> database operations. *Are there any experiences with keeping the
> model in a DB and working with a local image “workbench” on it. Esp.
> with multiple clients?*

Most databases work badly with this kind of problem. You might want to
expicitly model the mapping of data to multiple images. You are
basically creating a blackboard system

>
> “Lab Notebook”: This one is a bit fuzzy - sorry if the intention is
> unclear. During a case investigation (IMHO an investigation is also
> part of “the” model) different findings (e.g. data, visualisations)
> need to be documented. My impression from Moose is that you “play”
> with the data (in inspectors) until you come to an conclusion based
> on an assessment. Once you have the result you can decide and tackle
> the next problem. In my context I’d need to document not only the
> result but also the way taken to achieve it. Maybe something simple
> like accumulating the research way through all inspector panes would
> be enough. I’m not quite sure though. *So how do I document different
> “results (e.g. data/visualizations) including the “way” they were
> achieved?*

This probably needs a double registration. From a governance p.o.v.
you need the actual process steps, and you also need a log of how you
could have reached your conclusions with perfect hindsight
http://www.ics.uci.edu/~taylor/classes/121/IEEE86_Parnas_Clement.pdf


> All in all I’m pretty impressed how Moose might fit on my problem
> domain. Especially using it as a “workbench” to explore/investigate
> the data is something which is much more pleasure to work with than
> digging through various SQL/noSQL/Splunk sources with different
> tools. Not to mention that the visualizations I get are simply
> astounding and can be used 1:1 on the reports we generate. However
> ignoring my first impression I’m not sure whether the data model I
> need to work on is “compatible” with the expectations Moose had/was
> designed with. Hence the questions.

Moose supports creating your own tools very well. We did a data
migration with Moose, and created 60+ glamour browsers for it,
and daily visualisations of the process. We managed to keep things
in a single image, though we ran out of memory several times.
To me it sounds like you'd need to develop something to manage lots
of images, starting new ones on demand. That fits well with our
Pharo vision.

Stephan
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Offray Vladimir Luna Cárdenas-2
In reply to this post by Offray Vladimir Luna Cárdenas-2
Hi,


On 30/12/16 09:09, Offray Vladimir Luna Cárdenas wrote:

> Hi,
>
>
> On 29/12/16 07:31, Udo Schneider wrote:
>> Hi Offray,
>>
>>> Install it from the Pharo catalog, because I'm having problems with
>>> ConfigurationOfGrafoscopio now (seems that I'm not using the proper
>>> repo).
>>>
>> Yepp - discovered this the hard way :-)
>
> Sorry. I will fix it soon. This are slow days for family and friends
> at the end of year, but I'll let you know as soon as is fixed.
>


Now installation works from Gopher and instructions are fixed.

Cheers,

Offray
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Udo Schneider
In reply to this post by Stephan Eggermont-3

On 01 Jan 2017, at 16:15, stephan <[hidden email]> wrote:
The more classical way to provide real-time data-analysis is using
gemstone technology http://www.gemstone.com/customers/DISA
I fully agree they should be be using that. However most companies don’t (want to) know Smalltalk anymore. For those RDF was an answer to their prayers and SPARQL the promise to be db-backend independent for eternity :-)

You could, and there might not be much added value in doing
that instead of just creating a domain model directly in smalltalk.
That’s a main thing I have to research - which benefits do I get from using/implementing this with a matching meta model.

Moose expects a dynamic model, but one that is not so large.
With a 64-bit image the size is less of a problem than the
processing speed. You might want to distribute the processing
over many images, and create different mappings of data sources
to images to optimize your queries.
Distributed images … that’s something I didn’t think about yet. This might really solve some concerns I have with the backend.

By default log files are extremely redundant. Compressing them by
using a decent domain model, introducing value objects to reduce
data size can be useful.
True - if (only) log data would be the (only) problem. Having done this for another for another project (REST API forensics) we easily achieved 1:1000+ ratios in compression. Especially modelling the IPv6 addresses as Singletons and using a tree model for the paths/queries provided amazing compression. And once the paths were objects and not mere dumb strings the final verdict came fast.

Most databases work badly with this kind of problem. You might want to
expicitly model the mapping of data to multiple images. You are
basically creating a blackboard system
Never heard of if - but researching it definitely matched the idea I have in mind!!!
This is a great pointer I need to investigate further. I hope to find more patterns there helping me to do the system “right” with less iterations.
Although I did some research on the stuff I plan to build I found no matching paper on the subject whatsoever. It seems I was missing the right terms for concepts and ideas :-(

This probably needs a double registration. From a governance p.o.v.
you need the actual process steps, and you also need a log of how you
could have reached your conclusions with perfect hindsight
http://www.ics.uci.edu/~taylor/classes/121/IEEE86_Parnas_Clement.pdf
Another gem!

Moose supports creating your own tools very well. We did a data
migration with Moose, and created 60+ glamour browsers for it,
and daily visualisations of the process. We managed to keep things
in a single image, though we ran out of memory several times.
To me it sounds like you'd need to develop something to manage lots
of images, starting new ones on demand. That fits well with our
Pharo vision.
I reached the point where the decision for Moose/Pharo is already taken (positively!). I’m just not decided yet on the backend(s) side.

Thank you very much for your comments! Very helpful and highly appreciated!

CU,

Udo




_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Nicolas Anquetil
In reply to this post by Udo Schneider


Hi,

I haven't read this thread closely.

But I can propose a presentation on extending FAMIX, the concepts should help you create another metamodel.

https://www.dropbox.com/s/iu8zlstfrz26gmo/Famix-XML.pdf

hope you find it useful

nicolas

On 01/01/2017 14:12, Udo Schneider wrote:
Hi Alexandre,

thanks for the clarification. I think I have to dig in into MSE/Fame then.

My conclusion at the moment (which as both positive and negative impacts) is that I can reuse parts of Moose (e.g. FAME) but not the actual Moose UI. This also means that I can build a model with storage hooks backed in right from the beginning.

Thanks,

Udo



On 29 Dec 2016, at 20:28, Alexandre Bergel [hidden email] wrote:

Using Fame has the nice property to be able to save a model as a .mse file, for free.
There are very little documentation about fame unfortunately.
There is this document:
http://scg.unibe.ch/archive/papers/Kuhn08cFame.pdf

Famix is made for programming language. In your case, it looks like you absolutely do not need Famix (but you probably want to look into it to have example on how to use Fame).

Let us know how it goes.

Cheers,
Alexandre


On Dec 26, 2016, at 1:19 PM, Udo Schneider [hidden email] wrote:

ok - here we go :-)

First of all the context: I might (90%+) start a new job on January 1st. A big part of this is cybercrime research/investigation/forensics. The idea is to model different facts (e.g. hosts, IP addresses, groups, hashes …) and their relation as objects and use Moose to reason about them. The classical way to implement this in our industry is to use a Graph/RDF database and describe everything in terms of relations/tuples (Subject/Predicate/Object).
However taking a look at Moose I’m not 100% sure anymore whether I’d bend Moose to much for my needs based on it's original design/origin.

Defining the (Meta-)Model: I assume I need to define my own subject model representing the entities im interested it with Fame (FM3 is just the Fame classes prefix, correct?).
So I’d end up with something like FAMIX just for my problem domain, correct?

Properties for Links/“Predicates”: The predicate linking to objects need to have properties. E.g. given a tuple hostname/resolves to (DNS)/ip address - the “resolves to” predicate needs properties like the time/date of resolving and the used DNS server. In addition multiple predicates might connect the same hostname and ip address - denoting different points in time the resolve was done.
How would I model such properties for links with Fame?

Actions on subjects: I need to describe actions on subjects. Using the example above (DNS) I’d need an action on host objects which resolves the host address and creates a link (denoting the resolve) to the (to be created) ip address.
Is there any hook in Fame to express those actions on subjects?

Continuous import: The model continues to grow (in terms of subjects and links) over time. This can happen due to actions (see above) or mass import of new data (e.g. DNS zone files).
Is this kind of “growing” model compatible with Moose or does Moose expect a static model?

Model in DB/Multiple clients: As you might imagine the model for a forensic case can grow pretty big very fast. Even for a quick investigation of data we’re easily talking about 10+Gigs of data. In addition we might need to work on the same model (although on different aspects) with multiple people at the same time. So storing everything “in-image” is a no-go IMHO. I have pretty good experience with MongoDB/Voyage and assume this could be made working. Especially because I have to define my own model anyway - thus taking care of database operations.
Are there any experiences with keeping the model in a DB and working with a local image “workbench” on it. Esp. with multiple clients?

“Lab Notebook”: This one is a bit fuzzy - sorry if the intention is unclear. During a case investigation (IMHO an investigation is also part of “the” model) different findings (e.g. data, visualisations) need to be documented. My impression from Moose is that you “play” with the data (in inspectors) until you come to an conclusion based on an assessment. Once you have the result you can decide and tackle the next problem. In my context I’d need to document not only the result but also the way taken to achieve it. Maybe something simple like accumulating the research way through all inspector panes would be enough. I’m not quite sure though.
So how do I document different “results (e.g. data/visualizations) including the “way” they were achieved?

All in all I’m pretty impressed how Moose might fit on my problem domain. Especially using it as a “workbench” to explore/investigate the data is something which is much more pleasure to work with than digging through various SQL/noSQL/Splunk sources with different tools. Not to mention that the visualizations I get are simply astounding and can be used 1:1 on the reports we generate. However ignoring my first impression I’m not sure whether the data model I need to work on is “compatible” with the expectations Moose had/was designed with. Hence the questions.

Thanks,

Udo




On 26 Dec 2016, at 11:39, Alexandre Bergel [hidden email] wrote:

Yes

Alexandre


On Dec 26, 2016, at 11:37 AM, Udo Schneider [hidden email] wrote:

All,

I’m dabbling with the idea of using Moose in a totally different questions. However I do have some questions about using it in my context. Would this ML be the right place to ask those questions?

CU,

Udo

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.



_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

      

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

-- 
Nicolas Anquetil -- MCF (HDR)
Project-Team RMod

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Udo Schneider

On 02 Jan 2017, at 17:40, Nicolas Anquetil <[hidden email]> wrote:
But I can propose a presentation on extending FAMIX, the concepts should help you create another metamodel.
https://www.dropbox.com/s/iu8zlstfrz26gmo/Famix-XML.pdf

Thanks for the link. Very helpful!

CU,

Udo


_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev

signature.asc (858 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Using Moose outside of source assessment context?

Peter Uhnak
In reply to this post by Udo Schneider
Hi Udo,

I will try to reread your questions soon, but right now I am drowning in work;

however based on some of your questions (related to modeling, visualizations), you might be interested also in OpenPonk ( https://openponk.github.io/ ),
E.g. we have several different models (simple state machines, business modeling, software modeling (based on Moose' FAMIX)), with visualizations, validations, etc. This can be installed on top of Moose, so you can use it in complement.
One of OpenPonk ideas is to provide a more supportive environment for custom models and working with them; the project is still in quite infant phase, but we are slowly advancing. :)
Your use case certainly sounds like something that could greatly benefit from Pharo and tools and frameworks around (Moose, OpenPonk, Roassal, Grafoscopio, ...), so I will back to this when I stop drowning. :)

Peter

On Sun, Jan 01, 2017 at 02:14:24PM +0100, Udo Schneider wrote:

> Hi Offray,
>
> >> I just started to play around with it and hit some issues? Where to report them best?
> >
> > Use the fossil repo for Grafoscopio, at http://mutabit.com/repos.fossil/grafoscopio/ticket <http://mutabit.com/repos.fossil/grafoscopio/ticket>Will do.
>
> And thanks for the clarification on MD/pandoc. I have not yet decided which route to follow. On the one Hand I’m very comfortable using Pillar - esp. because most of it is “in-image”. On the other hand I can imagine my coworkers already know MD - so this might be the easier route.
>
> Best Regards,
>
> Udo
>
>
> > On 30 Dec 2016, at 15:09, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
> >
> > Hi,
> >
> >
> > On 29/12/16 07:31, Udo Schneider wrote:
> >> Hi Offray,
> >>
> >>> Install it from the Pharo catalog, because I'm having problems with ConfigurationOfGrafoscopio now (seems that I'm not using the proper repo).
> >>>
> >> Yepp - discovered this the hard way :-)
> >
> > Sorry. I will fix it soon. This are slow days for family and friends at the end of year, but I'll let you know as soon as is fixed.
> >
> >>
> >> I just started to play around with it and hit some issues? Where to report them best?
> >
> > Use the fossil repo for Grafoscopio, at http://mutabit.com/repos.fossil/grafoscopio/ticket
> >
> >>
> >> Just a general question: What was the reason to go for markdown/pandoc as opposed to Pillar?
> >>
> >>
> >
> > I was using pandoc before for writing my PhD thesis before (now I have migrated to LaTeX) with the key advantage of using of @keywords for Zotero bibliography integration and I need compatibility with my own past :-). Also markdown is widely spread compared to Pillar, with scholar efforts to use it, like scholmd [1][2] and a wider community of current practitioners. Pandoc is a strong suite for working with markdown and extending it, covering most writing needs, including footnotes, academic bibliography support (which is poorly or not covered at all in most markup languages), importing and exporting to several formats (as you can see on [3] at bottom) and possibilities to hack it with several languages via Abstract Syntax Trees using JSON exchange language[4]. By supporting Pandoc I think I will cover my own needs for academic writing in several scenarios and compel a wider audience of present and future researchers, as key population for Grafoscopio.
> >
> > [1] http://scholmd.org/
> > [2] http://scholmd.org/basic_setup/what-is-scholarly-markdown.html
> > [3] http://pandoc.org/
> > [4] http://pandoc.org/scripting.html
> >
> > Cheers,
> >
> > Offray
> >
> >
> > _______________________________________________
> > Moose-dev mailing list
> > [hidden email]
> > https://www.list.inf.unibe.ch/listinfo/moose-dev
>



> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.list.inf.unibe.ch/listinfo/moose-dev

_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.list.inf.unibe.ch/listinfo/moose-dev