designing a log parser

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

designing a log parser

Tudor Girba-2
Hi,

I am currently working on analyzing some log files using PetitParser, and I have some metaphysical questions about the design I use :).

The log file contains all sorts of events, including method start, method end and others. Furthermore, the log can collect events coming from multiple clients. See below:

2011-12-12 00:00:00,000 TRACE  MethodTraceLogInterceptor: Client 1: START method package.Class#method
2011-12-12 00:00:00,010 TRACE  MethodTraceLogInterceptor: Client 1: START method package2.Class2#methodWithoutEnd
2011-12-12 00:00:00,020 DEBUG Something else
2011-12-12 00:00:00,030 TRACE  MethodTraceLogInterceptor: Client 2: START method package2.Class2#method
2011-12-12 00:00:00,040 DEBUG  Something else
2011-12-12 00:00:02,000 TRACE  MethodTraceLogInterceptor: Client 1: END after 2000 ms method package.Class#method
2011-12-12 00:00:02,030 TRACE  MethodTraceLogInterceptor: Client 2: END after 2000 ms method

I would like to retrieve for each client a tree based on the nesting of method calls.

The grammar is quite straightforward. But, the question is how to get the tree from the parser.

My current solution is to hold a stack in the parser and push/pop to it when a method starts/ends (actually, I have a stack for each client). Something like this:

methodStart
        super methodStart.

        ==> [:token |
                ...
                (self topForClient: (token at: 2)) addChild: newMethodLogEntry.
                self push: newMethodLogEntry forClient: (token at: 2)
        ]

methodEnd
        super methodEnd.

        ==> [:token |
                ...
                self popForClient: (token at: 2)
        ]


This works Ok, and in the end I get my tree in the stack instance variable of the parser. But what makes me uneasy is that the result of the parser is somewhat unclean, because both for methodStart and for methodEnd I will get the same object. However, the good thing about this is that I do not have to create an intermediary representation for the log and directly produce the tree (which can save memory when we talk about large log files).

What do you think? Do you have a more elegant solution?

Cheers,
Doru


--
www.tudorgirba.com

"One cannot do more than one can do."




_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev
Reply | Threaded
Open this post in threaded view
|

Re: designing a log parser

abergel
I would do the same than you.
One thing that you may be careful with, is exception. You may not have an END item if the method has raised an exception.

Alexandre


On 19 Dec 2011, at 03:37, Tudor Girba wrote:

> Hi,
>
> I am currently working on analyzing some log files using PetitParser, and I have some metaphysical questions about the design I use :).
>
> The log file contains all sorts of events, including method start, method end and others. Furthermore, the log can collect events coming from multiple clients. See below:
>
> 2011-12-12 00:00:00,000 TRACE  MethodTraceLogInterceptor: Client 1: START method package.Class#method
> 2011-12-12 00:00:00,010 TRACE  MethodTraceLogInterceptor: Client 1: START method package2.Class2#methodWithoutEnd
> 2011-12-12 00:00:00,020 DEBUG Something else
> 2011-12-12 00:00:00,030 TRACE  MethodTraceLogInterceptor: Client 2: START method package2.Class2#method
> 2011-12-12 00:00:00,040 DEBUG  Something else
> 2011-12-12 00:00:02,000 TRACE  MethodTraceLogInterceptor: Client 1: END after 2000 ms method package.Class#method
> 2011-12-12 00:00:02,030 TRACE  MethodTraceLogInterceptor: Client 2: END after 2000 ms method
>
> I would like to retrieve for each client a tree based on the nesting of method calls.
>
> The grammar is quite straightforward. But, the question is how to get the tree from the parser.
>
> My current solution is to hold a stack in the parser and push/pop to it when a method starts/ends (actually, I have a stack for each client). Something like this:
>
> methodStart
> super methodStart.
>
> ==> [:token |
> ...
> (self topForClient: (token at: 2)) addChild: newMethodLogEntry.
> self push: newMethodLogEntry forClient: (token at: 2)
> ]
>
> methodEnd
> super methodEnd.
>
> ==> [:token |
> ...
> self popForClient: (token at: 2)
> ]
>
>
> This works Ok, and in the end I get my tree in the stack instance variable of the parser. But what makes me uneasy is that the result of the parser is somewhat unclean, because both for methodStart and for methodEnd I will get the same object. However, the good thing about this is that I do not have to create an intermediary representation for the log and directly produce the tree (which can save memory when we talk about large log files).
>
> What do you think? Do you have a more elegant solution?
>
> Cheers,
> Doru
>
>
> --
> www.tudorgirba.com
>
> "One cannot do more than one can do."
>
>
>
>
> _______________________________________________
> Moose-dev mailing list
> [hidden email]
> https://www.iam.unibe.ch/mailman/listinfo/moose-dev

--
_,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:
Alexandre Bergel  http://www.bergel.eu
^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;._,.;:~^~:;.





_______________________________________________
Moose-dev mailing list
[hidden email]
https://www.iam.unibe.ch/mailman/listinfo/moose-dev