Reclaiming Memory in Aidaweb

Hi,

In my trunk image I have a lot of objects I can't get rid of.
I believe when the server is stopped (SwazooAida demoStop)
I should have 0 HTTPRequest subInstances.
In my 3.8 Image I have 2 in Trunk I have 791.
Same numbers in both Images for SwazooTask.

Also I have 1 Instance of HTTPServer in 3.8 while I have 8 in my trunk
image.

What should be the numbers for those classes and how to get rid of of
the many instances in trunk?

Basically I took a trunk image (Cog), updated it to 10606, loaded Sport,
Swazoo, Aida and then my app and started playing.

Another question:
Janko, which measures where taken to keep the memory of Squeak.org
low? I refer to the discussion with Ken on Squeak dev.


Thanks,

Herbert                          mailto:[hidden email]

_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi Herbert. What about expling an instance and see the pointers to it?  you can take the object, explore it, and there you have the option "chase pointers" or similar (I don't know they are called in Squeak). But basically, you can browse the objects that are pointing to that objects.

Maybe there are Processes, Contexts, Sessions or something like that, which are ponting to those objects.



_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi Mariano,

MMP> Hi Herbert. What about expling an instance and see the pointers to it?  you
MMP> can take the object, explore it, and there you have the option "chase
MMP> pointers" or similar (I don't know they are called in Squeak). But
MMP> basically, you can browse the objects that are pointing to that objects.

MMP> Maybe there are Processes, Contexts, Sessions or something like that, which
MMP> are ponting to those objects.

that's the process I use but these objects are greatly interconnected,
so each of them is pointed to by hundreds of objects again. I hope the
8 HTTPServer's are the culprit but don't know how to get rid of them.
again they are pointed to by hundreds of objects.

Maybe the whole mess only is linked to a Global by one single
reference which I need to break.

BTW the PointerFinder on many of these Objects finds SuspendedDelays
(see attachment). At which point I'm stuck. Seems I need to learn
Smalltalk eventually :-)) Any tips?


Cheers,

Herbert                            mailto:[hidden email]
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi,

Sometimes in my DEVELOPMENT environment I evaluate the following:

HTTPConnection allInstances do: [:each | each loop: nil; server: nil; task:
nil.].

Be sure to save the image first !!!

Regards,
Bruno
-----Mensaje original-----
De: [hidden email] [mailto:[hidden email]] En nombre de
Herbert König
Enviado el: Sunday, October 24, 2010 8:16 PM
Para: AIDA/Web general discussion list
Asunto: Re: [aida] Reclaiming Memory in Aidaweb

Hi Mariano,

MMP> Hi Herbert. What about expling an instance and see the pointers to
MMP> it?  you can take the object, explore it, and there you have the
MMP> option "chase pointers" or similar (I don't know they are called in
MMP> Squeak). But basically, you can browse the objects that are pointing to
that objects.

MMP> Maybe there are Processes, Contexts, Sessions or something like
MMP> that, which are ponting to those objects.

that's the process I use but these objects are greatly interconnected, so
each of them is pointed to by hundreds of objects again. I hope the
8 HTTPServer's are the culprit but don't know how to get rid of them.
again they are pointed to by hundreds of objects.

Maybe the whole mess only is linked to a Global by one single reference
which I need to break.

BTW the PointerFinder on many of these Objects finds SuspendedDelays (see
attachment). At which point I'm stuck. Seems I need to learn Smalltalk
eventually :-)) Any tips?


Cheers,

Herbert                            mailto:[hidden email]

_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi Herbert,

Here is a method as an extension of SwazooServer in BiArt (also in
Scribo) for hourly backup but also hourly and nightly cleanup of session
application state on 6.x Aida:

SwazooServer>>watchdogOther

  self isNewHour ifTrue:

    [WebSessionManager allInstances do:
        [:each | each removeNonactiveGuestSessions].
    [(Delay forSeconds: self watchdogPeriod) wait.
    self imageBackupAndSnapshot]
        fork ].

   self isNightlyBackupTime ifTrue:

     [HTTPConnection allInstances do: [:each | each close].
     WebSessionManager allInstances do:
       [:each | each removeGuestSessions].
     WebSessionManager allInstances do:
       [:each | each releaseApplicationState].
    (Delay forSeconds: self watchdogPeriod) wait].


Note specially the new method:

        WebSessionManager>>releaseApplicationState

This method now do a really detailed cleanup of sessions, it clears all
instances of Apps, but also nils last requests etc. The same goes with
methods #removeGuestSessions and #removeNonactiveGuestSessions.

I namely also had similar problems for a while but with this patch now I
manage to control session state successfully.

BTW: I came to the root of the problem exactly as Mariano suggests, just
on VW which has a very good inspector for such undertakings. It seems
that a new Pharo inspector is a good step in that direction too.

Best regards
Janko

On 24. 10. 2010 22:40, Herbert König wrote:
--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi,

thanks to all who replied. Bruno's Tip worked and brought back 60MB of
memory. After garbageCollect and forgetDoit's didn't help an image
saveAndQuit actually brought it back. Then following Mariano's clue,
in Delay's class var SuspendedDelays I unscheduled all 161 delays and
then sent that heap a removeAll. Another save shrunk the image for
another 40MB.


JM>     [WebSessionManager allInstances do:
JM> [:each | each removeNonactiveGuestSessions].

This one hung the image, very little Processor load, alt dot got the
debugger but the UI stayed unresponsive.

JM>      [HTTPConnection allInstances do: [:each | each close].

No help from this.

JM>      WebSessionManager allInstances do:
JM>        [:each | each removeGuestSessions].

Hung the image as before.

JM>      WebSessionManager allInstances do:
JM>        [:each | each releaseApplicationState].

Hung the image.

JM> WebSessionManager>>>releaseApplicationState

JM> This method now do a really detailed cleanup of sessions, it clears all
JM> instances of Apps, but also nils last requests etc. The same goes with
JM> methods #removeGuestSessions and #removeNonactiveGuestSessions.

JM> I namely also had similar problems for a while but with this patch now I
JM> manage to control session state successfully.

Thanks I' ll look into these to learn. They should come handy in an
intact image.

All in all it looks like I messed up the image badly (though I have no
clue how I did it). Maybe I stumbled across an oddity of cog.

What I actually did was creating one million datapoints in my app in
an attempt to measure cog's performance. Actually I did it twice
because the first time the jitting of methods used up the performance.
I got 4600 ms in Squeak 3.8 with 3.7 VM and 3400 and 984ms in trunk with cog.

I'll continue this because I'm interested in the performance of Aida +
my app. Actually I still look for ideas of how to do this apart from
wgetting the whole site from several computers.


I'll report any results here.

Cheers,

Herbert                            mailto:[hidden email]

_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi Herbert,

On 25. 10. 2010 22:17, Herbert König wrote:

> What I actually did was creating one million datapoints in my app in
> an attempt to measure cog's performance. Actually I did it twice
> because the first time the jitting of methods used up the performance.
> I got 4600 ms in Squeak 3.8 with 3.7 VM and 3400 and 984ms in trunk with cog.
>
> I'll continue this because I'm interested in the performance of Aida +
> my app. Actually I still look for ideas of how to do this apart from
> wgetting the whole site from several computers.

Do you feel that Aida is slow somewhere and it is a time to start
optimizing it?

I'm specially wary about the "climbing the stack" methods, which find a
reference to some object through thisContext and then recursively up in
the stack. Like Object>>firstAppFromStack. Those methods are used more
and more frequenty, because are so handy and they simplify the code a lot.

Best regards
Janko


--
Janko Mivšek
AIDA/Web
Smalltalk Web Application Server
http://www.aidaweb.si
_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi Janko,

long post, I did some coding inbetween :-))
JM> Do you feel that Aida is slow somewhere and it is a time to start
JM> optimizing it?

right now I'm simply not qualified to make such observations. That's
why I plan to benchmark my app + Aida. In my old 3.8 image after
spending those 4 seconds to generate the data I found browsing the data
quick and responsive.

The one place I found slow I can attribute to how my app prepares the
data for Aida to display. This could be simply improved by some caching.

If you remember the "onClickDoAndUpdateMany" discussion with Alex
Baran, this also created something slow in Aida.

Imagine a WebGrid, 4 columns 10 rows. Each of these 40 elements needs
to AjaxUpdate 5 Elements on the page, including the Grid. So each of these
elements needs 5 AjaxUpdaters. Creating 240 times the more or less
identical javascript necessary for an AjaxUpdater and transferring all
these to the browser was slow. This happened on every click into the
table.

I changed that to have one single function like: (just count the
bytes)
<script type="text/javascript">function updProf(elementId){new Ajax.Updater('id4', '/ellingweb.html', {method: 'post', postBody: 'view=profile&&ajaxRequest&ajaxGetElementId=id4&aidaCtx=10786&blockElementId=' + elementId + '&blockName=onClickBlock', onComplete: function() {new Ajax.Updater('id16', '/ellingweb.html', {method: 'post', postBody: 'view=profile&&ajaxRequest&ajaxGetElementId=id16&aidaCtx=10786', onComplete: function() {new Ajax.Updater('id7', '/ellingweb.html', {method: 'post', postBody: 'view=profile&&ajaxRequest&ajaxGetElementId=id7&aidaCtx=10786', onComplete: function() { inCall = false; ; }, evalScripts: true});; }, evalScripts: true});; }, evalScripts: true}); };</script>

and an Element of the WebGrid now only renders as:
<td align="right"><span id="id28" onClick="updProf('id28');"; ">2.500</span></td>

whereas before it had the whole script in every onClick function.

This was a good speedup in response time to a single click of a single
user. I can share this on request otherwise I will propose it as
a change when it has left the status of a raw hack. (not this year I
assume)


JM> I'm specially wary about the "climbing the stack" methods, which find a
JM> reference to some object through thisContext and then recursively up in
JM> the stack. Like Object>>firstAppFromStack. Those methods are used more
JM> and more frequenty, because are so handy and they simplify the code a lot.

Do they show up in a message tally as a bottleneck? If you drill down
firstAppFromStack until Behaviour>>isKindOf: you'll find several (not
nested) loops. This looks slow but I learned to not trust my own
assessment "this looks slow".

For benchmarking I believe I have to change the place in swazoo where
the HTTPRequest objects are generated to collect all these.

Hey that's easy I just coded up a class RequestLog which has a
ClassVar to collect the request and one method to to add a request to
the log. Then I added:
RequestLog addRequest: request to HTTPConnection>readRequestFor:
after the request gets its timestamp set.

Then I'll log a single Session of me using my app (maybe tomorrow)

and then:

and replay this log from say 10 images (or one image running 10
HTTPclients) simultaneously and watch what happens to the response
times of the server if I increase the number of clients.

Do you have / know of such a tool? Something that gets a bunch of
HTTPRequests fires them at a server and retrieves the replies?

Or do you think the whole idea is bad?

If I have that, I could confidently tell you in what situations I want
Aida to be faster.

Cheers,

Herbert                            mailto:[hidden email]

_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida

Hi,


HK> and replay this log from say 10 images (or one image running 10
HK> HTTPclients) simultaneously and watch what happens to the response
HK> times of the server if I increase the number of clients.

ok, the requests contain id's which may be invalid in the next
session. Still I like the idea, maybe also log the response in the
first recording and try to match the id's before replaying the
requests.



Cheers,

Herbert                            mailto:[hidden email]

_______________________________________________
Aida mailing list
[hidden email]
http://lists.aidaweb.si/mailman/listinfo/aida