Optimizing Speed of Fetching Source Code

Looks like over 40% of the time is spent forming filenames from strings. We've also noticed this performance problem, in our case forming filenames for database files. Given that the filenames are already in the correct format, and don't need anything like #{VISUALWORKS} to be converted, it does seem a little odd that VW is so slow.

 

20% of the time is spent building a new encoded stream for ever individual character in the filename, to see if it #isBadCharacter:. Surely #chop:to: could make one encoded stream for the filename and use the same stream to test all the characters? Or then simply optimize the UTF16 case?

 

If we do that, 50% of the time for building a filename is in the primitive #getFileSystemAttributes:. I imagine this could be cached quite easily, based on a simple comparison of the first component of the string. We could clear the cache when the OS announces a new disk: http://msdn.microsoft.com/en-us/library/aa363215.aspx

 

Steve

---

From: [hidden email] on behalf of Runar Jordahl
Sent: Fri 04/11/2011 22:07
To: [hidden email]
Subject: [vwnc] Optimizing Speed of Fetching Source Code

For various internal tools we access source code of methods. These
tools spend 90% - 95% of their time getting the source code strings,
then 5% - 10% of total time processing them. In our image, getting the
source code for all methods takes around 16 seconds. This only covers
fetching source code, not getting the methods.

To see where the time is spent, try evaluating the following snippet:
TimeProfiler profile: [
        (Store.Registry bundleNamed: 'Base VisualWorks')
withAllContainedItems do: [:pundle |
                pundle isPackage ifTrue: [pundle methods do: [:method | method sourceCode]]]]

          75.4 [] in Store.MethodDescriptor>>sourceCode
            69.1 Behavior>>sourceCodeAt:
              69.1 Behavior>>sourceCodeForMethod:at:
(...)
                                    69.1 BlockClosure>>on:do:
                                      69.1 [] in
XMLSourceFileFormat>>methodSourceAt:in:
                                        69.1 XML.XMLParser>>parseElement:
                                          43.7 XML.XMLParser>>on:
(...)
                                                    43.7
LogicalFilename>>asFilename
                                                      43.7
CharacterArray>>asFilename
                                                        43.7 Filename
class>>named:
                                                          43.7
PCFilename class>>createInstanceNamed:
                                                            25.3
NTFSFilename class>>canonicalize:forFileSystemAttributes:
                                                              25.3
PCFilename class>>canonicalize:forFileSystemAttributes:
                                                                22.7
OrderedCollection>>do:
                                                                  22.7
[] in PCFilename class>>canonicalize:forFileSystemAttributes:

22.7 Filename class>>chop:to:

22.7 PCFilename class>>isBadCharacter:

Please note that we do not have any problems with the current
performance. It just would be nice if the speed could be improved
without too much work.

We are now starting to cache source code in the development image. For
our project, with 6000 classes, Seaside and VisualWorks Base, around
100 MB is used by the cache. This speeds up our internal tools that
work on source code.

I noticed that performance does not improve a lot when upgrading to a
solid-state drive. I assume most source files get cached by the
operating system in RAM.


Kind regards
Runar Jordahl
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Holger Guhl
-- 
Senior Consultant * Certified Scrum Master * [hidden email]
Tel: +49 231 9 75 99 21 * Fax: +49 231 9 75 99 20
Georg Heeg eK Dortmund
Handelsregister: Amtsgericht Dortmund  A 12812

Res98610.zip (10K) Download Attachment

Holger Guhl
-- 
Senior Consultant * Certified Scrum Master * [hidden email]
Tel: +49 231 9 75 99 21 * Fax: +49 231 9 75 99 20
Georg Heeg eK Dortmund
Handelsregister: Amtsgericht Dortmund  A 12812

I don't pretend to understand the nuances of either proposed solution, so I'll just ask if they properly handle links?

 

On Windows I've created link to directories so that I can access them as a subfolder of one drive when they physically reside on another. I don't know if Windows allows linking between partitions with different file systems (ntfs/fat32, etc), but if it does, then any solution should take this into account. I would think the same consideration should be made on unixy systems.

 

And are there any repercussions for remote references:

    \\myServer\someFolder\someFile.txt

 

or remote folders mapped as a local drive letter?
 

Dave Stevenson
[hidden email]

---

From: Steven Kelly [hidden email]
To: [hidden email]
Sent: Mon, November 7, 2011 5:56:19 AM
Subject: Re: [vwnc] Optimizing Speed of Fetching Source Code

That’s great, and very useful to us as developers. However, it achieves its speed up by avoiding a couple of calls to #asFilename, rather than by improving the speed of #asFilename. It’s thus only useful during development, and not to end users of VW applications. It would be great if the low-hanging fruit available in #getFileSystemAttributes: and encoded stream use could be harvested to improve overall performance, or at least performance in the commonest case: Windows and #UTF16.

 

Steve

 

From: Holger Guhl [[hidden email]]
Sent: 7. marraskuuta 2011 12:13
To: Steven Kelly
Cc: Runar Jordahl; [hidden email]
Subject: Re: [vwnc] Optimizing Speed of Fetching Source Code

 

I have observed this during development and Support work (AR 60922) and created a couple of solutions, one of them Res98610. With that solution the need to create filenames drops to zero.
Source code retrieval starts at some point with a file, but for accessing the XML formatted text the file is wrapped as InputSource. For some purposes it seemed necessary to keep the filename (although most recent method comments negate that). The filename access seems cheap, and the final implementation in FileConnection>>name (^... fileName asFilename asString) seems harmless. But in the concrete scenario the fileName is a PortableFilename where #asFilename is implemented as "self asResolvedString asFilename". The "funny" thing is that the new Filename is converted to a String and never used thereafter.
The Resolution knows that and discards the filename access. So caching filenames is not necessary at all.
The second part of the solution is avoiding a duplicate file access during class comment retrieval. This is more or less to make the solution complete, since class comment access is less frequent. Nevertheless, source code analysis tools may benefit from the optimization.
The Resolution has not yet been integrated. I guess there is a ton of more pressing things to do. So far I cannot tell about negative side effects. Maybe you want to give it a try...

Am 04.11.2011 23:15, schrieb Steven Kelly:

Looks like over 40% of the time is spent forming filenames from strings. We've also noticed this performance problem, in our case forming filenames for database files. Given that the filenames are already in the correct format, and don't need anything like #{VISUALWORKS} to be converted, it does seem a little odd that VW is so slow.

 

20% of the time is spent building a new encoded stream for ever individual character in the filename, to see if it #isBadCharacter:. Surely #chop:to: could make one encoded stream for the filename and use the same stream to test all the characters? Or then simply optimize the UTF16 case?

 

If we do that, 50% of the time for building a filename is in the primitive #getFileSystemAttributes:. I imagine this could be cached quite easily, based on a simple comparison of the first component of the string. We could clear the cache when the OS announces a new disk: http://msdn.microsoft.com/en-us/library/aa363215.aspx

 

Steve

---

From: [hidden email] on behalf of Runar Jordahl
Sent: Fri 04/11/2011 22:07
To: [hidden email]
Subject: [vwnc] Optimizing Speed of Fetching Source Code

For various internal tools we access source code of methods. These
tools spend 90% - 95% of their time getting the source code strings,
then 5% - 10% of total time processing them. In our image, getting the
source code for all methods takes around 16 seconds. This only covers
fetching source code, not getting the methods.

To see where the time is spent, try evaluating the following snippet:
TimeProfiler profile: [
        (Store.Registry bundleNamed: 'Base VisualWorks')
withAllContainedItems do: [:pundle |
                pundle isPackage ifTrue: [pundle methods do: [:method | method sourceCode]]]]

          75.4 [] in Store.MethodDescriptor>>sourceCode
            69.1 Behavior>>sourceCodeAt:
              69.1 Behavior>>sourceCodeForMethod:at:
(...)
                                    69.1 BlockClosure>>on:do:
                                      69.1 [] in
XMLSourceFileFormat>>methodSourceAt:in:
                                        69.1 XML.XMLParser>>parseElement:
                                          43.7 XML.XMLParser>>on:
(...)
                                                    43.7
LogicalFilename>>asFilename
                                                      43.7
CharacterArray>>asFilename
                                                        43.7 Filename
class>>named:
                                                          43.7
PCFilename class>>createInstanceNamed:
                                                            25.3
NTFSFilename class>>canonicalize:forFileSystemAttributes:
                                                              25.3
PCFilename class>>canonicalize:forFileSystemAttributes:
                                                                22.7
OrderedCollection>>do:
                                                                  22.7
[] in PCFilename class>>canonicalize:forFileSystemAttributes:

22.7 Filename class>>chop:to:

22.7 PCFilename class>>isBadCharacter:

Please note that we do not have any problems with the current
performance. It just would be nice if the speed could be improved
without too much work.

We are now starting to cache source code in the development image. For
our project, with 6000 classes, Seaside and VisualWorks Base, around
100 MB is used by the cache. This speeds up our internal tools that
work on source code.

I noticed that performance does not improve a lot when upgrading to a
solid-state drive. I assume most source files get cached by the
operating system in RAM.


Kind regards
Runar Jordahl
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

_______________________________________________

vwnc mailing list

[hidden email]

http://lists.cs.uiuc.edu/mailman/listinfo/vwnc

Holger Guhl

-- 

Senior Consultant * Certified Scrum Master * [hidden email]

Tel: +49 231 9 75 99 21 * Fax: +49 231 9 75 99 20

Georg Heeg eK Dortmund

Handelsregister: Amtsgericht Dortmund  A 12812

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc