Issue 5222 in pharo: mc slowdown with big package-cache

Status: Accepted
Owner: [hidden email]

New issue 5222 by [hidden email]: mc slowdown with big package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

I use a shared package-cache to avoid multiple downloading of the same  
files.
By now its some 3'000 files and 450 MB large which slows down monticello  
significantly.

When checking if a file has already been cached before it has to walk all  
the files in the package-cache.
A cache miss / hit takes around 1.5seconds on my machine.

1. Fix the FilePlugin primitives to get the file properties for a specific  
path without iterating
2. Fix Monticello to not do "self allFileNames includes: ..." but rather  
check directly


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #1 on issue 5222 by marianopeck: mc slowdown with big package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

Hi cami. My package cache is also 700MB and it is unusable now...the worst  
thing ever is to take a repo and say "add to package" .... go to drink a  
beer and when you come back MC is still working :(

Please CAmi, anything you can do to improve this is more than welcome. I  
was so much happy with my global package cache...until I found these  
problems :(


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #2 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

I have a hack working on my machine that speeds up the cache-hits  
significantly.

But for decent support we will have to update the VM for OS X / Cocoa since  
one FS plugin was not properly implemented. Since jenkins is down I cannot  
properly test this :P.

I will attach a first solution in a moment.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #3 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

first fix using FileStream >> isAFileNamed: which should already give an  
impressive speed up

Attachments:
        mc-package-cache-improvement.1.cs  972 bytes


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: FixToInclude
        Labels: Milestone-1.4

Comment #4 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

Ok I will integrate the cs now. Is it ok?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Integrated

Comment #5 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

in 14298


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Workneeded

Comment #6 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

After adding this, now it seems that MC warns for existing files where it  
didn't use to.

Result: all builds broken on Jenkins




_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #7 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

hmm ok then we will have to go the long way...

- make sure the blessed VM includes the latest fixes to FilePlugin
- update the system to use the old (but on OSX incomplete)  
directoryLookupEntry primitive
- fix the MC implementation


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Comment #8 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

I'm a moron :D, obviously I should pass along the absolute path and not  
just the mcz filename :/

includesFileNamed: aString
        "HACK: speed up the cache hits"
        ^ FileStream isAFileNamed: (directory fullPathFor: aString)

will do.



Attachments:
        mc-package-cache-improvement.2.cs  994 bytes


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker

Updates:
        Status: Integrated

Comment #9 on issue 5222 by [hidden email]: mc slowdown with big  
package-cache
http://code.google.com/p/pharo/issues/detail?id=5222

added as a change set to the update stream.




_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker