Status: Accepted
Owner: ---- CC: [hidden email] Labels: Milestone-2.0 Type-Feature New issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 The pharo sources files should be downloaded when it is not found. We propose to put it in a default location related to the preferences FileLocator preferences / 'pharo' / 'PharoV10.sources' _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #1 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 I think it would be better to get rid of the .sources and .changes and just have a transaction log for code... _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #2 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 On the other hand it would be cool: - we no longer have to distribute .sources explicitely - when somebody misses, them for whatever reason, its automagically downloaded ZnClient new systemPolicy; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/PharoV10.sources'; downloadTo: FileSystem disk workingDirectory. The above link serves the uncompressed version, we should set up a server that does a gzip HTTP tranfer. _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #3 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 We should do that :)! I put up the files on our update site... http://pharo.gforge.inria.fr/ci/image/PharoV10.sources.zip http://pharo.gforge.inria.fr/ci/image/PharoV10.sources (though amazon will most certainly outperform the availability :P) _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #4 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 Almost ;-) The goal is that the Apache server is configured such that it will do a Content-Encoding:gzip on the uncompressed PharoV10.sources file. The HTTP client (ZnClient in our case, or curl) will then decompress on the fly and treat the result as a download of PharoV10.sources and not of PharoV10.sources.zip which we would then have to decompress (not that it is hard to do, it is just reinventing the wheel). The difference is roughly 16Mb to 4Mb on the wire. I made 1 minor change to my Apache 2 on AWS (Ubuntu) and now it works as expected: [ | client | '/tmp/PharoV10.sources' asFileReference ensureDeleted. (client := ZnClient new) systemPolicy; logToTranscript; url: 'http://stfx.eu/PharoV10.sources'. client request setAcceptEncodingGzip. client downloadTo: '/tmp/' ] timeToRun. 2395 (BTW, I will add #setAcceptEncodingGzip directly to ZnClient). With the following Transcript output: 2012-09-30 16:56:50 866792 I Wrote a ZnRequest(GET /PharoV10.sources) 2012-09-30 16:56:50 866792 D Sent headers Accept: */* User-Agent: Zinc HTTP Components 1.0 Accept-Encoding: gzip Host: stfx.eu 2012-09-30 16:56:50 866792 I Read a ZnResponse(200 OK text/plain ) 2012-09-30 16:56:50 866792 D Received headers Content-Encoding: gzip Content-Length: nil Vary: Accept-Encoding Accept-Ranges: bytes Last-Modified: Sun, 30 Sep 2012 14:34:10 GMT Content-Type: text/plain Etag: "20200-f7bb6c-4caec2ea5e2fe" Date: Sun, 30 Sep 2012 14:52:22 GMT Transfer-Encoding: chunked Server: Apache/2.2.22 (Ubuntu) 2012-09-30 16:56:50 866792 T GET /PharoV10.sources 200 nilB 273ms (The request/response time is so low because the actual tranfer happens outside the transaction in a streaming fashion, the length is nil because it is not specified, #upToEnd thus.) Here is a curl call to show the difference with the INRIA server: $ curl -X HEAD -D - -H 'Accept-Encoding:gzip' http://pharo.gforge.inria.fr/ci/image/PharoV10.sources HTTP/1.1 200 OK Date: Sun, 30 Sep 2012 15:01:45 GMT Server: Apache/2.2.16 (Debian) Last-Modified: Sun, 30 Sep 2012 10:52:07 GMT ETag: "6080bc-f7bb6c-4cae9148f882d" Accept-Ranges: bytes Content-Length: 16235372 Content-Type: text/plain $ curl -X HEAD -D - -H 'Accept-Encoding: gzip' http://stfx.eu/PharoV10.sources HTTP/1.1 200 OK Date: Sun, 30 Sep 2012 14:58:02 GMT Server: Apache/2.2.22 (Ubuntu) Last-Modified: Sun, 30 Sep 2012 14:34:10 GMT ETag: "20200-f7bb6c-4caec2ea5e2fe" Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Content-Type: text/plain Check the Apache 2 modules mime and deflate to make sure that .source is text/plain and that all text/plain gets compressed when the client asks for it. _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #5 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 I changed the contents type to text/plain, seems like compression is enabled now :) curl -X HEAD -D - -H 'Accept-Encoding:gzip' http://pharo.gforge.inria.fr/ci/image/PharoV10.sources HTTP/1.1 200 OK Date: Sun, 30 Sep 2012 16:01:43 GMT Server: Apache/2.2.16 (Debian) Last-Modified: Sun, 30 Sep 2012 10:52:07 GMT ETag: "6080bc-f7bb6c-4cae9148f882d" Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Content-Type: text/plain _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #6 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 BTW would it make sense to enable gzip compression by default for ZnClient? _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #7 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 Yes is works ! We could even use the INRIA one as primary and mine as secondary. So where do we have to add code for this to work ? I suppose we will just attempt to download when the file is missing, and when the download fails, we report more or less the same error as today, right ? What is the best way to get at the directory where the .source file should be put ? _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #8 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 @comment 6 I am not sure, browsers do that, but curl does not, but it kind of depends on how good/efficient our gzip decompression is. _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #9 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 perfect. download: SO yeah, first try our mirror, then yours (so you save some cent$ on aws ;). warning: exactly, first try the 2 locations, then fallback as before location: FileLocator imageDirectory / 'PharoV10.sources' FileLocator vmDirectory / 'PharoV10.sources'. though I'd prefer some more common location for shared data... FileLocator preferences maybe? comment: see issue 6751, yet another improvement _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #10 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 OK, I'll give it a shot. Let's make it simple for now and download to vmDirectory, trying 2 URLs. The sources or the package-cache are not really preferences, more shared application data, for which there is not yet a predefined location. _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Updates:
Status: FixReviewNeeded Comment #11 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 Name: SLICE-Issue-6277-PharoV10sources-files-should-be-loaded-on-demand-SvenVanCaekenberghe.1 Author: SvenVanCaekenberghe Time: 30 September 2012, 8:16:23.185 pm UUID: ba33eaa7-71e0-4c60-8f63-80387c9e8a76 Ancestors: Dependencies: System-Support-SvenVanCaekenberghe.713 modify SmalltalkImage>>#checkAndOpenSourcesAndChanges to call #downloadSources when nothing is found at the regular locations that are being checked; add SmalltalkImage>>#downloadSources to try downloading a sources file from 2 known locations into a shared directory, currently #vmDirectory ----------------- I tested on Mac OS X and it worked well. The image startup naturally hangs for a second or two before properly drawing the world, I think it is too early then to show a dialog. Maybe we could do this as a background task, I guess the presences of the sources file is not really critical until actually browsing code. Let's take it one step at a time. We can now start saving 3.65 MB compressed data from each download all over the place, for everyone. _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Updates:
Status: Integrated Comment #12 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 in 2.0 317 _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Comment #13 on issue 6277 by [hidden email]: PharoV10.sources files should be loaded on demand http://code.google.com/p/pharo/issues/detail?id=6277 Thanks! We'll have to improve on this a bit later on because there are two failure scenarios that we have to deal with: - there is no internet, you need a proxy that is not set in the image, or whatever - the vmDirectory is not writable For the first situation I think we need a very low timeout, like maybe 2s For the second I don't know ;-) _______________________________________________ Pharo-bugtracker mailing list [hidden email] http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker |
Free forum by Nabble | Edit this page |