Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo
Status: Accepted
Owner: ----
CC: [hidden email]
Labels: Milestone-2.0 Type-Feature

New issue 6277 by [hidden email]: PharoV10.sources files should be  
loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

The pharo sources files should be downloaded when it is not found.

We propose to put it in a default location related to the preferences

      FileLocator preferences / 'pharo' / 'PharoV10.sources'




_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #1 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

I think it would be better to get rid of the .sources and .changes and just  
have a transaction log for code...


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #2 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

On the other hand it would be cool:
  - we no longer have to distribute .sources explicitely
  - when somebody misses, them for whatever reason, its automagically  
downloaded

ZnClient new
        systemPolicy;
        url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/PharoV10.sources';
        downloadTo: FileSystem disk workingDirectory.

The above link serves the uncompressed version, we should set up a server  
that does a gzip HTTP tranfer.



_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #3 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

We should do that :)! I put up the files on our update site...

http://pharo.gforge.inria.fr/ci/image/PharoV10.sources.zip
http://pharo.gforge.inria.fr/ci/image/PharoV10.sources

(though amazon will most certainly outperform the availability :P)


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #4 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

Almost ;-)

The goal is that the Apache server is configured such that it will do a  
Content-Encoding:gzip on the uncompressed PharoV10.sources file. The HTTP  
client (ZnClient in our case, or curl) will then decompress on the fly and  
treat the result as a download of PharoV10.sources and not of  
PharoV10.sources.zip which we would then have to decompress (not that it is  
hard to do, it is just reinventing the wheel). The difference is roughly  
16Mb to 4Mb on the wire.

I made 1 minor change to my Apache 2 on AWS (Ubuntu) and now it works as  
expected:

[ | client |
'/tmp/PharoV10.sources' asFileReference ensureDeleted.
(client := ZnClient new)
        systemPolicy;
        logToTranscript;
        url: 'http://stfx.eu/PharoV10.sources'.
client request setAcceptEncodingGzip.
client downloadTo: '/tmp/' ] timeToRun.  2395

(BTW, I will add #setAcceptEncodingGzip directly to ZnClient).

With the following Transcript output:

2012-09-30 16:56:50 866792 I Wrote a ZnRequest(GET /PharoV10.sources)
2012-09-30 16:56:50 866792 D Sent headers
Accept: */*
User-Agent: Zinc HTTP Components 1.0
Accept-Encoding: gzip
Host: stfx.eu

2012-09-30 16:56:50 866792 I Read a ZnResponse(200 OK text/plain )
2012-09-30 16:56:50 866792 D Received headers
Content-Encoding: gzip
Content-Length: nil
Vary: Accept-Encoding
Accept-Ranges: bytes
Last-Modified: Sun, 30 Sep 2012 14:34:10 GMT
Content-Type: text/plain
Etag: "20200-f7bb6c-4caec2ea5e2fe"
Date: Sun, 30 Sep 2012 14:52:22 GMT
Transfer-Encoding: chunked
Server: Apache/2.2.22 (Ubuntu)

2012-09-30 16:56:50 866792 T GET /PharoV10.sources 200 nilB 273ms

(The request/response time is so low because the actual tranfer happens  
outside the transaction in a streaming fashion, the length is nil because  
it is not specified, #upToEnd thus.)

Here is a curl call to show the difference with the INRIA server:

$ curl -X HEAD -D - -H 'Accept-Encoding:gzip'  
http://pharo.gforge.inria.fr/ci/image/PharoV10.sources

HTTP/1.1 200 OK
Date: Sun, 30 Sep 2012 15:01:45 GMT
Server: Apache/2.2.16 (Debian)
Last-Modified: Sun, 30 Sep 2012 10:52:07 GMT
ETag: "6080bc-f7bb6c-4cae9148f882d"
Accept-Ranges: bytes
Content-Length: 16235372
Content-Type: text/plain

$ curl -X HEAD -D - -H 'Accept-Encoding: gzip'  
http://stfx.eu/PharoV10.sources
HTTP/1.1 200 OK
Date: Sun, 30 Sep 2012 14:58:02 GMT
Server: Apache/2.2.22 (Ubuntu)
Last-Modified: Sun, 30 Sep 2012 14:34:10 GMT
ETag: "20200-f7bb6c-4caec2ea5e2fe"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: text/plain

Check the Apache 2 modules mime and deflate to make sure that .source is  
text/plain and that all text/plain gets compressed when the client asks for  
it.




_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #5 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

I changed the contents type to text/plain, seems like compression is  
enabled now :)

curl -X HEAD -D - -H 'Accept-Encoding:gzip'  
http://pharo.gforge.inria.fr/ci/image/PharoV10.sources
HTTP/1.1 200 OK
Date: Sun, 30 Sep 2012 16:01:43 GMT
Server: Apache/2.2.16 (Debian)
Last-Modified: Sun, 30 Sep 2012 10:52:07 GMT
ETag: "6080bc-f7bb6c-4cae9148f882d"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: text/plain


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #6 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

BTW would it make sense to enable gzip compression by default for ZnClient?


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #7 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

Yes is works !

We could even use the INRIA one as primary and mine as secondary.

So where do we have to add code for this to work ?

I suppose we will just attempt to download when the file is missing, and  
when the download fails, we report more or less the same error as today,  
right ?

What is the best way to get at the directory where the .source file should  
be put ?



_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #8 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

@comment 6

I am not sure, browsers do that, but curl does not, but it kind of depends  
on how good/efficient our gzip decompression is.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #9 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

perfect.
download: SO yeah, first try our mirror, then yours (so you save some cent$  
on aws ;).
warning: exactly, first try the 2 locations, then fallback as before
location:
     FileLocator imageDirectory / 'PharoV10.sources'
     FileLocator vmDirectory / 'PharoV10.sources'.
     though I'd prefer some more common location for shared data...
    FileLocator preferences maybe?

comment: see issue 6751, yet another improvement


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #10 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

OK, I'll give it a shot.

Let's make it simple for now and download to vmDirectory, trying 2 URLs.

The sources or the package-cache are not really preferences, more shared  
application data, for which there is not yet a predefined location.


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo
Updates:
        Status: FixReviewNeeded

Comment #11 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

Name:  
SLICE-Issue-6277-PharoV10sources-files-should-be-loaded-on-demand-SvenVanCaekenberghe.1
Author: SvenVanCaekenberghe
Time: 30 September 2012, 8:16:23.185 pm
UUID: ba33eaa7-71e0-4c60-8f63-80387c9e8a76
Ancestors:
Dependencies: System-Support-SvenVanCaekenberghe.713

modify SmalltalkImage>>#checkAndOpenSourcesAndChanges to call  
#downloadSources when nothing is found at the regular locations that are  
being checked;
add SmalltalkImage>>#downloadSources to try downloading a sources file from  
2 known locations into a shared directory, currently #vmDirectory

-----------------

I tested on Mac OS X and it worked well.

The image startup naturally hangs for a second or two before properly  
drawing the world, I think it is too early then to show a dialog. Maybe we  
could do this as a background task, I guess the presences of the sources  
file is not really critical until actually browsing code. Let's take it one  
step at a time.

We can now start saving 3.65 MB compressed data from each download all over  
the place, for everyone.



_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo
Updates:
        Status: Integrated

Comment #12 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

in 2.0 317


_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker
Reply | Threaded
Open this post in threaded view
|

Re: Issue 6277 in pharo: PharoV10.sources files should be loaded on demand

pharo

Comment #13 on issue 6277 by [hidden email]: PharoV10.sources files  
should be loaded on demand
http://code.google.com/p/pharo/issues/detail?id=6277

Thanks!

We'll have to improve on this a bit later on because there are two failure  
scenarios that we have to deal with:
- there is no internet, you need a proxy that is not set in the image, or  
whatever
- the vmDirectory is not writable
For the first situation I think we need a very low timeout, like maybe 2s
For the second I don't know ;-)



_______________________________________________
Pharo-bugtracker mailing list
[hidden email]
http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/pharo-bugtracker