we just discovered a strange side-effect of enabling progress report
on ZnClient when loading mc packages: - with progress notification: ~1s - without notification: ~30ms I didn't dig more into the details bug this is quite a diff :D, and most probably the main cause that pharo loads configurations and mc stuff slowly. The latest pharo2.0 release has the progress report disabled. best cami |
On 14 September 2012 18:09, Camillo Bruni <[hidden email]> wrote:
> we just discovered a strange side-effect of enabling progress report > on ZnClient when loading mc packages: > - with progress notification: ~1s > - without notification: ~30ms > > I didn't dig more into the details bug this is quite a diff :D, and most > probably the main cause that pharo loads configurations and mc stuff slowly. > The latest pharo2.0 release has the progress report disabled. > > best > cami Does Zn use Notifications? Perhaps all that time's spent crawling up and down the stack. frank |
In reply to this post by Camillo Bruni-3
Not good ;-)
The actual signalling happens in ZnUtils class>>#streamFrom:to:[size:] every 16Kb using an HTTPProgress notification/exception. Maybe that is too much or too fast; that would mean there are 32 notifications for downloading 512Kb. Any suggestions on what would make more sense ? -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill On 14 Sep 2012, at 19:09, Camillo Bruni <[hidden email]> wrote: > we just discovered a strange side-effect of enabling progress report > on ZnClient when loading mc packages: > - with progress notification: ~1s > - without notification: ~30ms > > I didn't dig more into the details bug this is quite a diff :D, and most > probably the main cause that pharo loads configurations and mc stuff slowly. > The latest pharo2.0 release has the progress report disabled. > > best > cami |
Is it possible to notify every time interval? Say, every second or so.
That way the cost would also not vary with download/upload speed. frank On 14 September 2012 19:15, Sven Van Caekenberghe <[hidden email]> wrote: > Not good ;-) > > The actual signalling happens in ZnUtils class>>#streamFrom:to:[size:] every 16Kb using an HTTPProgress notification/exception. > Maybe that is too much or too fast; that would mean there are 32 notifications for downloading 512Kb. > Any suggestions on what would make more sense ? > > -- > Sven Van Caekenberghe > http://stfx.eu > Smalltalk is the Red Pill > > On 14 Sep 2012, at 19:09, Camillo Bruni <[hidden email]> wrote: > >> we just discovered a strange side-effect of enabling progress report >> on ZnClient when loading mc packages: >> - with progress notification: ~1s >> - without notification: ~30ms >> >> I didn't dig more into the details bug this is quite a diff :D, and most >> probably the main cause that pharo loads configurations and mc stuff slowly. >> The latest pharo2.0 release has the progress report disabled. >> >> best >> cami > > |
On 14 Sep 2012, at 20:57, Frank Shearar <[hidden email]> wrote: > Is it possible to notify every time interval? Say, every second or so. > That way the cost would also not vary with download/upload speed. Good idea, it will take some tweaking to get this to work well, I guess. I remember that some changes like this were done for the unit tests progress as well, I'll try find that code. -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill |
On 2012-09-14, at 23:11, Sven Van Caekenberghe <[hidden email]> wrote: > On 14 Sep 2012, at 20:57, Frank Shearar <[hidden email]> wrote: > >> Is it possible to notify every time interval? Say, every second or so. >> That way the cost would also not vary with download/upload speed. > > Good idea, it will take some tweaking to get this to work well, I guess. > I remember that some changes like this were done for the unit tests progress as well, I'll try find that code. I still think it's not the amount of notifications that causes the pauses there must be more to it... given that the download took around 1000ms longer we would have spend more than 30ms per notification! no-way :) running the download in the profiler showed the 1s lost is spent in the idle process ;) IMO that's rather funny :) |
On 15 Sep 2012, at 01:02, Camillo Bruni <[hidden email]> wrote: > On 2012-09-14, at 23:11, Sven Van Caekenberghe <[hidden email]> wrote: >> On 14 Sep 2012, at 20:57, Frank Shearar <[hidden email]> wrote: >> >>> Is it possible to notify every time interval? Say, every second or so. >>> That way the cost would also not vary with download/upload speed. >> >> Good idea, it will take some tweaking to get this to work well, I guess. >> I remember that some changes like this were done for the unit tests progress as well, I'll try find that code. > > I still think it's not the amount of notifications that causes the pauses > there must be more to it... given that the download took around 1000ms longer > we would have spend more than 30ms per notification! no-way :) > > running the download in the profiler showed the 1s lost is spent in the idle > process ;) IMO that's rather funny :) I wrote some code to try to have something executable to work with, but I can't find anything. (The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 ) I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo: '/tmp/10Mb.bin' asFileReference ensureDeleted. [ ZnClient new systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] timeToRun. [ UIManager default informUserDuring: [ :bar | bar label: 'Downloading...'. [ ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; downloadTo: '/tmp/' ] on: HTTPProgress do: [ :progress | bar label: progress printString. progress isEmpty ifFalse: [ bar current: progress percentage ]. self crLog: progress printString. progress resume ] ] ] timeToRun. As fas as I can tell, both run at the same speed (more or less). Is is possible to show the code where you noticed the slowdown ? Sven -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill |
> I wrote some code to try to have something executable to work with, but I can't find anything.
> > (The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 ) > > I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo: > > '/tmp/10Mb.bin' asFileReference ensureDeleted. > > [ ZnClient new > systemPolicy; > beOneShot; > url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; > downloadTo: '/tmp/' ] timeToRun. > > [ UIManager default informUserDuring: [ :bar | > bar label: 'Downloading...'. > [ ZnClient new > signalProgress: true; > systemPolicy; > beOneShot; > url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; > downloadTo: '/tmp/' ] > on: HTTPProgress > do: [ :progress | > bar label: progress printString. > progress isEmpty ifFalse: [ bar current: progress percentage ]. > self crLog: progress printString. > progress resume ] ] ] timeToRun. > > As fas as I can tell, both run at the same speed (more or less). > > Is is possible to show the code where you noticed the slowdown ? I assume that the overhead only occurs for small files... ZnClient new signalProgress: true; systemPolicy; beOneShot; url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; get. with progress 2.5secs without, 57ms.. :/ well might be that the server responds with an incomplete header or so... |
On 2012-09-16, at 21:50, Camillo Bruni <[hidden email]> wrote: >> I wrote some code to try to have something executable to work with, but I can't find anything. >> >> (The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 ) >> >> I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo: >> >> '/tmp/10Mb.bin' asFileReference ensureDeleted. >> >> [ ZnClient new >> systemPolicy; >> beOneShot; >> url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; >> downloadTo: '/tmp/' ] timeToRun. >> >> [ UIManager default informUserDuring: [ :bar | >> bar label: 'Downloading...'. >> [ ZnClient new >> signalProgress: true; >> systemPolicy; >> beOneShot; >> url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; >> downloadTo: '/tmp/' ] >> on: HTTPProgress >> do: [ :progress | >> bar label: progress printString. >> progress isEmpty ifFalse: [ bar current: progress percentage ]. >> self crLog: progress printString. >> progress resume ] ] ] timeToRun. >> >> As fas as I can tell, both run at the same speed (more or less). >> >> Is is possible to show the code where you noticed the slowdown ? > > I assume that the overhead only occurs for small files... > > > ZnClient new > signalProgress: true; > systemPolicy; > beOneShot; > url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; > get. > > with progress 2.5secs without, 57ms.. :/ well might be that the server > responds with an incomplete header or so... another remark with signalProgress the time to run highly varies, 250ms - 2500ms. whereas without progress notification it stays <140ms |
>> I assume that the overhead only occurs for small files...
>> >> >> ZnClient new >> signalProgress: true; >> systemPolicy; >> beOneShot; >> url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; >> get. >> >> with progress 2.5secs without, 57ms.. :/ well might be that the server >> responds with an incomplete header or so... > > another remark with signalProgress the time to run highly varies, 250ms - 2500ms. > whereas without progress notification it stays <140ms the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a on: Exception do:[ ... retry ]. so that means on an HTTPProgress you initiate a retry? (and wait for retryDelay), AUCH! :D |
In reply to this post by Camillo Bruni-3
I think that I integrated some of the changes of camillo yesterday.
Stef On Sep 16, 2012, at 9:50 PM, Camillo Bruni wrote: >> I wrote some code to try to have something executable to work with, but I can't find anything. >> >> (The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 ) >> >> I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo: >> >> '/tmp/10Mb.bin' asFileReference ensureDeleted. >> >> [ ZnClient new >> systemPolicy; >> beOneShot; >> url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; >> downloadTo: '/tmp/' ] timeToRun. >> >> [ UIManager default informUserDuring: [ :bar | >> bar label: 'Downloading...'. >> [ ZnClient new >> signalProgress: true; >> systemPolicy; >> beOneShot; >> url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin'; >> downloadTo: '/tmp/' ] >> on: HTTPProgress >> do: [ :progress | >> bar label: progress printString. >> progress isEmpty ifFalse: [ bar current: progress percentage ]. >> self crLog: progress printString. >> progress resume ] ] ] timeToRun. >> >> As fas as I can tell, both run at the same speed (more or less). >> >> Is is possible to show the code where you noticed the slowdown ? > > I assume that the overhead only occurs for small files... > > > ZnClient new > signalProgress: true; > systemPolicy; > beOneShot; > url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; > get. > > with progress 2.5secs without, 57ms.. :/ well might be that the server > responds with an incomplete header or so... > > > > > |
In reply to this post by Camillo Bruni-3
On 16 Sep 2012, at 21:59, Camillo Bruni <[hidden email]> wrote:
>>> I assume that the overhead only occurs for small files... >>> >>> ZnClient new >>> signalProgress: true; >>> systemPolicy; >>> beOneShot; >>> url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw'; >>> get. >>> >>> with progress 2.5secs without, 57ms.. :/ well might be that the server >>> responds with an incomplete header or so... >> >> another remark with signalProgress the time to run highly varies, 250ms - 2500ms. >> whereas without progress notification it stays <140ms > > the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a > on: Exception do:[ ... retry ]. so that means on an HTTPProgress you > initiate a retry? (and wait for retryDelay), AUCH! :D Yeah, that is probably it, and that is horrible, terrible. .. ;-) In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly. Tomorrow, I'll have another look, with a fresh head. Thx again! -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill |
>> the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a
>> on: Exception do:[ ... retry ]. so that means on an HTTPProgress you >> initiate a retry? (and wait for retryDelay), AUCH! :D > > Yeah, that is probably it, and that is horrible, terrible. .. ;-) > In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly. > Tomorrow, I'll have another look, with a fresh head. cool ;) I assume a simple Exception => Error will do :P |
On 16 Sep 2012, at 23:14, Camillo Bruni <[hidden email]> wrote:
>>> the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a >>> on: Exception do:[ ... retry ]. so that means on an HTTPProgress you >>> initiate a retry? (and wait for retryDelay), AUCH! :D >> >> Yeah, that is probably it, and that is horrible, terrible. .. ;-) >> In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly. >> Tomorrow, I'll have another look, with a fresh head. > > cool ;) I assume a simple Exception => Error will do :P Well, no ;-) I think it would be better to only do a retry on NetworkErrors and ZnParseErrors. On the other hand, the ifFail functionality should trigger on any Error. We'll see. Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-) Sven Sven Van Caekenberghe uploaded a new version of Zinc-HTTP to project Zinc HTTP Components: http://www.squeaksource.com/ZincHTTPComponents/Zinc-HTTP-SvenVanCaekenberghe.301.mcz ==================== Summary ==================== Name: Zinc-HTTP-SvenVanCaekenberghe.301 Author: SvenVanCaekenberghe Time: 17 September 2012, 10:10:49 am UUID: 85632c09-a6c4-40e9-b29b-1c5e86d07ead Ancestors: Zinc-HTTP-SvenVanCaekenberghe.300 Fixed a bug where HTTPProgress notifications would trigger a retry. Thanks Camillo Bruni for finding this problem and suggesting a solution. Now, retries are only triggered by (NetworkError, ZnParseError), while the #ifFailBlock will be trigger on any Error. Furthermore, #noteRetrying: and noteIgnoringExceptionOnReusedConnection: will report on the actual exception. The default #ifFailBlock is now [ :exception | exception pass ] for some cleaner code. -- Sven Van Caekenberghe http://stfx.eu Smalltalk is the Red Pill |
On 17 Sep 2012, at 10:18, Sven Van Caekenberghe <[hidden email]> wrote: > Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-) After fixing this in my working image, all MC operations just fly. I am happy and sad at the same time. Sorry guys. Sven |
On 2012-09-17, at 11:05, Sven Van Caekenberghe <[hidden email]> wrote:
> On 17 Sep 2012, at 10:18, Sven Van Caekenberghe <[hidden email]> wrote: > >> Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-) > > After fixing this in my working image, all MC operations just fly. > I am happy and sad at the same time. Sorry guys. haha :D if you have a nice design a performance issue per year is allowed :P |
Free forum by Nabble | Edit this page |