Zinc Progress

we just discovered a strange side-effect of enabling progress report
on ZnClient when loading mc packages:
- with progress notification: ~1s
- without notification:       ~30ms

I didn't dig more into the details bug this is quite a diff :D, and most
probably the main cause that pharo loads configurations and mc stuff slowly.
The latest pharo2.0 release has the progress report disabled.

best
cami

On 14 September 2012 18:09, Camillo Bruni <[hidden email]> wrote:
Does Zn use Notifications? Perhaps all that time's spent crawling up
and down the stack.

frank

Not good ;-)

The actual signalling happens in ZnUtils class>>#streamFrom:to:[size:] every 16Kb using an HTTPProgress notification/exception.
Maybe that is too much or too fast; that would mean there are 32 notifications for downloading 512Kb.
Any suggestions on what would make more sense ?

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

On 14 Sep 2012, at 19:09, Camillo Bruni <[hidden email]> wrote:

Is it possible to notify every time interval? Say, every second or so.
That way the cost would also not vary with download/upload speed.

frank

On 14 September 2012 19:15, Sven Van Caekenberghe <[hidden email]> wrote:

On 14 Sep 2012, at 20:57, Frank Shearar <[hidden email]> wrote:

> Is it possible to notify every time interval? Say, every second or so.
> That way the cost would also not vary with download/upload speed.

Good idea, it will take some tweaking to get this to work well, I guess.
I remember that some changes like this were done for the unit tests progress as well, I'll try find that code.

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

On 2012-09-14, at 23:11, Sven Van Caekenberghe <[hidden email]> wrote:
> On 14 Sep 2012, at 20:57, Frank Shearar <[hidden email]> wrote:
>
>> Is it possible to notify every time interval? Say, every second or so.
>> That way the cost would also not vary with download/upload speed.
>
> Good idea, it will take some tweaking to get this to work well, I guess.
> I remember that some changes like this were done for the unit tests progress as well, I'll try find that code.

I still think it's not the amount of notifications that causes the pauses
there must be more to it... given that the download took around 1000ms longer
we would have spend more than 30ms per notification! no-way :)

running the download in the profiler showed the 1s lost is spent in the idle
process ;) IMO that's rather funny :)

On 15 Sep 2012, at 01:02, Camillo Bruni <[hidden email]> wrote:

I wrote some code to try to have something executable to work with, but I can't find anything.

(The examples use the extended version of HTTPProgress, http://code.google.com/p/pharo/issues/detail?id=6674 )

I created 1Mb and 10Mb files somewhere public. Full progress is currently only available in Zn when doing stream, as in #downloadTo:

'/tmp/10Mb.bin' asFileReference ensureDeleted.

[ ZnClient new
        systemPolicy;
        beOneShot;
        url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin';
        downloadTo: '/tmp/' ] timeToRun.

[ UIManager default informUserDuring: [ :bar |
        bar label: 'Downloading...'.
        [ ZnClient new
                signalProgress: true;
                systemPolicy;
                beOneShot;
                url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/10Mb.bin';
                downloadTo: '/tmp/' ]
                        on: HTTPProgress
                        do: [ :progress |
                                bar label: progress printString.
                                progress isEmpty ifFalse: [ bar current: progress percentage ].
                                self crLog: progress printString.
                                progress resume ] ] ] timeToRun.

As fas as I can tell, both run at the same speed (more or less).

Is is possible to show the code where you noticed the slowdown ?

Sven

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

I assume that the overhead only occurs for small files...


ZnClient new
        signalProgress: true;
        systemPolicy;
        beOneShot;
        url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main?format=raw';
        get.

with progress 2.5secs without, 57ms.. :/ well might be that the server
responds with an incomplete header or so...


 

On 2012-09-16, at 21:50, Camillo Bruni <[hidden email]> wrote:

another remark with signalProgress the time to run highly varies, 250ms - 2500ms.
whereas without progress notification it stays <140ms

the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a
on: Exception do:[ ... retry ]. so that means on an HTTPProgress you
initiate a retry? (and wait for retryDelay), AUCH! :D

I think that I integrated some of the changes of camillo yesterday.

Stef

On Sep 16, 2012, at 9:50 PM, Camillo Bruni wrote:

On 16 Sep 2012, at 21:59, Camillo Bruni <[hidden email]> wrote:

Yeah, that is probably it, and that is horrible, terrible. .. ;-)
In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly.
Tomorrow, I'll have another look, with a fresh head.

Thx again!

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

>> the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a
>> on: Exception do:[ ... retry ]. so that means on an HTTPProgress you
>> initiate a retry? (and wait for retryDelay), AUCH! :D
>
> Yeah, that is probably it, and that is horrible, terrible. .. ;-)
> In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly.
> Tomorrow, I'll have another look, with a fresh head.

cool ;) I assume a simple Exception => Error will do :P

On 16 Sep 2012, at 23:14, Camillo Bruni <[hidden email]> wrote:

>>> the culprit IMO is ZnClient>>#executeWithRetriesRemaining: which does a
>>> on: Exception do:[ ... retry ]. so that means on an HTTPProgress you
>>> initiate a retry? (and wait for retryDelay), AUCH! :D
>>
>> Yeah, that is probably it, and that is horrible, terrible. .. ;-)
>> In #executeWithTimeout a special #exceptionSetForIfFail is used to avoid that, if I remember correctly.
>> Tomorrow, I'll have another look, with a fresh head.
>
> cool ;) I assume a simple Exception => Error will do :P

Well, no ;-) I think it would be better to only do a retry on NetworkErrors and ZnParseErrors.
On the other hand, the ifFail functionality should trigger on any Error. We'll see.
Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-)

Sven

Sven Van Caekenberghe uploaded a new version of Zinc-HTTP to project Zinc HTTP Components:
http://www.squeaksource.com/ZincHTTPComponents/Zinc-HTTP-SvenVanCaekenberghe.301.mcz

==================== Summary ====================

Name: Zinc-HTTP-SvenVanCaekenberghe.301
Author: SvenVanCaekenberghe
Time: 17 September 2012, 10:10:49 am
UUID: 85632c09-a6c4-40e9-b29b-1c5e86d07ead
Ancestors: Zinc-HTTP-SvenVanCaekenberghe.300

Fixed a bug where HTTPProgress notifications would trigger a retry.
Thanks Camillo Bruni for finding this problem and suggesting a solution.
Now, retries are only triggered by (NetworkError, ZnParseError), while the #ifFailBlock will be trigger on any Error.
Furthermore, #noteRetrying: and noteIgnoringExceptionOnReusedConnection: will report on the actual exception.
The default #ifFailBlock is now [ :exception | exception pass ] for some cleaner code.

--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill

On 17 Sep 2012, at 10:18, Sven Van Caekenberghe <[hidden email]> wrote:

> Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-)

After fixing this in my working image, all MC operations just fly.
I am happy and sad at the same time. Sorry guys.

Sven

On 2012-09-17, at 11:05, Sven Van Caekenberghe <[hidden email]> wrote:
> On 17 Sep 2012, at 10:18, Sven Van Caekenberghe <[hidden email]> wrote:
>
>> Thanks again for the help, you made Pharo faster by removing my explicit Delays ;-)
>
> After fixing this in my working image, all MC operations just fly.
> I am happy and sad at the same time. Sorry guys.

haha :D if you have a nice design a performance issue per year is allowed :P

http://code.google.com/p/pharo/issues/detail?id=6687

:)