[ENH]: Zinc - Streaming Download Progress

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[ENH]: Zinc - Streaming Download Progress

Sean P. DeNigris
Administrator
It would be really nice if Zinc sent notifications of streaming download progress to log listeners and HTTPProgress handlers.

The following method showed nice progress notifications via both routes until the file started to download, which is the bulk of the time spent...

downloadLatestVmMakerImage

        | client |
        client := ZnClient new.
        client signalProgress: true.
        client log addListener: self.
       
        [client
                url: 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/artifact/vmmaker-image.zip';
                downloadTo: '/Volumes/Fast/Squeak/Cog/vmmaker-image.zip'.]
        on: HTTPProgress
        do: [ :ex |
                Transcript show: ex asString; cr.
                ex resume ]
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe
Hi Sean,

Yeah, that would be nice.

But do you agree that the only solution is to use a dynamic variable ?
Because signalling code has to known whether there is someone catching the notifications, right ?

The #signalProgress option in ZnClient was just the first step, the interface.
The real, interesting signalling has to happen in ZnUtils class>>#streamFrom:to:size:

If you agree that the dynamic variable is the way to go, I can quite easily implement it.

Sven

On 12 Jul 2012, at 17:58, Sean P. DeNigris wrote:

> It would be /really/ nice if Zinc sent notifications of streaming download
> progress to log listeners and HTTPProgress handlers.
>
> The following method showed nice progress notifications via both routes
> until the file started to download, which is the bulk of the time spent...
>
> downloadLatestVmMakerImage
>
> | client |
> client := ZnClient new.
> client signalProgress: true.
> client log addListener: self.
>
> [client
> url:
> 'https://ci.lille.inria.fr/pharo/job/Cog%20Git%20Tracker/lastSuccessfulBuild/artifact/vmmaker-image.zip';
> downloadTo: '/Volumes/Fast/Squeak/Cog/vmmaker-image.zip'.]
> on: HTTPProgress
> do: [ :ex |
> Transcript show: ex asString; cr.
> ex resume ]
>
> --
> View this message in context: http://forum.world.st/ENH-Zinc-Streaming-Download-Progress-tp4639686.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>


Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sean P. DeNigris
Administrator
Sven Van Caekenberghe wrote
But do you agree that the only solution is to use a dynamic variable ?
Because signalling code has to known whether there is someone catching the notifications, right ?
Could you say more about that? Googling "dynamic variable" turned up academic stuff. Also, could Announcements be helpful here?

Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Frank Shearar-3
On 12 July 2012 17:40, Sean P. DeNigris <[hidden email]> wrote:
>
> Sven Van Caekenberghe wrote
>>
>> But do you agree that the only solution is to use a dynamic variable ?
>> Because signalling code has to known whether there is someone catching the
>> notifications, right ?
>>
> Could you say more about that? Googling "dynamic variable" turned up
> academic stuff. Also, could Announcements be helpful here?

You've used them all the time: they're just like (shell) environment
variables, only the environment variables can nest. So in a default
Squeak image (which has the DynamicVariables package by default),

MyDynVar value: 1 during: [
  Transcript showln: (MyDynVar value printString).
  MyDynVar value: 2 during: [
    Transcript showln: (MyDynVar value printString).
  ].
  Transcript showln: (MyDynVar value printString).
]

will output

1
2
1

There is a gotcha with naively implementing dynamic variables: they
_do not_ play nicely with delimited continuations:
http://www.lshift.net/blog/2012/06/27/resumable-exceptions-can-macro-express-delimited-dynamic-variables
shows a solution which should Just Work in Pharo ( and if it doesn't,
let me know, and I'll make it work). The link also has links
explaining in detail why the two don't work nicely. This bad
interaction will bite you as soon as you do anything interesting to
the stack, and given that you always have thisContext just a reference
away, that could be any time you like!

Oh. I see I _didn't_ put a link to the code. Sigh.
http://ss3.gemstone.com/ss/Control.html is the place to get the code.

frank

> Sean
>
> --
> View this message in context: http://forum.world.st/ENH-Zinc-Streaming-Download-Progress-tp4639686p4639695.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe
In reply to this post by Sean P. DeNigris

On 12 Jul 2012, at 18:40, Sean P. DeNigris wrote:

> Could you say more about that? Googling "dynamic variable" turned up
> academic stuff. Also, could Announcements be helpful here?

There is already a usecase in ZnConnectionTimeout.

Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sean P. DeNigris
Administrator
Sven Van Caekenberghe wrote
There is already a usecase in ZnConnectionTimeout.
Okay, I checked out the code. How does that concept apply here? Maybe a quick example?

Thanks,
Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe
It is better explained with code ;-)

With:

Sven Van Caekenberghe uploaded a new version of Zinc-HTTP to project Zinc HTTP Components:
http://www.squeaksource.com/ZincHTTPComponents/Zinc-HTTP-SvenVanCaekenberghe.285.mcz

==================== Summary ====================

Name: Zinc-HTTP-SvenVanCaekenberghe.285
Author: SvenVanCaekenberghe
Time: 12 July 2012, 9:58 am
UUID: e992fd76-efde-4b31-b4b4-bd468f8176f2
Ancestors: Zinc-HTTP-SvenVanCaekenberghe.284

enabled HTTPProgress signalling during streaming up/downloads.
introduction of ZnSignalProgress with #enabled method

=================================================

You should now be able to do something like:

UIManager default informUserDuring: [ :bar |
        [
                 '/Users/sven/Desktop/test-99425.txt' asFileReference ensureDeleted.
                ^ ZnClient new
                        signalProgress: true;
                        url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/test-99425.txt';
                        downloadTo: '/Users/sven/Desktop/' ]
                on: HTTPProgress
                do: [ :progress |
                        bar label: progress printString.
                        progress total ifNotNil: [
                                        1 second asDelay wait.
                                        bar current: (progress amount / progress total) * 100 ].
                        progress resume ] ].

Sven

On 12 Jul 2012, at 19:11, Sean P. DeNigris wrote:

>
> Sven Van Caekenberghe wrote
>>
>> There is already a usecase in ZnConnectionTimeout.
>>
>
> Okay, I checked out the code. How does that concept apply here? Maybe a
> quick example?
>
> Thanks,
> Sean
>
> --
> View this message in context: http://forum.world.st/ENH-Zinc-Streaming-Download-Progress-tp4639686p4639711.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>


Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Frank Shearar-3
On 12 July 2012 21:00, Sven Van Caekenberghe <[hidden email]> wrote:

> It is better explained with code ;-)
>
> With:
>
> Sven Van Caekenberghe uploaded a new version of Zinc-HTTP to project Zinc HTTP Components:
> http://www.squeaksource.com/ZincHTTPComponents/Zinc-HTTP-SvenVanCaekenberghe.285.mcz
>
> ==================== Summary ====================
>
> Name: Zinc-HTTP-SvenVanCaekenberghe.285
> Author: SvenVanCaekenberghe
> Time: 12 July 2012, 9:58 am
> UUID: e992fd76-efde-4b31-b4b4-bd468f8176f2
> Ancestors: Zinc-HTTP-SvenVanCaekenberghe.284
>
> enabled HTTPProgress signalling during streaming up/downloads.
> introduction of ZnSignalProgress with #enabled method
>
> =================================================
>
> You should now be able to do something like:
>
> UIManager default informUserDuring: [ :bar |
>         [
>                  '/Users/sven/Desktop/test-99425.txt' asFileReference ensureDeleted.
>                 ^ ZnClient new
>                         signalProgress: true;
>                         url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/test-99425.txt';
>                         downloadTo: '/Users/sven/Desktop/' ]
>                 on: HTTPProgress
>                 do: [ :progress |
>                         bar label: progress printString.
>                         progress total ifNotNil: [
>                                         1 second asDelay wait.
>                                         bar current: (progress amount / progress total) * 100 ].
>                         progress resume ] ].
>
> Sven

Yes: my Control library just provides a touch more formalism than
arbitrary resumable exceptions, but it's otherwise just that.

frank

> On 12 Jul 2012, at 19:11, Sean P. DeNigris wrote:
>
>>
>> Sven Van Caekenberghe wrote
>>>
>>> There is already a usecase in ZnConnectionTimeout.
>>>
>>
>> Okay, I checked out the code. How does that concept apply here? Maybe a
>> quick example?
>>
>> Thanks,
>> Sean
>>
>> --
>> View this message in context: http://forum.world.st/ENH-Zinc-Streaming-Download-Progress-tp4639686p4639711.html
>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe
Frank,

When I have more time, I sure have to read your stuff, it seems quite interesting.

Sven

On 12 Jul 2012, at 22:56, Frank Shearar wrote:

> On 12 July 2012 21:00, Sven Van Caekenberghe <[hidden email]> wrote:
>> It is better explained with code ;-)
>>
>> With:
>>
>> Sven Van Caekenberghe uploaded a new version of Zinc-HTTP to project Zinc HTTP Components:
>> http://www.squeaksource.com/ZincHTTPComponents/Zinc-HTTP-SvenVanCaekenberghe.285.mcz
>>
>> ==================== Summary ====================
>>
>> Name: Zinc-HTTP-SvenVanCaekenberghe.285
>> Author: SvenVanCaekenberghe
>> Time: 12 July 2012, 9:58 am
>> UUID: e992fd76-efde-4b31-b4b4-bd468f8176f2
>> Ancestors: Zinc-HTTP-SvenVanCaekenberghe.284
>>
>> enabled HTTPProgress signalling during streaming up/downloads.
>> introduction of ZnSignalProgress with #enabled method
>>
>> =================================================
>>
>> You should now be able to do something like:
>>
>> UIManager default informUserDuring: [ :bar |
>>        [
>>                 '/Users/sven/Desktop/test-99425.txt' asFileReference ensureDeleted.
>>                ^ ZnClient new
>>                        signalProgress: true;
>>                        url: 'http://s3-eu-west-1.amazonaws.com/public-stfx-eu/test-99425.txt';
>>                        downloadTo: '/Users/sven/Desktop/' ]
>>                on: HTTPProgress
>>                do: [ :progress |
>>                        bar label: progress printString.
>>                        progress total ifNotNil: [
>>                                        1 second asDelay wait.
>>                                        bar current: (progress amount / progress total) * 100 ].
>>                        progress resume ] ].
>>
>> Sven
>
> Yes: my Control library just provides a touch more formalism than
> arbitrary resumable exceptions, but it's otherwise just that.
>
> frank
>
>> On 12 Jul 2012, at 19:11, Sean P. DeNigris wrote:
>>
>>>
>>> Sven Van Caekenberghe wrote
>>>>
>>>> There is already a usecase in ZnConnectionTimeout.
>>>>
>>>
>>> Okay, I checked out the code. How does that concept apply here? Maybe a
>>> quick example?
>>>
>>> Thanks,
>>> Sean
>>>
>>> --
>>> View this message in context: http://forum.world.st/ENH-Zinc-Streaming-Download-Progress-tp4639686p4639711.html
>>> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>>>
>>
>>
>


Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sean P. DeNigris
Administrator
In reply to this post by Sven Van Caekenberghe
Sven Van Caekenberghe wrote
You should now be able to do something like...
That didn't seem to work in my Pharo 1.4 image with 284 loaded. Does it require something else? I noticed the only method in my image that seems to set the progress total for downloading is HTTPSocket>>getRestOfBuffer:totalLength:, which has no senders in my image.

Thanks!
Sean
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe
Did you load Zinc-HTTP-SvenVanCaekenberghe.285 ?

On 13 Jul 2012, at 18:22, Sean P. DeNigris wrote:

>
> Sven Van Caekenberghe wrote
>>
>> You should now be able to do something like...
>>
>
> That didn't seem to work in my Pharo 1.4 image with 284 loaded. Does it
> require something else? I noticed the only method in my image that seems to
> set the progress total for downloading is
> HTTPSocket>>getRestOfBuffer:totalLength:, which has no senders in my image.
>
> Thanks!
> Sean
>
> --
> View this message in context: http://forum.world.st/ENH-Zinc-Streaming-Download-Progress-tp4639686p4639880.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>


Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sean P. DeNigris
Administrator
On Jul 13, 2012, at 12:48 PM, "Sven Van Caekenberghe [via Smalltalk]" <[hidden email]> wrote:

> Did you load Zinc-HTTP-SvenVanCaekenberghe.285 ?
Ah, that's what happened. I loaded the latest version from sqs, which seems to be 284
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe
It is hard to keep repos in sync manually…

On 13 Jul 2012, at 19:04, Sean P. DeNigris wrote:

> On Jul 13, 2012, at 12:48 PM, "Sven Van Caekenberghe [via Smalltalk]" <[hidden email]> wrote:
>
> > Did you load Zinc-HTTP-SvenVanCaekenberghe.285 ?
> Ah, that's what happened. I loaded the latest version from sqs, which seems to be 284
> View this message in context: Re: [ENH]: Zinc - Streaming Download Progress
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.


Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sean P. DeNigris
Administrator
Sven Van Caekenberghe wrote
It is hard to keep repos in sync manually…
Oh, I forgot about http://mc.stfx.eu/ZincHTTPComponents.... Is it better to try there first, in general?

_________________________


Cool! It works great :) One question - why exceptions instead of announcements?
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe

On 13 Jul 2012, at 21:21, Sean P. DeNigris wrote:

> Oh, I forgot about http://mc.stfx.eu/ZincHTTPComponents.... Is it better to
> try there first, in general?

Yes, that is my primary one, but I normally copy immediately to SqS, unless under hacking stress

> _________________________
>
>
> Cool! It works great :) One question - why exceptions instead of
> announcements?

Uhh, HTTPProgress was already there. I believe all progress is done that way, no ?

With Announcements the problem is that you have to specify an Announcer, Notifications are just thrown in the air ;-)
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sean P. DeNigris
Administrator
Sven Van Caekenberghe wrote
Uhh, HTTPProgress was already there. I believe all progress is done that way, no ?
Oh, okay. Using exceptions for non-exceptional conditions seems like a hack. My first cut at cleaning up the SystemProgressMorph was in preparation to re-implement using announcements instead.

Sven Van Caekenberghe wrote
With Announcements the problem is that you have to specify an Announcer, Notifications are just thrown in the air ;-)
How about an announcer per client that also forwards to one for the whole Zinc library?
Cheers,
Sean
Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Sven Van Caekenberghe
Sean,

On 13 Jul 2012, at 22:23, Sean P. DeNigris wrote:

> Sven Van Caekenberghe wrote
>>
>> Uhh, HTTPProgress was already there. I believe all progress is done that way, no ?

> Oh, okay. Using exceptions for non-exceptional conditions seems like a hack.
> My first cut at cleaning up the SystemProgressMorph was in preparation to
> re-implement using announcements instead.

Yes, it seems like a hack.

> Sven Van Caekenberghe wrote
>>
>> With Announcements the problem is that you have to specify an Announcer,
>> Notifications are just thrown in the air ;-)

> How about an announcer per client that also forwards to one for the whole Zinc library?

I think I understand what you are trying to achieve, yes it makes sense.
But this feels like a architectural decision - I don't feel qualified to decide.

Anybody else care to comment ?

Sven


--
Sven Van Caekenberghe
http://stfx.eu
Smalltalk is the Red Pill





Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Frank Shearar-3
In reply to this post by Sean P. DeNigris
On 13 July 2012 21:23, Sean P. DeNigris <[hidden email]> wrote:
>
> Sven Van Caekenberghe wrote
>>
>> Uhh, HTTPProgress was already there. I believe all progress is done that
>> way, no ?
>>
> Oh, okay. Using exceptions for non-exceptional conditions seems like a hack.
> My first cut at cleaning up the SystemProgressMorph was in preparation to
> re-implement using announcements instead.

Well, using Exceptions for non-exceptional conditions is a hack. But
these aren't, they're Notifications, which are entirely different!

Seriously though, resumable exceptions are at the core of how
Squeak/Pharo work. They're really useful.

frank

> Sven Van Caekenberghe wrote
>>
>> With Announcements the problem is that you have to specify an Announcer,
>> Notifications are just thrown in the air ;-)
>>
> How about an announcer per client that also forwards to one for the whole
> Zinc library?
>
> --
> View this message in context: http://forum.world.st/ENH-Zinc-Streaming-Download-Progress-tp4639686p4639930.html
> Sent from the Pharo Smalltalk mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Stéphane Ducasse
In reply to this post by Sven Van Caekenberghe
>> Sven Van Caekenberghe wrote
>>>
>>> With Announcements the problem is that you have to specify an Announcer,
>>> Notifications are just thrown in the air ;-)
>
>> How about an announcer per client that also forwards to one for the whole Zinc library?
>
> I think I understand what you are trying to achieve, yes it makes sense.
> But this feels like a architectural decision - I don't feel qualified to decide.
>
> Anybody else care to comment ?

I did not get it.

Stef


Reply | Threaded
Open this post in threaded view
|

Re: [ENH]: Zinc - Streaming Download Progress

Stéphane Ducasse
In reply to this post by Frank Shearar-3
>
> Well, using Exceptions for non-exceptional conditions is a hack. But
> these aren't, they're Notifications, which are entirely different!
>
> Seriously though, resumable exceptions are at the core of how
> Squeak/Pharo work. They're really useful.
>
> frank

I agree

12