Downloading large files with a progress bar

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Downloading large files with a progress bar

Offray Vladimir Luna Cárdenas-2
Hi,

I'm using

myUrl asUrl saveContentsToFile: file

to download large files and is working fine. But I would like to add a
progress bar to show the progress to the user. Previously I used

===============
[: bar |
         bar title: aString.
         [
             "A classical ZnClient download with signalProgress: true;"
         ]
             on: HTTPProgress
             do: [ :progress |
                 progress isEmpty ifFalse: [ bar current: progress
percentage ].
                 progress resume ].
     ] asJob run.
===============

but I don't know how to signal progress with "saveContentsToFile".

Any hints?

Thanks,

Offray

Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Sven Van Caekenberghe-2
Hi Offray,

The actual implementation of #saveContentsToFile: can be found in ZnHttpSaveContentsToFile>>#performOperation where you can see how ZnClient is invoked. Just copy that with signalProgress: true added and you should be good.

Alternatively, you could wrap the first expression as follows:

ZnSignalProgress
  value: true
  during: [ myUrl asUrl saveContentsToFile: file ]

ZnSignalProgress is a DynamicVariable that is used by ZnClient to figure out if progress should be signalled. It takes precedence over the option #signalProgress. See #withProgressDo: if you are interested.

HTH,

Sven

> On 13 May 2016, at 00:14, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>
> Hi,
>
> I'm using
>
> myUrl asUrl saveContentsToFile: file
>
> to download large files and is working fine. But I would like to add a progress bar to show the progress to the user. Previously I used
>
> ===============
> [: bar |
>        bar title: aString.
>        [
>            "A classical ZnClient download with signalProgress: true;"
>        ]
>            on: HTTPProgress
>            do: [ :progress |
>                progress isEmpty ifFalse: [ bar current: progress percentage ].
>                progress resume ].
>    ] asJob run.
> ===============
>
> but I don't know how to signal progress with "saveContentsToFile".
>
> Any hints?
>
> Thanks,
>
> Offray
>


Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Offray Vladimir Luna Cárdenas-2
Hi Sven,

Thanks for your quick answer. I tried both approaches but I'm missing
something. See my last attempt at http://ws.stfx.eu/KBVBAP72IWOO with
the last one. I get a progress bar message, but not the advancements on
it. What I'm missing?

Cheers,

Offray

ps: The file I'm trying to download is covered by the Open Data Base
License.

On 12/05/16 17:24, Sven Van Caekenberghe wrote:

> Hi Offray,
>
> The actual implementation of #saveContentsToFile: can be found in ZnHttpSaveContentsToFile>>#performOperation where you can see how ZnClient is invoked. Just copy that with signalProgress: true added and you should be good.
>
> Alternatively, you could wrap the first expression as follows:
>
> ZnSignalProgress
>    value: true
>    during: [ myUrl asUrl saveContentsToFile: file ]
>
> ZnSignalProgress is a DynamicVariable that is used by ZnClient to figure out if progress should be signalled. It takes precedence over the option #signalProgress. See #withProgressDo: if you are interested.
>
> HTH,
>
> Sven
>
>> On 13 May 2016, at 00:14, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>
>> Hi,
>>
>> I'm using
>>
>> myUrl asUrl saveContentsToFile: file
>>
>> to download large files and is working fine. But I would like to add a progress bar to show the progress to the user. Previously I used
>>
>> ===============
>> [: bar |
>>         bar title: aString.
>>         [
>>             "A classical ZnClient download with signalProgress: true;"
>>         ]
>>             on: HTTPProgress
>>             do: [ :progress |
>>                 progress isEmpty ifFalse: [ bar current: progress percentage ].
>>                 progress resume ].
>>     ] asJob run.
>> ===============
>>
>> but I don't know how to signal progress with "saveContentsToFile".
>>
>> Any hints?
>>
>> Thanks,
>>
>> Offray
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Offray Vladimir Luna Cárdenas-2
Hi again,


This implementatin http://ws.stfx.eu/FSNU7TV3O5FY also shows a static
downloading progress bar without any download, but the download works.
Ofcourse any example with another url for a smilar large file could work
for me. I'm just testing with the file for this [1] (Soon a proper
announcement).

[1] http://mutabit.com/offray/blog/en/entry/panama-papers-1

Cheers,

Offray

On 12/05/16 18:00, Offray Vladimir Luna Cárdenas wrote:

> Hi Sven,
>
> Thanks for your quick answer. I tried both approaches but I'm missing
> something. See my last attempt at http://ws.stfx.eu/KBVBAP72IWOO with
> the last one. I get a progress bar message, but not the advancements
> on it. What I'm missing?
>
> Cheers,
>
> Offray
>
> ps: The file I'm trying to download is covered by the Open Data Base
> License.
>
> On 12/05/16 17:24, Sven Van Caekenberghe wrote:
>> Hi Offray,
>>
>> The actual implementation of #saveContentsToFile: can be found in
>> ZnHttpSaveContentsToFile>>#performOperation where you can see how
>> ZnClient is invoked. Just copy that with signalProgress: true added
>> and you should be good.
>>
>> Alternatively, you could wrap the first expression as follows:
>>
>> ZnSignalProgress
>>    value: true
>>    during: [ myUrl asUrl saveContentsToFile: file ]
>>
>> ZnSignalProgress is a DynamicVariable that is used by ZnClient to
>> figure out if progress should be signalled. It takes precedence over
>> the option #signalProgress. See #withProgressDo: if you are interested.
>>
>> HTH,
>>
>> Sven
>>
>>> On 13 May 2016, at 00:14, Offray Vladimir Luna Cárdenas
>>> <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I'm using
>>>
>>> myUrl asUrl saveContentsToFile: file
>>>
>>> to download large files and is working fine. But I would like to add
>>> a progress bar to show the progress to the user. Previously I used
>>>
>>> ===============
>>> [: bar |
>>>         bar title: aString.
>>>         [
>>>             "A classical ZnClient download with signalProgress: true;"
>>>         ]
>>>             on: HTTPProgress
>>>             do: [ :progress |
>>>                 progress isEmpty ifFalse: [ bar current: progress
>>> percentage ].
>>>                 progress resume ].
>>>     ] asJob run.
>>> ===============
>>>
>>> but I don't know how to signal progress with "saveContentsToFile".
>>>
>>> Any hints?
>>>
>>> Thanks,
>>>
>>> Offray
>>>
>>
>>
>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Sven Van Caekenberghe-2
In reply to this post by Offray Vladimir Luna Cárdenas-2
When the UI thread is really busy, updating the UI can be problematic.

An extra fork seems to help:

(FileLocator temp / 'PharoV30.sources') ensureDelete.

[ [ :bar |
                bar title: 'Downloading Sources...'.
                [
                        ZnClient new
                                url: 'http://files.pharo.org/sources/PharoV30.sources'; 
                                signalProgress: true;
                                downloadTo: FileLocator temp ]
                        on: HTTPProgress
                        do: [ :progress |
                                progress isEmpty ifFalse: [ bar current: progress percentage ].
                                progress resume ] ] asJob run ] fork.

> On 13 May 2016, at 01:00, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>
> Hi Sven,
>
> Thanks for your quick answer. I tried both approaches but I'm missing something. See my last attempt at http://ws.stfx.eu/KBVBAP72IWOO with the last one. I get a progress bar message, but not the advancements on it. What I'm missing?
>
> Cheers,
>
> Offray
>
> ps: The file I'm trying to download is covered by the Open Data Base License.
>
> On 12/05/16 17:24, Sven Van Caekenberghe wrote:
>> Hi Offray,
>>
>> The actual implementation of #saveContentsToFile: can be found in ZnHttpSaveContentsToFile>>#performOperation where you can see how ZnClient is invoked. Just copy that with signalProgress: true added and you should be good.
>>
>> Alternatively, you could wrap the first expression as follows:
>>
>> ZnSignalProgress
>>   value: true
>>   during: [ myUrl asUrl saveContentsToFile: file ]
>>
>> ZnSignalProgress is a DynamicVariable that is used by ZnClient to figure out if progress should be signalled. It takes precedence over the option #signalProgress. See #withProgressDo: if you are interested.
>>
>> HTH,
>>
>> Sven
>>
>>> On 13 May 2016, at 00:14, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I'm using
>>>
>>> myUrl asUrl saveContentsToFile: file
>>>
>>> to download large files and is working fine. But I would like to add a progress bar to show the progress to the user. Previously I used
>>>
>>> ===============
>>> [: bar |
>>>        bar title: aString.
>>>        [
>>>            "A classical ZnClient download with signalProgress: true;"
>>>        ]
>>>            on: HTTPProgress
>>>            do: [ :progress |
>>>                progress isEmpty ifFalse: [ bar current: progress percentage ].
>>>                progress resume ].
>>>    ] asJob run.
>>> ===============
>>>
>>> but I don't know how to signal progress with "saveContentsToFile".
>>>
>>> Any hints?
>>>
>>> Thanks,
>>>
>>> Offray
>>>
>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Sven Van Caekenberghe-2
In reply to this post by Offray Vladimir Luna Cárdenas-2

> On 13 May 2016, at 01:00, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>
> Hi Sven,
>
> Thanks for your quick answer. I tried both approaches but I'm missing something. See my last attempt at http://ws.stfx.eu/KBVBAP72IWOO with the last one.

That last code makes no sense and is not how the API is designed to be used.

> I get a progress bar message, but not the advancements on it. What I'm missing?
>
> Cheers,
>
> Offray
>
> ps: The file I'm trying to download is covered by the Open Data Base License.
>
> On 12/05/16 17:24, Sven Van Caekenberghe wrote:
>> Hi Offray,
>>
>> The actual implementation of #saveContentsToFile: can be found in ZnHttpSaveContentsToFile>>#performOperation where you can see how ZnClient is invoked. Just copy that with signalProgress: true added and you should be good.
>>
>> Alternatively, you could wrap the first expression as follows:
>>
>> ZnSignalProgress
>>   value: true
>>   during: [ myUrl asUrl saveContentsToFile: file ]
>>
>> ZnSignalProgress is a DynamicVariable that is used by ZnClient to figure out if progress should be signalled. It takes precedence over the option #signalProgress. See #withProgressDo: if you are interested.
>>
>> HTH,
>>
>> Sven
>>
>>> On 13 May 2016, at 00:14, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I'm using
>>>
>>> myUrl asUrl saveContentsToFile: file
>>>
>>> to download large files and is working fine. But I would like to add a progress bar to show the progress to the user. Previously I used
>>>
>>> ===============
>>> [: bar |
>>>        bar title: aString.
>>>        [
>>>            "A classical ZnClient download with signalProgress: true;"
>>>        ]
>>>            on: HTTPProgress
>>>            do: [ :progress |
>>>                progress isEmpty ifFalse: [ bar current: progress percentage ].
>>>                progress resume ].
>>>    ] asJob run.
>>> ===============
>>>
>>> but I don't know how to signal progress with "saveContentsToFile".
>>>
>>> Any hints?
>>>
>>> Thanks,
>>>
>>> Offray
>>>
>>
>>
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Offray Vladimir Luna Cárdenas-2
Hi,

On 13/05/16 00:35, Sven Van Caekenberghe wrote:
>> On 13 May 2016, at 01:00, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>
>> Hi Sven,
>>
>> Thanks for your quick answer. I tried both approaches but I'm missing something. See my last attempt at http://ws.stfx.eu/KBVBAP72IWOO with the last one.
> That last code makes no sense and is not how the API is designed to be used.

Sorry. I was tired and poking the code to see what was wrong. Your
example with fork and Pharo.sources work pretty well. I will implement it.

Thanks,

Offray

Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Offray Vladimir Luna Cárdenas-2
Hi again,

Umm.... this is kind of strange. When I try with the Sven code
downloading the Pharo, downloading progress bar works, but when I try
the follow one, download works and forking lets me use the environment,
as with Sven's but progress bar don't advance:

=================
OffshoreLeaksDB class >>downloadDatabase
     "I download the data of the panama papers from its page at the
DataHub community repository:
     https://datahub.io/dataset/panama-papers
     "
     [[ :bar |
         bar title: 'Downloading Database...'.
             [
                 ZnClient new
                     url:
'https://datahub.io/dataset/06f27df3-ec88-47ea-b428-7ec138f7835e/resource/50a9bda8-e44a-4aac-b265-d07fabde5612/download/offshore-leaks.sqlite.zip'; 

                     signalProgress: true;
                     downloadTo: FileLocator temp
             ]
                 on: HTTPProgress
                 do: [ :progress |
                     progress isEmpty ifFalse: [ bar current: progress
percentage ].
                     progress resume ] ] asJob run ] fork.
=================

As an alternative I'm thinking in creating a Roassal animation for
background processes, similar to the rolling circles in the browser,
tablet of phone, to show that something is advancing, but you still need
to wait... Anyway, if someone can tell me about why this is happening I
will really appreciate it.

Thanks,

Offray

On 13/05/16 13:47, Offray Vladimir Luna Cárdenas wrote:

> Hi,
>
> On 13/05/16 00:35, Sven Van Caekenberghe wrote:
>>> On 13 May 2016, at 01:00, Offray Vladimir Luna Cárdenas
>>> <[hidden email]> wrote:
>>>
>>> Hi Sven,
>>>
>>> Thanks for your quick answer. I tried both approaches but I'm
>>> missing something. See my last attempt at
>>> http://ws.stfx.eu/KBVBAP72IWOO with the last one.
>> That last code makes no sense and is not how the API is designed to
>> be used.
>
> Sorry. I was tired and poking the code to see what was wrong. Your
> example with fork and Pharo.sources work pretty well. I will implement
> it.
>
> Thanks,
>
> Offray
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Downloading large files with a progress bar

Offray Vladimir Luna Cárdenas-2
In reply to this post by Sven Van Caekenberghe-2
Hi,

This is the code that finally did it:

http://ws.stfx.eu/458U3WRKXAW1.

I was really puzzled by the progress bar not being updated despite of
the download being made (it took me several days to find what happen!).
The thing is that the total size of the file was not being delivered by
the datahub.io site, so there was no way to put it into the progress
bar. I used an alternative approach reading the actual size against the
total expected size (once you know the cause the solution took me a
couple of hours, mostly learning about SystemProgressBar and
JobProgressBar, the actual script took minutes.).

Thanks Sven and Johan for helping me to figure this out.

Cheers,

Offray


On 13/05/16 00:35, Sven Van Caekenberghe wrote:

> When the UI thread is really busy, updating the UI can be problematic.
>
> An extra fork seems to help:
>
> (FileLocator temp / 'PharoV30.sources') ensureDelete.
>
> [ [ :bar |
> bar title: 'Downloading Sources...'.
> [
> ZnClient new
> url: 'http://files.pharo.org/sources/PharoV30.sources';
> signalProgress: true;
> downloadTo: FileLocator temp ]
> on: HTTPProgress
> do: [ :progress |
> progress isEmpty ifFalse: [ bar current: progress percentage ].
> progress resume ] ] asJob run ] fork.
>
>> On 13 May 2016, at 01:00, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>
>> Hi Sven,
>>
>> Thanks for your quick answer. I tried both approaches but I'm missing something. See my last attempt at http://ws.stfx.eu/KBVBAP72IWOO with the last one. I get a progress bar message, but not the advancements on it. What I'm missing?
>>
>> Cheers,
>>
>> Offray
>>
>> ps: The file I'm trying to download is covered by the Open Data Base License.
>>
>> On 12/05/16 17:24, Sven Van Caekenberghe wrote:
>>> Hi Offray,
>>>
>>> The actual implementation of #saveContentsToFile: can be found in ZnHttpSaveContentsToFile>>#performOperation where you can see how ZnClient is invoked. Just copy that with signalProgress: true added and you should be good.
>>>
>>> Alternatively, you could wrap the first expression as follows:
>>>
>>> ZnSignalProgress
>>>    value: true
>>>    during: [ myUrl asUrl saveContentsToFile: file ]
>>>
>>> ZnSignalProgress is a DynamicVariable that is used by ZnClient to figure out if progress should be signalled. It takes precedence over the option #signalProgress. See #withProgressDo: if you are interested.
>>>
>>> HTH,
>>>
>>> Sven
>>>
>>>> On 13 May 2016, at 00:14, Offray Vladimir Luna Cárdenas <[hidden email]> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm using
>>>>
>>>> myUrl asUrl saveContentsToFile: file
>>>>
>>>> to download large files and is working fine. But I would like to add a progress bar to show the progress to the user. Previously I used
>>>>
>>>> ===============
>>>> [: bar |
>>>>         bar title: aString.
>>>>         [
>>>>             "A classical ZnClient download with signalProgress: true;"
>>>>         ]
>>>>             on: HTTPProgress
>>>>             do: [ :progress |
>>>>                 progress isEmpty ifFalse: [ bar current: progress percentage ].
>>>>                 progress resume ].
>>>>     ] asJob run.
>>>> ===============
>>>>
>>>> but I don't know how to signal progress with "saveContentsToFile".
>>>>
>>>> Any hints?
>>>>
>>>> Thanks,
>>>>
>>>> Offray
>>>>
>>>
>>
>