Fetch zip file from web, unzip, save constituent parts?

I’ve got dizzy trying to dig through all the remote file and server directory and URI and Url andHttp* classes. Do we really have three completely different ways of handling files out on the web?

What I need to do is access a zip file on a server, download it and save a directory of the constituent parts. The contained files are assorted gifs, jpegs, wav etc media files. Ideally I’d like to decode those without even saving the separate files and then re-loading them but let’s see if I can just get the data and unzip it first.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Why do we want intelligent terminals when there are so many stupid users?

> On 16-06-2016, at 5:22 PM, tim Rowledge <[hidden email]> wrote:
>
> What I need to do is access a zip file on a server, download it

OK, that bit I can now do with “http://foo.com/thingy.zip' asUrl retrieveContents, which even makes sense.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Never test for an error condition you don't know how to handle.

> On 16-06-2016, at 5:41 PM, tim Rowledge <[hidden email]> wrote:
>
>
>> On 16-06-2016, at 5:22 PM, tim Rowledge <[hidden email]> wrote:
>>
>> What I need to do is access a zip file on a server, download it
>
> OK, that bit I can now do with “http://foo.com/thingy.zip' asUrl retrieveContents, which even makes sense.

Except that any attempt to get a zip file from github, for example, fails because of some problem with unknown protocol. THat’s even after discovering that github wants 'https://github.com/foo/bar/master/thing.zip?raw=true' in order to actually see the data and not the damn page.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DST: Deadlock System Tables

On 17.06.2016, at 03:03, tim Rowledge <[hidden email]> wrote:

Or even:

https://raw.githubusercontent.com/foo/bar/master/thing.zip

Best
        -Tobias


> tim

> On 17-06-2016, at 6:15 AM, karl ramberg <[hidden email]> wrote:
>
> With projects you can do this in the etoys image. No need really for any files other than the project/ image segment. But getting the projects/ image segments to to work in a trunk image could be tricky ;-)

In many ways I’d love to use an image segment, especially if there is a way to include one on squeaksource etc. However, the best route - I think - it to download the zip file provided by a third party and either save it to local storage, unzip it and read in, or directly split it in-image and save trying to write local. That would save any worries about permissions.

I may be over-thinking this I suppose. Perhaps the simplest answer is to make sure the initialisation code is ok with *not* having those files there and make sure to warn the user to fetch them, unzip, and run some method to load up? I do like the idea of magically doing it though...

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
29A, the hexadecimal of the Beast.

> On 17-06-2016, at 12:11 AM, Tobias Pape <[hidden email]> wrote:
>
> https://raw.githubusercontent.com/foo/bar/master/thing.zip

That works better, thanks. It also helps if you have the ssl plugin working, as I discovered while trying this out on my iMac instead of the Pi. So that means I have to see if it even gets built on a Pi…

So far (just for the record) this is showing promise -
fetch the data with
        'https://raw.githubusercontent.com/user/repo/branch/myimportant.zip' asUrl retrieveContents
then extract with
        UIManager default informUserDuring: [:bar| (ZipArchive new readFrom: (ReadStream on: content asByteArray)) extractAllTo: (FileDirectory default) informing: bar].
where ‘content’ is the content from the MIMEDocument you get back. The files are all extracted ok.

Next step is to see how I can ‘load’ each file more directly.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: PSM: Print and SMear

> On 17-06-2016, at 10:54 AM, tim Rowledge <[hidden email]> wrote:
>
>
> That works better, thanks. It also helps if you have the ssl plugin working, as I discovered while trying this out on my iMac instead of the Pi. So that means I have to see if it even gets built on a Pi…

Looks like the ssl plugin is indeed built and used on my Pi but fails somehow. The ssl tests all claim to succeed though.

The error is reported in the stdout stream as
1996092684:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:795:

It appears to be a fairly common issue in some manner (quite a lot of google results) but I have no experience at all with ssl and so on. Can’t find any of the plausible looking words in the vm sources so far. That’s going to make it interesting to try catching in gdb.

Oh, and actually I see Chris M reported the same error last July so at least it’s nothing I’ve recently caused...


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
.signature not found!  reformat hard drive? [Yn]

On 17.06.2016, at 20:49, tim Rowledge <[hidden email]> wrote:

Mea culpa, maybe.

The State of SqueakSSL is not satisfying but in the last month I hadn't
had enough time on my hand to delve in to the pit that Proper SSL is (hint: it's a mess).

I had prepared SqueakSSL binaries for Linux[1] that are linked statically agains LibreSSL
(To avoid the Linux equivalent of DLL hell: different .so-names for RedHat/Centos and Debian-based systems).
There, the SSL23_GET_SERVER_HELLO:unknown protocol: error does not show.
Alas, these are x86 (32bit) only, yet, and integrating this thing into the main vm generation process
posed as hard for me. I'll tackle that again.

Also, there are 2..3 bugs in WebClient[2] and the image-Part of SqueakSSL.
Also, verification only really works on Windows.


Please keep poking me.

Best regards
        -Tobias


[1]: https://github.com/squeak-smalltalk/squeakssl/releases
[2]: One is fixed in WebClient-topa.101

> On 17-06-2016, at 12:03 PM, Tobias Pape <[hidden email]> wrote:
>
> The State of SqueakSSL is not satisfying but in the last month I hadn't
> had enough time on my hand to delve in to the pit that Proper SSL is (hint: it's a mess).

I wouldn’t know enough to be able to tell :-)

Since Raspbian is based on Debian it is possible that the ssl libraries are older and may lack bugfixes? The good news is that the mac version is fine so the basics of the code clearly get it mostly right.

For now I think I’ll have to go with alerting the user when attempting to load my package and get them to manually download & unzip the relevant files.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- Permanently out to lunch.