Problem with ZipArchive (probably bug due to large file)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with ZipArchive (probably bug due to large file)

Mariano Martinez Peck
Hi guys,

I discovered a problem with ZipArchive when using Metacello to download a zipball of a github project. The project in question is HighchartsSt (soon we will make a new release and ANN) and this repo has *a lot* of autogenerated code. By a lot, I mean that the ZIP file is about 90MB. 

I do understand the recommended approach with git is to clone (either externally or with Iceberg), then work with your local clone via  filetree or tonel, and finally lock it. When you do this, ZipArchive is not used and hence I have no problem. 

However, even if not the recommended path, I know there are people still getting projects with Metacello from remote urls from github and hence the zipball download is in play. 

The problem in question is an error reading the zip signature. We believe it happens only with very big zips (smaller zips do work) and one way to reproduce it is as simple as this:


    outputFileName := '/tmp/test.zip'.
    ZnClient new
        downloadTo: outputFileName.
     ZipArchive new 
        readFrom: outputFileName asFileReference


Some info I could get is that the zip itself looks good:

Curl is able to download:


 ❯ ls -lah highcharts.zip                                                                 [23:18:06]
-rw-r--r-- 1 mariano staff 79M Dec 18 23:14 highcharts.zip

and zip seems valid:

 ❯ unzip -q -t highcharts.zip                                                             [23:18:33]
No errors detected in compressed data of highcharts.zip.


Anyway, if there is someone with some knowledge in ZipArchive that could give us a hand....

Thank you very much in advance,



--
Reply | Threaded
Open this post in threaded view
|

Re: Problem with ZipArchive (probably bug due to large file)

Pavel Krivanek-3
Pharo does not support Zip64 file format so it cannot work with big files nor files with huge amount of files.

-- Pavel

2017-12-21 12:54 GMT+01:00 Mariano Martinez Peck <[hidden email]>:
Hi guys,

I discovered a problem with ZipArchive when using Metacello to download a zipball of a github project. The project in question is HighchartsSt (soon we will make a new release and ANN) and this repo has *a lot* of autogenerated code. By a lot, I mean that the ZIP file is about 90MB. 

I do understand the recommended approach with git is to clone (either externally or with Iceberg), then work with your local clone via  filetree or tonel, and finally lock it. When you do this, ZipArchive is not used and hence I have no problem. 

However, even if not the recommended path, I know there are people still getting projects with Metacello from remote urls from github and hence the zipball download is in play. 

The problem in question is an error reading the zip signature. We believe it happens only with very big zips (smaller zips do work) and one way to reproduce it is as simple as this:


    outputFileName := '/tmp/test.zip'.
    ZnClient new
        downloadTo: outputFileName.
     ZipArchive new 
        readFrom: outputFileName asFileReference


Some info I could get is that the zip itself looks good:

Curl is able to download:


 ❯ ls -lah highcharts.zip                                                                 [23:18:06]
-rw-r--r-- 1 mariano staff 79M Dec 18 23:14 highcharts.zip

and zip seems valid:

 ❯ unzip -q -t highcharts.zip                                                             [23:18:33]
No errors detected in compressed data of highcharts.zip.


Anyway, if there is someone with some knowledge in ZipArchive that could give us a hand....

Thank you very much in advance,



--