I revisited my MonticelloProjects experiment yesterday, and am still crashing the vm when serializing to a compressed fuel file. I wrote about it at the time, and then gave up as the vm was less stable during that period. http://forum.world.st/Working-with-a-compressed-Fuel-file-td4869105.html and it looks like the issue referenced: http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html To reproduce in a pharo6 image: |project| Gofer it smalltalkhubUser: 'StephanEggermont' project: 'MonticelloProjects'; package: 'MonticelloProjects'; load. project := (Smalltalk at: #MCProject) new location: 'http://smalltalkhub.com/mc/StephanEggermont/Documentation/main'. project read. project saveToFile. That reliably crashes in saveToFile. It seems to be independent of 32/64 bit and mac/linux, as I tried some variations. Stephan
|
Any more information needed? Stephan |
> On Jul 2, 2017, at 5:38 AM, stephan <[hidden email]> wrote: > > Any more information needed? No, just time :-/ > > Stephan > |
> Op 3 jul. 2017 om 05:20 heeft Eliot Miranda <[hidden email]> geschreven: > > No, just time :-/ No problem, if I find the time I'll try to create a smaller example. Stephan |
Still reproducible with current versions of Pharo7 and vm Stephan crash.dmp (16K) Download Attachment |
On 21-09-17 14:51, stephan wrote: > Still reproducible with current versions of Pharo7 and vm I tried again after Pharo build #420 The Pull Request #654 was integrated: "Updated Fuel to 2.2.0 with the 64 bit adaptations" Ubuntu 16.04LTS vm: opensmalltalk / vm / cog / 201801121936 / cog_linux64x64_pharo.cog.spur_201801121936 image: Pharo-7.0.0-alpha.build.425.sha.eb0a6fb.arch.64bit.image crash.dmp (11K) Download Attachment |
In reply to this post by Stephan Eggermont-3
Hi Stephan, I've opened an issue here: https://github.com/theseion/Fuel/issues/229. I'll try to reproduce and see what could be the problem. Cheers, Max On 13 January 2018 at 17:13:23, [hidden email] ([hidden email]) wrote:
|
Max Leske <[hidden email]> wrote: > Hi Stephan, > > I've opened an issue here: https://github.com/theseion/Fuel/issues/229. > I'll try to reproduce and see what could be the problem. Thanks |
In reply to this post by Max Leske
So, I seem to have identified the problem. According to the RFC on Deflate (https://tools.ietf.org/html/rfc1951#page-5) the maximum number of non-compressible blocks is limited to (2^16) - 1: "A compressed data set consists of a series of blocks, corresponding to successive blocks of input data. The block sizes are arbitrary, except that non-compressible blocks are limited to 65,535 bytes." With Fuel loaded (Pharo 7, 64-bit) the following produces a reproducible validation error, which is misleading, the problem is actually that the number of blocks was exceeded (I think that's because of the 16 bit bit mask that is being used in the implementation): (FLGZipStrategy newWithTarget: FLByteArrayStreamStrategy new) writeStreamDo: [ :stream | FLSerializer newDefault serialize: ((1 to: (2 raisedTo: 16) - 82) collect: [ :i | 1 ]) asByteArray on: stream ] I'm not sure what should be done in this case but it appears to me that all remaining data could simply be appended to the last block, at least in the implementation I found in GCC (https://gcc.gnu.org/ml/gcc/2009-09/txt00001.txt): /* Stored blocks are limited to 0xffff bytes, pending_buf is limited * to pending_buf_size, and each stored block has a 5 byte header: */ ulg max_block_size = 0xffff; ulg max_start; if (max_block_size > s->pending_buf_size - 5) { max_block_size = s->pending_buf_size - 5; } Where should we track this issue? Can someone take care to open an issue or should I do it? In any case, I will close the Fuel issue, as this is clearly problem of the Deflate / Zip implementation. Cheers, Max On 13 January 2018 at 18:31:09, Max Leske ([hidden email]) wrote:
|
Hi Max, Great debugging capabilities! Since you are already in deep waters....could you check if it's the same problem I reported in the email "Problem with ZipArchive (probably bug due to large file)"? Thanks a lot in advance, On Sun, Jan 14, 2018 at 4:52 PM, Max Leske <[hidden email]> wrote:
|
In reply to this post by Max Leske
Max Leske <[hidden email]> wrote: > So, I seem to have identified the problem. Great! > According to the RFC on Deflate ( > https://tools.ietf.org/html/rfc1951#page-5) the maximum number of > non-compressible blocks is limited to (2^16) - 1: Uhm, the size of each individual block or the number of blocks? Stephan |
In reply to this post by Stephan Eggermont-3
On 15 January 2018 at 13:00:03, [hidden email] ([hidden email]) wrote:
Yeah, my face looked the same when I read that :D. It's really the number of blocks. Max
|
In reply to this post by Stephan Eggermont-3
Hi Mariano, On 14 January 2018 at 21:10:17, [hidden email] ([hidden email]) wrote:
|
In reply to this post by Stephan Eggermont-3
I've taken the liberty of opening issues for the problem with Deflate. Image code issue: https://pharo.fogbugz.com/f/cases/20973/DeflateStream-can-t-handle-input-that-exceeds-the-maximum-number-of-non-compressible-blocks Cheers, Max On 15 January 2018 at 19:21:20, [hidden email] ([hidden email]) wrote:
|
In reply to this post by Max Leske
Hi Mariano, The archive is in Zip64 format. The Zip format documentation (https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT) states: 4.4.21 total number of entries in the central dir on this disk: (2 bytes) The number of central directory entries on this disk. If an archive is in ZIP64 format and the value in this field is 0xFFFF, the size will be in the corresponding 8 byte zip64 end of central directory field. That's exactly what I see, when I look at the data extracted from the central directory. As was pointed out, the implementation in Pharo doesn't support Zip64, although it would be nice to have. Cheers, Max On 16 January 2018 at 07:40:05, Max Leske ([hidden email]) wrote:
|
On Wed, Jan 17, 2018 at 6:41 PM, Max Leske <[hidden email]> wrote:
Hi Max, thanks for the pointer. So what you are saying is the generated zip that github offers for downloading is a Zip64 bit format because otherwise it would haven't fit in the regular 32, right? And of course, then we don't have Zip64 support. Is that it?
|
In reply to this post by Stephan Eggermont-3
On 19 January 2018 at 13:00:04, [hidden email] ([hidden email]) wrote:
Probably, yes. According to Wikipedia (https://en.wikipedia.org/wiki/Zip_(file_format)#ZIP64) Zip64 overcomes the 4 GiB limit as well as the limit of (2^16) - 1 members. In the case of this particular archive I'd say the number of members could possibly exceed the limit, although Zip64 can also be used without the limits being exceeded. A quick test with a small Zip archive downloaded from Github shows that it can be extracted in Pharo, suggesting that Zip64 is only used by Github when either limit is exceeded. Cheers, Max
|
Free forum by Nabble | Edit this page |