Working with a compressed Fuel file?

I'm trying to work with a compressed Fuel file, without having to expand
everything at the same time. It looks like not all streams are created
equally. I seem to need a File instead of a FileReference:

saveToFile
        |zipStream fileRef|
        fileRef := File named: (FileSystem disk workingDirectory / (self
fileName, '.zip')) pathString.
        fileRef writeStreamDo: [ :s |
                zipStream := GZipWriteStream on: s.
                repository := nil.
                FLSerializer serialize: self on: zipStream.
                zipStream close].


And materializing doesn't like this stream

from: aFileName
        |zipStream fileRef theNew|
        fileRef := File named: aFileName.
        fileRef readStreamDo: [ :s |
                zipStream := GZipReadStream on: s.
                theNew := (FLMaterializer new materializeFrom: zipStream) root.
                zipStream close].
        ^theNew

How am I supposed to work with compressed Fuel files?

Stephan

I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message?

On 03-01-16 20:31, Mariano Martinez Peck wrote:
> I may have a hint. Which is the error upon materialization? A dnu on the
> stream? Which message?

A segmentation fault when saving

Smalltalk stack dump:
0xffcb7390 M GZipWriteStream(DeflateStream)>deflateBlock 0xe66c808: a(n)
GZipWriteStream
0xffcb73b4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt:
0xe66c808: a(n) GZipWriteStream
0xffcb73d8 M FLBufferedWriteStream>flushBuffer 0xe66cb48: a(n)
FLBufferedWriteStream
0xffcb73f0 M FLBufferedWriteStream>nextBytesPutAll: 0xe66cb48: a(n)
FLBufferedWriteStream
0xffcb7410 M FLEncoder>encodeString: 0xe66c720: a(n) FLEncoder
0xffcb742c M ByteString>serializeOn: 0xcce25a0: a(n) ByteString
0xffcb7448 M FLHookPrimitiveCluster>serializeInstance:with: 0xe713af8:
a(n) FLHookPrimitiveCluster
0xffcb746c M [] in
FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith:
0xe713af8: a(n) FLHookPrimitiveCluster
0xffcb7494 M IdentitySet(Set)>do: 0xf70b230: a(n) IdentitySet
0xffcb74b0 M
FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith:
0xe713af8: a(n) FLHookPrimitiveCluster
0xffcb74cc M FLSerialization>clusterInstancesStepOf: 0xe66c6e8: a(n)
FLSerialization
0xffcb74e8 M [] in FLSerialization>instancesStep 0xe66c6e8: a(n)
FLSerialization
0xffcb750c M OrderedCollection>do: 0xf70b1e0: a(n) OrderedCollection
0xffcb7530 I FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization
0xffcb7550 I FLSerialization>run 0xe66c6e8: a(n) FLSerialization
0xffcb7574 I [] in FLSerializer>setDefaultSerialization 0xe66c700: a(n)
FLSerializer
0xffcb75a0 I [] in FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer
0xffcb75cc I [] in FLEncoder class>on:globalEnvironment:do: 0x8f20878:
a(n) FLEncoder class
0xffcb75ec M BlockClosure>ensure: 0xe66c770: a(n) BlockClosure
0xffcb7614 I FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n)
FLEncoder class
0xffcb7640 I FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer
0xffcb7668 I FLSerializer class>serialize:on: 0x8eb91d8: a(n)
FLSerializer class
0xffcb7694 I [] in MCProject>saveToFile 0xb92d790: a(n) MCProject
0xffcb76c0 I [] in File>writeStreamDo: 0xe612500: a(n) File
0xffcb76e0 M BlockClosure>ensure: 0xe66c900: a(n) BlockClosure
0xffcb7708 I File>writeStreamDo: 0xe612500: a(n) File
0xffcb7734 I MCProject>saveToFile 0xb92d790: a(n) MCProject
0xffcb7750 M UndefinedObject>(nil) 0x8b75b00: a(n) UndefinedObject
  0xb73ad00 s OpalCompiler>evaluate
  0xb92dec8 s RubSmalltalkEditor>evaluate:andDo:
  0xb92df58 s RubSmalltalkEditor>highlightEvaluateAndDo:
  0xb743418 s [] in
GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate:
  0xb73ad60 s RubEditingArea(RubAbstractTextArea)>handleEdit:
  0xb92dff8 s [] in
GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate:
  0xb743478 s WorldState>runStepMethodsIn:
  0xb73adc0 s WorldMorph>runStepMethods
  0xb92e070 s WorldState>doOneCycleNowFor:
  0xb7434d8 s WorldState>doOneCycleFor:
  0xb73ae20 s WorldMorph>doOneCycle
  0x944db08 s [] in MorphicUIManager>(nil)
  0x944da80 s [] in BlockClosure>(nil)

On 04-01-16 14:29, Mariano Martinez Peck wrote:
> Ahh I reported that years ago:
> http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html

Ah. That is also when writing a string. Is the automatic conversion of
ByteString to WideString safe here?

Stephan

On 04-01-16 14:29, Mariano Martinez Peck wrote:
> Ahh I reported that years ago:
> http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-serializing-td4177725.html

I can't seem to find an issue for this. Did we lose it?

Stephan

On 03-01-16 20:31, Mariano Martinez Peck wrote:
> I may have a hint. Which is the error upon materialization? A dnu on the
> stream? Which message?

Here is a further problem. I've done:

|project|
Gofer it
        smalltalkhubUser: 'StephanEggermont' project: 'MonticelloProjects';
        package: 'MonticelloProjects';
        load.

project := (Smalltalk at: #MCProject) new location:
'http://source.squeak.org/VMMaker'.
project read.
project repository: nil.
FLSerializer serialize: project toFileNamed: 'VMMaker.fl'.

That takes a while (downloads 5 GB or so).
It results in a 1.2 GB Fuel file.
That is way too much, so somewhere I'm pulling in too many objects,
or they are not recognized as equal.

Writing out the 71712 unique definitions in this project separately is
54MB. The project consists of 17 packages with 2723 package versions,
the total number of pointers to the definitions is 2.2 million.
The total should be below 100MB

Stephan

On 05-01-16 16:29, Stephan Eggermont wrote:
> Writing out the 71712 unique definitions in this project separately is
> 54MB. The project consists of 17 packages with 2723 package versions,
> the total number of pointers to the definitions is 2.2 million.
> The total should be below 100MB

Thanks to the help of Max and Thierry I found the problem.

Set withAll: (MCVersionInfo allInstances collect: [  :each | each id ]) 7155

MCVersionInfo allInstances size 2726588

2030 versions of VMMaker-> 2029+2028+2027+...+1 MCVersionInfo's ideally...

Stephan