FUEL serialization to a file

Hi everyone,

after playing with FUEL in-memory (byte arrays) for a while, I am ready
to attack files. But... that's not so easy.

At
https://ci.inria.fr/pharo-contribution/job/EnterprisePharoBook/lastSuccessfulBuild/artifact/book-result/Fuel/Fuel.html
I find the basic exxample

   'demo.fuel' asFileReference writeStreamDo: [ :aStream |
      FLSerializer newDefault
         serialize: 'stringToSerialize'
         on: aStream binary ].

It fails because ZnCharacterWriteStream does not understand #binary.

Browsing around a bit, I found another way:

   'demo.fuel' asFileReference writeStreamDo: [ :aStream |
      FLSerializer newDefault
         serialize: 'stringToSerialize'
         on: aStream binary ].

This works fine, the aStream being a ZnBufferedWriteStream. But on
larger objects, it ends up failing in

    FLBufferedWriteStream >> #nextBytePutAll:

which sends the message

    stream nextBytesPutAll: collection

But ZnBufferedWriteStream does not understand nextBytesPutAll: because
it is not a subclass of Stream for whatever reason.

Some further browing showed deprecated classes FileStream etc., and I
kind of suspect that file streams in Pharo were changed at some point
but FUEL still expects the old behavior.

Does anyone have an idea how to fix this, or work around it?

Thanks in advance,
  Konrad

Hmmm - yesterday I had someone serialise their debug stack (the top right menu bar action) to a file and send it to me and it worked  treat - so I wonder what the difference is?

Tim

Sent from my iPhone

FUEL is a binary serialiser, in the new Pharo 7 stream approach, a stream is either binary or textual, not both, nor can they be switched on the fly.

FileLocator temp / 'test.fuel' binaryWriteStreamDo: [ :out | FLSerializer newDefault serialize: { 'Foo'. #bar. Float pi. 42 } on: out ].

FileLocator temp / 'test.fuel' binaryReadStreamDo: [ :in | (FLMaterializer newDefault materializeFrom: in) root ].

Sven Van Caekenberghe <[hidden email]> writes:

> FUEL is a binary serialiser, in the new Pharo 7 stream approach, a
> stream is either binary or textual, not both, nor can they be switched
> on the fly.

OK, thanks for confirming my suspicion that there was a change in Pharo.

> FileLocator temp / 'test.fuel' binaryWriteStreamDo: [ :out | FLSerializer newDefault serialize: { 'Foo'. #bar. Float pi. 42 } on: out ].

So what I have been using is correct.


Tim Mackinnon <[hidden email]> writes:

> Hmmm - yesterday I had someone serialise their debug stack (the top
> right menu bar action) to a file and send it to me and it worked treat
> - so I wonder what the difference is?

The problem I have occurs in buffering code. I suspect that it happens
as a function of the size distribution of the objects that are
serialized.


The problem is best analyzed by looking at the implementors of
#nextBytesPutAll:. That's an extension method that Fuel puts on the
Stream hierarchy, including its own class FLBufferedWriteStream where
the method contains the lines

     collection size > (self buffer size / 2)
        ifTrue: [ stream nextBytesPutAll: collection ]
        ifFalse: [ self nextBytesPutAll: collection ]

In the second line, it sends nextBytesPutAll: to the unbuffered
stream. Which should be fine if it inherits from Stream. But
ZnBufferedWriteStream does not inherit from Stream.

I tried unraveling the construction of ZnBufferedWriteStream, to use
directly its underlying stream:

    | ref stream |
    ref := FileLocator temp / 'test.fuel'.
    stream := ref fileSystem binaryWriteStreamOn: ref path.
    [ FLSerializer newDefault serialize: { 'Foo'. #bar. Float pi. 42 }
         on: stream ]
    ensure: [ stream close ]

On a standard file, that yields a BinaryFileStream, and everything works
fine. But for my test cases I use a MemoryFileSystem, and then I get
the same problem again - MemoryFileWriteStream does not inherit from
Stream either.

I wonder what's wrong with the Stream class - why are there stream-type
classes that don't inherit from it?

Konrad.

Am 09.03.19 um 08:16 schrieb Konrad Hinsen:

> In the second line, it sends nextBytesPutAll: to the unbuffered
> stream. Which should be fine if it inherits from Stream. But
> ZnBufferedWriteStream does not inherit from Stream.

The fix I found is simple, but I don't understand the whole system well
enough to be sure that it doesn't cause trouble elsewhere. I added the
method ZnBufferedWriteStream >> nextBytesPutAll:

   nextBytesPutAll: aCollection
       self nextPutAll: aCollection

and all my problems are gone.

Konrad.

Konrad,

You are raising many different points, I'll try to answer them.

But first, it would help a lot that you give a self-contained reproducible snippet that raises your problem, like mine. I tried reproducing your failure but could not.

Also note that all FUEL tests are green on a recent Pharo image.

Regarding architecture and design.

The idea of the 'new' streams is about simplification, streamlining and composability (if this is a real word) based on separation of concerns. It is my strong opinion that the current stream API is way to broad and complex.

That is the main reason why it is not or should not be necessary to inherit from Stream.

We want read and write streams to be separate, we want binary and character streams to be separate. Buffering, line end conversions, etc, should be pluggable.

I remember that FLBufferedWriteStream was actual based on ZnBufferedWriteStream (one of FUEL's goals is performance), at a time when the primitive binary streams were unbuffered. That was years ago.

Today, the high level binary streams are in principle buffered (ZnBufferedWriteStream) (although the raw ones still exist).

It is thus no longer necessary to use FLBufferedWriteStream (although its API is slightly larger, it does a bit more than just buffer output).

Now, looking at the specific selector you are talking about, #nextBytesPutAll: the way it is implemented and used, I can't understand why it even exists. Unless I am missing something it is the same as #nextPutAll: It is probably related to backwards or cross platform compatibility. Because today in Pharo, given a (buffered) binary stream, #nextPutAll: always takes bytes as argument.

Another point is that right now, it seems double buffering is happening, which is not good.

I guess we'll have to wait for one of the FUEL developers to chime in.

All this being said, we are still in a transition period.

Sven

Hi Sven,

Thanks for all that background information on the history of streams. It
makes everything a lot clearer for me.

> But first, it would help a lot that you give a self-contained
> reproducible snippet that raises your problem, like mine. I tried
> reproducing your failure but could not.

My bug is difficult to reproduce because it is triggered only when the
"right" kind of buffer overflow happens. I didn't succeed yet to provoke
it while serializing simple objects such as strings or arrays.

I did manage to reduce my code quite a lot, so here is the shortest
example I have for the moment that provokes the bug:

   Metacello new
     baseline: 'ComputationalDocuments';
     repository: 'github://khinsen/computational-documents-with-gtoolkit:minimal-code-for-serialization-bug';
     load.

And then

   APLibraryTest new setUp; testScanDirectory

Konrad.