Is there an adhoc header field in Fuel?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
22 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Is there an adhoc header field in Fuel?

Yanni Chiu
I'd like to store some metadata in a Fuel serialized file. I'd like to
be able to read that metadata, without de-serializing the file. So, if
there were a fixed header at the start of the file, with some "unused"
fields, that would be ideal.


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

tinchodias
Hi!
We don't have unused fields, but there is no problem to add your own prefix:

FileStream newFileNamed: 'demoMetadata.fuel' do: [:aStream |
aStream binary.
aStream nextStringPut: 'metadata=5'.
FLSerializer newDefault 
serialize: 'stuff' 
  on: aStream ].

FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
aStream binary.
metadata := aStream nextString.
stuff := (FLMaterializer newDefault 
materializeFrom: aStream) root ].

Is this ok for you?

Best regards,
Martin


On Thu, Feb 9, 2012 at 8:25 PM, Yanni Chiu <[hidden email]> wrote:
I'd like to store some metadata in a Fuel serialized file. I'd like to be able to read that metadata, without de-serializing the file. So, if there were a fixed header at the start of the file, with some "unused" fields, that would be ideal.



Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Yanni Chiu
On 09/02/12 10:20 PM, Martin Dias wrote:
> Hi!
> We don't have unused fields, but there is no problem to add your own prefix:
>[...]
>
> Is this ok for you?

It'll do the job. Here's what the file would have:

  yanni$ od -c demoMetadata.fuel
0000000   \n   m   e   t   a   d   a   t   a   =   5   F   U   E   L  \0
0000020  021 017   F   L   U   I   n   t   1   6   C   l   u   s   t   e
0000040    r  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0  \0 023   F   L
0000060    B   y   t   e   S   t   r   i   n   g   C   l   u   s   t   e
0000100    r  \0  \0  \0 001 005   s   t   u   f   f  \0 001
0000115

It's strange that there is a '\n' at the start. [Okay, figured it out,
it's the character count.]

Having "FUEL" and version number appear some where other than the start
of file doesn't feel right. And, for sure, FLMaterializer will not just
read the file, without some prior code to position the stream past the
metadata - i.e. the file is no longer in "standard" Fuel format.

Would Fuel have a problem with extra bytes at the end?


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Mariano Martinez Peck


On Fri, Feb 10, 2012 at 7:01 AM, Yanni Chiu <[hidden email]> wrote:
On 09/02/12 10:20 PM, Martin Dias wrote:
Hi!
We don't have unused fields, but there is no problem to add your own prefix:
[...]


Is this ok for you?

It'll do the job. Here's what the file would have:

 yanni$ od -c demoMetadata.fuel
0000000   \n   m   e   t   a   d   a   t   a   =   5   F   U   E   L  \0
0000020  021 017   F   L   U   I   n   t   1   6   C   l   u   s   t   e
0000040    r  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0  \0 023   F   L
0000060    B   y   t   e   S   t   r   i   n   g   C   l   u   s   t   e
0000100    r  \0  \0  \0 001 005   s   t   u   f   f  \0 001
0000115

It's strange that there is a '\n' at the start. [Okay, figured it out, it's the character count.]

Having "FUEL" and version number appear some where other than the start of file doesn't feel right. And, for sure, FLMaterializer will not just read the file, without some prior code to position the stream past the metadata - i.e. the file is no longer in "standard" Fuel format.

Would Fuel have a problem with extra bytes at the end?



No. Fuel doesn't care. All you need to do is to send it a stream. If the stream is already at a certain position, it doesn't matter it will start from that position. Say you have stream where you write something first and then you give it to Fuel to serialize. The stream would be at some position different than zero. Say, at position 10. Fuel will start serializing for there. Say we have written 40 bytes.  Then, at materialization, when you give it the stream, you have to be sure to give the stream starting at the position 10. Fuel will load EXACTLY 40 bytes and will stop. The stream can continue....fuel doesn't care and it will store exactly when it finishes reading his own bytes.

BTW, this is how some databases work. The save several serialized objects in the same stream...

Cheers

--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Ben Coman
Mariano Martinez Peck wrote:

> On Fri, Feb 10, 2012 at 7:01 AM, Yanni Chiu <[hidden email]> wrote:
>
>  
>> On 09/02/12 10:20 PM, Martin Dias wrote:
>>
>>    
>>> Hi!
>>> We don't have unused fields, but there is no problem to add your own
>>> prefix:
>>> [...]
>>>
>>>
>>> Is this ok for you?
>>>
>>>      
>> It'll do the job. Here's what the file would have:
>>
>>  yanni$ od -c demoMetadata.fuel
>> 0000000   \n   m   e   t   a   d   a   t   a   =   5   F   U   E   L  \0
>> 0000020  021 017   F   L   U   I   n   t   1   6   C   l   u   s   t   e
>> 0000040    r  \0  \0  \0 001  \0  \0  \0 001  \0  \0  \0  \0 023   F   L
>> 0000060    B   y   t   e   S   t   r   i   n   g   C   l   u   s   t   e
>> 0000100    r  \0  \0  \0 001 005   s   t   u   f   f  \0 001
>> 0000115
>>
>> It's strange that there is a '\n' at the start. [Okay, figured it out,
>> it's the character count.]
>>
>> Having "FUEL" and version number appear some where other than the start of
>> file doesn't feel right. And, for sure, FLMaterializer will not just read
>> the file, without some prior code to position the stream past the metadata
>> - i.e. the file is no longer in "standard" Fuel format.
>>
>> Would Fuel have a problem with extra bytes at the end?
>>
>>
>>
>>    
> No. Fuel doesn't care.
If you consider Fuel "the technology" you may be right.
> All you need to do is to send it a stream. If the
> stream is already at a certain position, it doesn't matter it will start
> from that position. Say you have stream where you write something first and
> then you give it to Fuel to serialize. The stream would be at some position
> different than zero. Say, at position 10. Fuel will start serializing for
> there. Say we have written 40 bytes.  Then, at materialization, when you
> give it the stream, you have to be sure to give the stream starting at the
> position 10.
However if you consider Fuel "the standard file format" - this would
appear to be broken by people using random offsets.
> Fuel will load EXACTLY 40 bytes and will stop. The stream can
> continue....fuel doesn't care and it will store exactly when it finishes
> reading his own bytes.
>
> BTW, this is how some databases work. The save several serialized objects
> in the same stream...
>  
To reach for the moon... Perhaps Fuel "the standard file format" holding
multiple streams might be formalized by tagging different each stream
with a 'name' and/or a 'type' and/or 'deserializerMethod'.  The first
two of these might be '#fuel -> #fuelDeserializer' and
'#meta->#textDeserializer'.  The the default Fuel system can then
automatically ignore 3rd-party data.
> Cheers
>
>  


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Mariano Martinez Peck

All you need to do is to send it a stream. If the
stream is already at a certain position, it doesn't matter it will start
from that position. Say you have stream where you write something first and
then you give it to Fuel to serialize. The stream would be at some position
different than zero. Say, at position 10. Fuel will start serializing for
there. Say we have written 40 bytes.  Then, at materialization, when you
give it the stream, you have to be sure to give the stream starting at the
position 10.
However if you consider Fuel "the standard file format" - this would appear to be broken by people using random offsets.


yes, sure. But I think its ok to fail from the serializer point of view. It is YOUR responsability to provide the correct streams.

 
Fuel will load EXACTLY 40 bytes and will stop. The stream can
continue....fuel doesn't care and it will store exactly when it finishes
reading his own bytes.

BTW, this is how some databases work. The save several serialized objects
in the same stream...
 
To reach for the moon... Perhaps Fuel "the standard file format" holding multiple streams might be formalized by tagging different each stream with a 'name' and/or a 'type' and/or 'deserializerMethod'.  The first two of these might be '#fuel -> #fuelDeserializer' and '#meta->#textDeserializer'.

I don't understand. Fuel holding multiple streams ? why? for what?
 
 The the default Fuel system can then automatically ignore 3rd-party data.

I want to make my point clear: Martin proposed an example that solves Yanni problem. If you serialize with a stream with a certain position, then you have to materialize it with its correct position. Dot. It is perfect that the materializer is broken if you do not give him the correct stream.
Now, I DO agree that there could be another solution rather than adhoc, supported by fuel so that BY DEFAULT some data could be set and ignored at the same time...so that there is no manipulation of the srteam by the user.

 
Cheers

 





--
Mariano
http://marianopeck.wordpress.com

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Igor Stasenko
On 10 February 2012 10:19, Mariano Martinez Peck <[hidden email]> wrote:

>
>>> All you need to do is to send it a stream. If the
>>> stream is already at a certain position, it doesn't matter it will start
>>> from that position. Say you have stream where you write something first
>>> and
>>> then you give it to Fuel to serialize. The stream would be at some
>>> position
>>> different than zero. Say, at position 10. Fuel will start serializing for
>>> there. Say we have written 40 bytes.  Then, at materialization, when you
>>> give it the stream, you have to be sure to give the stream starting at
>>> the
>>> position 10.
>>
>> However if you consider Fuel "the standard file format" - this would
>> appear to be broken by people using random offsets.
>
>
>
> yes, sure. But I think its ok to fail from the serializer point of view. It
> is YOUR responsability to provide the correct streams.
>
>
>>>
>>> Fuel will load EXACTLY 40 bytes and will stop. The stream can
>>> continue....fuel doesn't care and it will store exactly when it finishes
>>> reading his own bytes.
>>>
>>> BTW, this is how some databases work. The save several serialized objects
>>> in the same stream...
>>>
>>
>> To reach for the moon... Perhaps Fuel "the standard file format" holding
>> multiple streams might be formalized by tagging different each stream with a
>> 'name' and/or a 'type' and/or 'deserializerMethod'.  The first two of these
>> might be '#fuel -> #fuelDeserializer' and '#meta->#textDeserializer'.
>
>
> I don't understand. Fuel holding multiple streams ? why? for what?
>
>>
>>  The the default Fuel system can then automatically ignore 3rd-party data.
>
>
> I want to make my point clear: Martin proposed an example that solves Yanni
> problem. If you serialize with a stream with a certain position, then you
> have to materialize it with its correct position. Dot. It is perfect that
> the materializer is broken if you do not give him the correct stream.
> Now, I DO agree that there could be another solution rather than adhoc,
> supported by fuel so that BY DEFAULT some data could be set and ignored at
> the same time...so that there is no manipulation of the srteam by the user.
>
>
no i think your approach is fine. you store what you need, and don't
care/intervene with other parts of stream,
so if user wants to add some stuff before materialized data, he free to do so.


--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

tinchodias
Maybe is too much overhead, but it is also possible to serialize an object that represents the metadata as a prefix:

FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream |
aStream binary.
metadata := Dictionary with: #meta -> 5.
content := 'stuff'.
FLSerializer newDefault 
serialize: metadata on: aStream;
serialize: content on: aStream ].

FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
| aMaterializer |
aStream binary.
aMaterializer := FLMaterializer newDefault.
metadata := (aMaterializer materializeFrom: aStream) root.
content := (aMaterializer materializeFrom: aStream) root ].

Martin

On Fri, Feb 10, 2012 at 8:25 AM, Igor Stasenko <[hidden email]> wrote:
On 10 February 2012 10:19, Mariano Martinez Peck <[hidden email]> wrote:
>
>>> All you need to do is to send it a stream. If the
>>> stream is already at a certain position, it doesn't matter it will start
>>> from that position. Say you have stream where you write something first
>>> and
>>> then you give it to Fuel to serialize. The stream would be at some
>>> position
>>> different than zero. Say, at position 10. Fuel will start serializing for
>>> there. Say we have written 40 bytes.  Then, at materialization, when you
>>> give it the stream, you have to be sure to give the stream starting at
>>> the
>>> position 10.
>>
>> However if you consider Fuel "the standard file format" - this would
>> appear to be broken by people using random offsets.
>
>
>
> yes, sure. But I think its ok to fail from the serializer point of view. It
> is YOUR responsability to provide the correct streams.
>
>
>>>
>>> Fuel will load EXACTLY 40 bytes and will stop. The stream can
>>> continue....fuel doesn't care and it will store exactly when it finishes
>>> reading his own bytes.
>>>
>>> BTW, this is how some databases work. The save several serialized objects
>>> in the same stream...
>>>
>>
>> To reach for the moon... Perhaps Fuel "the standard file format" holding
>> multiple streams might be formalized by tagging different each stream with a
>> 'name' and/or a 'type' and/or 'deserializerMethod'.  The first two of these
>> might be '#fuel -> #fuelDeserializer' and '#meta->#textDeserializer'.
>
>
> I don't understand. Fuel holding multiple streams ? why? for what?
>
>>
>>  The the default Fuel system can then automatically ignore 3rd-party data.
>
>
> I want to make my point clear: Martin proposed an example that solves Yanni
> problem. If you serialize with a stream with a certain position, then you
> have to materialize it with its correct position. Dot. It is perfect that
> the materializer is broken if you do not give him the correct stream.
> Now, I DO agree that there could be another solution rather than adhoc,
> supported by fuel so that BY DEFAULT some data could be set and ignored at
> the same time...so that there is no manipulation of the srteam by the user.
>
>
no i think your approach is fine. you store what you need, and don't
care/intervene with other parts of stream,
so if user wants to add some stuff before materialized data, he free to do so.


--
Best regards,
Igor Stasenko.


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Igor Stasenko
On 10 February 2012 16:24, Martin Dias <[hidden email]> wrote:

> Maybe is too much overhead, but it is also possible to serialize an object
> that represents the metadata as a prefix:
>
> FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream |
> aStream binary.
> metadata := Dictionary with: #meta -> 5.
> content := 'stuff'.
> FLSerializer newDefault
> serialize: metadata on: aStream;
> serialize: content on: aStream ].
>
> FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
> | aMaterializer |
> aStream binary.
> aMaterializer := FLMaterializer newDefault.
> metadata := (aMaterializer materializeFrom: aStream) root.
> content := (aMaterializer materializeFrom: aStream) root ].
>
> Martin

which eventually could be turned into simple:

MyStuff>>serializeOn: stream
 serializer  := FLSerializer newDefault.

  serializer serialize: {myStuff. myMetaStuff. myMetaMetaStuff } on: stream ].

so, you don't even need to care about stream contents :)


--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Yanni Chiu
In reply to this post by tinchodias
On 10/02/12 10:24 AM, Martin Dias wrote:

> Maybe is too much overhead, but it is also possible to serialize an
> object that represents the metadata as a prefix:
>
> FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream |
> aStream binary.
> metadata := Dictionary with: #meta -> 5.
> content := 'stuff'.
> FLSerializer newDefault
> serialize: metadata on: aStream;
> serialize: content on: aStream ].
>
> FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
> | aMaterializer |
> aStream binary.
> aMaterializer := FLMaterializer newDefault.
> metadata := (aMaterializer materializeFrom: aStream) root.
> content := (aMaterializer materializeFrom: aStream) root ].

I like this solution best, since it doesn't change the Fuel "file format".

I added some helper methods to my code base to make this usage scenario
more obvious:

FLMaterializer>>nextMaterializedRootFrom: sourceStream
        ^ (self materializeFrom: sourceStream) root

FLMaterializer>>allMaterializedRootsFrom: sourceStream
        | contents |
        contents := OrderedCollection new.
        [ sourceStream atEnd ]
                whileFalse: [ contents add: (self nextMaterializedRootFrom:
sourceStream) ].
        ^ contents


Sample usage:

FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream |
| serializer |
aStream binary.
serializer := FLSerializer newDefault.
(1 to: 5) do: [ :each | serializer serialize: (Dictionary with: each ->
(each * each)) on: aStream ].
].

FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
| aMaterializer |
aStream binary.
aMaterializer := FLMaterializer newDefault.
aMaterializer allMaterializedRootsFrom: aStream ].


FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
| aMaterializer |
aStream binary.
aMaterializer := FLMaterializer newDefault.
aMaterializer nextMaterializedRootFrom: aStream.
aMaterializer nextMaterializedRootFrom: aStream.
].


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

tinchodias

On Fri, Feb 10, 2012 at 2:45 PM, Yanni Chiu <[hidden email]> wrote:
On 10/02/12 10:24 AM, Martin Dias wrote:
Maybe is too much overhead, but it is also possible to serialize an
object that represents the metadata as a prefix:

FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream |
aStream binary.
metadata := Dictionary with: #meta -> 5.
content := 'stuff'.
FLSerializer newDefault
serialize: metadata on: aStream;
serialize: content on: aStream ].

FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
| aMaterializer |
aStream binary.
aMaterializer := FLMaterializer newDefault.
metadata := (aMaterializer materializeFrom: aStream) root.
content := (aMaterializer materializeFrom: aStream) root ].

I like this solution best, since it doesn't change the Fuel "file format".

I added some helper methods to my code base to make this usage scenario more obvious:

FLMaterializer>>nextMaterializedRootFrom: sourceStream
       ^ (self materializeFrom: sourceStream) root

FLMaterializer>>allMaterializedRootsFrom: sourceStream
       | contents |
       contents := OrderedCollection new.
       [ sourceStream atEnd ]
               whileFalse: [ contents add: (self nextMaterializedRootFrom: sourceStream) ].
       ^ contents


Sample usage:


FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream |
| serializer |
aStream binary.
serializer := FLSerializer newDefault.
(1 to: 5) do: [ :each | serializer serialize: (Dictionary with: each -> (each * each)) on: aStream ].

].

FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
| aMaterializer |
aStream binary.
aMaterializer := FLMaterializer newDefault.
aMaterializer allMaterializedRootsFrom: aStream ].



FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream |
| aMaterializer |
aStream binary.
aMaterializer := FLMaterializer newDefault.
aMaterializer nextMaterializedRootFrom: aStream.
aMaterializer nextMaterializedRootFrom: aStream.
].

Great. Maybe we should include #nextMaterializedRootFrom: to fuel.

Based on metadata you might decide to not materialize the "real" content. Like:

metadata := (aMaterializer nextMaterializedRootFrom: aStream).
[ metadata isWhatINeed ] ifTrue: [ 
    realContent := aMaterializer nextMaterializedRootFrom: aStream ].

 

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Yanni Chiu
On 10/02/12 1:03 PM, Martin Dias wrote:
>
> Great. Maybe we should include #nextMaterializedRootFrom: to fuel.

That would be great. Probably best to leave off
#allMaterializedRootsFrom: - it seemed necessary for completeness, but
I've no actual need for it.

> Based on metadata you might decide to not materialize the "real"
> content. Like:
>
> metadata := (aMaterializer nextMaterializedRootFrom: aStream).
> [ metadata isWhatINeed ] ifTrue: [
>      realContent := aMaterializer nextMaterializedRootFrom: aStream ].

Yes, that's the usage scenario.


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Eliot Miranda-2
In reply to this post by Mariano Martinez Peck


On Fri, Feb 10, 2012 at 1:19 AM, Mariano Martinez Peck <[hidden email]> wrote:

All you need to do is to send it a stream. If the
stream is already at a certain position, it doesn't matter it will start
from that position. Say you have stream where you write something first and
then you give it to Fuel to serialize. The stream would be at some position
different than zero. Say, at position 10. Fuel will start serializing for
there. Say we have written 40 bytes.  Then, at materialization, when you
give it the stream, you have to be sure to give the stream starting at the
position 10.
However if you consider Fuel "the standard file format" - this would appear to be broken by people using random offsets.


yes, sure. But I think its ok to fail from the serializer point of view. It is YOUR responsability to provide the correct streams.

 
Fuel will load EXACTLY 40 bytes and will stop. The stream can
continue....fuel doesn't care and it will store exactly when it finishes
reading his own bytes.

BTW, this is how some databases work. The save several serialized objects
in the same stream...
 
To reach for the moon... Perhaps Fuel "the standard file format" holding multiple streams might be formalized by tagging different each stream with a 'name' and/or a 'type' and/or 'deserializerMethod'.  The first two of these might be '#fuel -> #fuelDeserializer' and '#meta->#textDeserializer'.

I don't understand. Fuel holding multiple streams ? why? for what?

e.g. for source code.  Imagine storing Monticello packages in Fuel so that the source code is stored in the Fuel file and doesn't have to be written to the changes file.  Instead, a more powerful SourceFilesArray (actually a source file manager) can maintain a set of source files, including Fuel files, and fetch source there-from.  Hence loading is faster, and the changes file isn't polluted with code loads, containing only one's own code.

e.g. for external resources.  imagine loading a Fuel package that also contains some dlls that can be unpacked as required.

But if you choose this route let me strongly suggest you use the zip file format.  Its very useful with Monticello mcz's to be able to pick them apart using unzip.
 
 
 The the default Fuel system can then automatically ignore 3rd-party data.

I want to make my point clear: Martin proposed an example that solves Yanni problem. If you serialize with a stream with a certain position, then you have to materialize it with its correct position. Dot. It is perfect that the materializer is broken if you do not give him the correct stream.
Now, I DO agree that there could be another solution rather than adhoc, supported by fuel so that BY DEFAULT some data could be set and ignored at the same time...so that there is no manipulation of the srteam by the user.

 
Cheers

 





--
Mariano
http://marianopeck.wordpress.com




--
best,
Eliot

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Igor Stasenko
On 10 February 2012 19:27, Eliot Miranda <[hidden email]> wrote:
>
>
> But if you choose this route let me strongly suggest you use the zip file
> format.  Its very useful with Monticello mcz's to be able to pick them apart
> using unzip.
>

yes but then it makes the whole thing completely orthogonal to Fuel.
because in zip you can have one file to hold serialized stuff, in
another your metadata and whatsoever..
i mean, you can organize your data packaging yourself, without forcing
fuel to support anything related to metadata.

and i think this is right direction.

So. I think it is useless to try extending fuel with metadata. if
people want to have metadata, they should make it by own,
in the way they see fit.
I think fuel should do one thing : serialize/materialize stuff.  Lets
not turn it into "i can do everything to please you" library.

--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Stéphane Ducasse
In reply to this post by Yanni Chiu
this is a good idea.

On Feb 10, 2012, at 12:25 AM, Yanni Chiu wrote:

> I'd like to store some metadata in a Fuel serialized file. I'd like to be able to read that metadata, without de-serializing the file. So, if there were a fixed header at the start of the file, with some "unused" fields, that would be ideal.
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Stéphane Ducasse
In reply to this post by tinchodias
>
> Great. Maybe we should include #nextMaterializedRootFrom: to fuel.

And I would love a chapter for our book :)

But after mariano finished his phd draft :)

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

NorbertHartl
In reply to this post by Igor Stasenko

Am 10.02.2012 um 19:51 schrieb Igor Stasenko:

> On 10 February 2012 19:27, Eliot Miranda <[hidden email]> wrote:
>>
>>
>> But if you choose this route let me strongly suggest you use the zip file
>> format.  Its very useful with Monticello mcz's to be able to pick them apart
>> using unzip.
>>
>
> yes but then it makes the whole thing completely orthogonal to Fuel.
> because in zip you can have one file to hold serialized stuff, in
> another your metadata and whatsoever..
> i mean, you can organize your data packaging yourself, without forcing
> fuel to support anything related to metadata.
>
> and i think this is right direction.
>
> So. I think it is useless to try extending fuel with metadata. if
> people want to have metadata, they should make it by own,
> in the way they see fit.
> I think fuel should do one thing : serialize/materialize stuff.  Lets
> not turn it into "i can do everything to please you" library.
>
+1

Fuel does what it does. Having a fuel with metadata is just a customized format. In the end it is all streams. You can open a file read the metadata and then hand over the stream to fuel. I don't see any need to tweak fuel for that. Changing it to be able to read multiple roots sounds great but has the potential for e.g. doubling objects and all sorts (if I understand it correct).

Norbert


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Ben Coman
Norbert Hartl wrote:

> Am 10.02.2012 um 19:51 schrieb Igor Stasenko:
>
>  
>> On 10 February 2012 19:27, Eliot Miranda <[hidden email]> wrote:
>>    
>>> But if you choose this route let me strongly suggest you use the zip file
>>> format.  Its very useful with Monticello mcz's to be able to pick them apart
>>> using unzip.
>>>
>>>      
>> yes but then it makes the whole thing completely orthogonal to Fuel.
>> because in zip you can have one file to hold serialized stuff, in
>> another your metadata and whatsoever..
>> i mean, you can organize your data packaging yourself, without forcing
>> fuel to support anything related to metadata.
>>
>> and i think this is right direction.
>>
>> So. I think it is useless to try extending fuel with metadata. if
>> people want to have metadata, they should make it by own,
>> in the way they see fit.
>> I think fuel should do one thing : serialize/materialize stuff.  Lets
>> not turn it into "i can do everything to please you" library.
>>
>>    
> +1
>
> Fuel does what it does. Having a fuel with metadata is just a customized format. In the end it is all streams. You can open a file read the metadata and then hand over the stream to fuel. I don't see any need to tweak fuel for that. Changing it to be able to read multiple roots sounds great but has the potential for e.g. doubling objects and all sorts (if I understand it correct).
>
> Norbert
>
>
>
>  
Using a standard container is better resuse and design - but maybe you
still want to specify exactly the format of a file ending in "*.fuel".  
Is it _always_ a zip file or do you look for the zipfile magic number
and if that is not there revert to it being a single fuel stream.

cheers, -ben

Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Yanni Chiu
In reply to this post by Stéphane Ducasse
On 10/02/12 3:45 PM, Stéphane Ducasse wrote:
> this is a good idea.

Hmmm. I think the .zip file approach is the "correct/better" approach to
take to my metadata problem.

This discussion did turn up a feature of Fuel that was not obvious to me
before - namely, that you can read/write multiple materializations on a
single stream.


Reply | Threaded
Open this post in threaded view
|

Re: Is there an adhoc header field in Fuel?

Mariano Martinez Peck
In reply to this post by Eliot Miranda-2


I don't understand. Fuel holding multiple streams ? why? for what?

e.g. for source code.  Imagine storing Monticello packages in Fuel so that the source code is stored in the Fuel file and doesn't have to be written to the changes file.  Instead, a more powerful SourceFilesArray (actually a source file manager) can maintain a set of source files, including Fuel files, and fetch source there-from.  Hence loading is faster, and the changes file isn't polluted with code loads, containing only one's own code.

e.g. for external resources.  imagine loading a Fuel package that also contains some dlls that can be unpacked as required.

But if you choose this route let me strongly suggest you use the zip file format.  Its very useful with Monticello mcz's to be able to pick them apart using unzip.
 

Ok, I see. That makes sense. However, that won't be part of Fuel, but rather a tool build on top of it. Now we have the Fuel core a simple serializer. FuelMetalevel, an optional package build on top of Fuel core to be able to correctly serialize classes, methods, traits, etc. FuelPackageLoader, an optional package build on top of FuelMetalevel to experiment import/export of packages in a binary way.   So....continue with that infrastructure, if there would be a Monticello integration with Fuel, then such package should take care about serializing different things in different streams.   But so far, for Fuel core, and I think the best approach is to let is simple as it is now.


 
 
 The the default Fuel system can then automatically ignore 3rd-party data.

I want to make my point clear: Martin proposed an example that solves Yanni problem. If you serialize with a stream with a certain position, then you have to materialize it with its correct position. Dot. It is perfect that the materializer is broken if you do not give him the correct stream.
Now, I DO agree that there could be another solution rather than adhoc, supported by fuel so that BY DEFAULT some data could be set and ignored at the same time...so that there is no manipulation of the srteam by the user.

 
Cheers

 





--
Mariano
http://marianopeck.wordpress.com




--
best,
Eliot




--
Mariano
http://marianopeck.wordpress.com

12