I'd like to store some metadata in a Fuel serialized file. I'd like to
be able to read that metadata, without de-serializing the file. So, if there were a fixed header at the start of the file, with some "unused" fields, that would be ideal. |
Hi!
We don't have unused fields, but there is no problem to add your own prefix: FileStream newFileNamed: 'demoMetadata.fuel' do: [:aStream | aStream binary.
aStream nextStringPut: 'metadata=5'. FLSerializer newDefault serialize: 'stuff'
on: aStream ]. FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream | aStream binary.
metadata := aStream nextString. stuff := (FLMaterializer newDefault materializeFrom: aStream) root ].
Is this ok for you? Best regards, Martin On Thu, Feb 9, 2012 at 8:25 PM, Yanni Chiu <[hidden email]> wrote: I'd like to store some metadata in a Fuel serialized file. I'd like to be able to read that metadata, without de-serializing the file. So, if there were a fixed header at the start of the file, with some "unused" fields, that would be ideal. |
On 09/02/12 10:20 PM, Martin Dias wrote:
> Hi! > We don't have unused fields, but there is no problem to add your own prefix: >[...] > > Is this ok for you? It'll do the job. Here's what the file would have: yanni$ od -c demoMetadata.fuel 0000000 \n m e t a d a t a = 5 F U E L \0 0000020 021 017 F L U I n t 1 6 C l u s t e 0000040 r \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 \0 023 F L 0000060 B y t e S t r i n g C l u s t e 0000100 r \0 \0 \0 001 005 s t u f f \0 001 0000115 It's strange that there is a '\n' at the start. [Okay, figured it out, it's the character count.] Having "FUEL" and version number appear some where other than the start of file doesn't feel right. And, for sure, FLMaterializer will not just read the file, without some prior code to position the stream past the metadata - i.e. the file is no longer in "standard" Fuel format. Would Fuel have a problem with extra bytes at the end? |
On Fri, Feb 10, 2012 at 7:01 AM, Yanni Chiu <[hidden email]> wrote:
No. Fuel doesn't care. All you need to do is to send it a stream. If the stream is already at a certain position, it doesn't matter it will start from that position. Say you have stream where you write something first and then you give it to Fuel to serialize. The stream would be at some position different than zero. Say, at position 10. Fuel will start serializing for there. Say we have written 40 bytes. Then, at materialization, when you give it the stream, you have to be sure to give the stream starting at the position 10. Fuel will load EXACTLY 40 bytes and will stop. The stream can continue....fuel doesn't care and it will store exactly when it finishes reading his own bytes. BTW, this is how some databases work. The save several serialized objects in the same stream... Cheers -- Mariano http://marianopeck.wordpress.com |
Mariano Martinez Peck wrote:
> On Fri, Feb 10, 2012 at 7:01 AM, Yanni Chiu <[hidden email]> wrote: > > >> On 09/02/12 10:20 PM, Martin Dias wrote: >> >> >>> Hi! >>> We don't have unused fields, but there is no problem to add your own >>> prefix: >>> [...] >>> >>> >>> Is this ok for you? >>> >>> >> It'll do the job. Here's what the file would have: >> >> yanni$ od -c demoMetadata.fuel >> 0000000 \n m e t a d a t a = 5 F U E L \0 >> 0000020 021 017 F L U I n t 1 6 C l u s t e >> 0000040 r \0 \0 \0 001 \0 \0 \0 001 \0 \0 \0 \0 023 F L >> 0000060 B y t e S t r i n g C l u s t e >> 0000100 r \0 \0 \0 001 005 s t u f f \0 001 >> 0000115 >> >> It's strange that there is a '\n' at the start. [Okay, figured it out, >> it's the character count.] >> >> Having "FUEL" and version number appear some where other than the start of >> file doesn't feel right. And, for sure, FLMaterializer will not just read >> the file, without some prior code to position the stream past the metadata >> - i.e. the file is no longer in "standard" Fuel format. >> >> Would Fuel have a problem with extra bytes at the end? >> >> >> >> > No. Fuel doesn't care. > All you need to do is to send it a stream. If the > stream is already at a certain position, it doesn't matter it will start > from that position. Say you have stream where you write something first and > then you give it to Fuel to serialize. The stream would be at some position > different than zero. Say, at position 10. Fuel will start serializing for > there. Say we have written 40 bytes. Then, at materialization, when you > give it the stream, you have to be sure to give the stream starting at the > position 10. However if you consider Fuel "the standard file format" - this would appear to be broken by people using random offsets. > Fuel will load EXACTLY 40 bytes and will stop. The stream can > continue....fuel doesn't care and it will store exactly when it finishes > reading his own bytes. > > BTW, this is how some databases work. The save several serialized objects > in the same stream... > To reach for the moon... Perhaps Fuel "the standard file format" holding multiple streams might be formalized by tagging different each stream with a 'name' and/or a 'type' and/or 'deserializerMethod'. The first two of these might be '#fuel -> #fuelDeserializer' and '#meta->#textDeserializer'. The the default Fuel system can then automatically ignore 3rd-party data. > Cheers > > |
yes, sure. But I think its ok to fail from the serializer point of view. It is YOUR responsability to provide the correct streams.
I don't understand. Fuel holding multiple streams ? why? for what? The the default Fuel system can then automatically ignore 3rd-party data. I want to make my point clear: Martin proposed an example that solves Yanni problem. If you serialize with a stream with a certain position, then you have to materialize it with its correct position. Dot. It is perfect that the materializer is broken if you do not give him the correct stream. Now, I DO agree that there could be another solution rather than adhoc, supported by fuel so that BY DEFAULT some data could be set and ignored at the same time...so that there is no manipulation of the srteam by the user. Cheers -- Mariano http://marianopeck.wordpress.com |
On 10 February 2012 10:19, Mariano Martinez Peck <[hidden email]> wrote:
> >>> All you need to do is to send it a stream. If the >>> stream is already at a certain position, it doesn't matter it will start >>> from that position. Say you have stream where you write something first >>> and >>> then you give it to Fuel to serialize. The stream would be at some >>> position >>> different than zero. Say, at position 10. Fuel will start serializing for >>> there. Say we have written 40 bytes. Then, at materialization, when you >>> give it the stream, you have to be sure to give the stream starting at >>> the >>> position 10. >> >> However if you consider Fuel "the standard file format" - this would >> appear to be broken by people using random offsets. > > > > yes, sure. But I think its ok to fail from the serializer point of view. It > is YOUR responsability to provide the correct streams. > > >>> >>> Fuel will load EXACTLY 40 bytes and will stop. The stream can >>> continue....fuel doesn't care and it will store exactly when it finishes >>> reading his own bytes. >>> >>> BTW, this is how some databases work. The save several serialized objects >>> in the same stream... >>> >> >> To reach for the moon... Perhaps Fuel "the standard file format" holding >> multiple streams might be formalized by tagging different each stream with a >> 'name' and/or a 'type' and/or 'deserializerMethod'. The first two of these >> might be '#fuel -> #fuelDeserializer' and '#meta->#textDeserializer'. > > > I don't understand. Fuel holding multiple streams ? why? for what? > >> >> The the default Fuel system can then automatically ignore 3rd-party data. > > > I want to make my point clear: Martin proposed an example that solves Yanni > problem. If you serialize with a stream with a certain position, then you > have to materialize it with its correct position. Dot. It is perfect that > the materializer is broken if you do not give him the correct stream. > Now, I DO agree that there could be another solution rather than adhoc, > supported by fuel so that BY DEFAULT some data could be set and ignored at > the same time...so that there is no manipulation of the srteam by the user. > > care/intervene with other parts of stream, so if user wants to add some stuff before materialized data, he free to do so. -- Best regards, Igor Stasenko. |
Maybe is too much overhead, but it is also possible to serialize an object that represents the metadata as a prefix:
FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream |
aStream binary. metadata := Dictionary with: #meta -> 5. content := 'stuff'.
FLSerializer newDefault serialize: metadata on: aStream; serialize: content on: aStream ].
FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream | | aMaterializer | aStream binary.
aMaterializer := FLMaterializer newDefault. metadata := (aMaterializer materializeFrom: aStream) root.
content := (aMaterializer materializeFrom: aStream) root ]. On Fri, Feb 10, 2012 at 8:25 AM, Igor Stasenko <[hidden email]> wrote:
|
On 10 February 2012 16:24, Martin Dias <[hidden email]> wrote:
> Maybe is too much overhead, but it is also possible to serialize an object > that represents the metadata as a prefix: > > FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream | > aStream binary. > metadata := Dictionary with: #meta -> 5. > content := 'stuff'. > FLSerializer newDefault > serialize: metadata on: aStream; > serialize: content on: aStream ]. > > FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream | > | aMaterializer | > aStream binary. > aMaterializer := FLMaterializer newDefault. > metadata := (aMaterializer materializeFrom: aStream) root. > content := (aMaterializer materializeFrom: aStream) root ]. > > Martin which eventually could be turned into simple: MyStuff>>serializeOn: stream serializer := FLSerializer newDefault. serializer serialize: {myStuff. myMetaStuff. myMetaMetaStuff } on: stream ]. so, you don't even need to care about stream contents :) -- Best regards, Igor Stasenko. |
In reply to this post by tinchodias
On 10/02/12 10:24 AM, Martin Dias wrote:
> Maybe is too much overhead, but it is also possible to serialize an > object that represents the metadata as a prefix: > > FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream | > aStream binary. > metadata := Dictionary with: #meta -> 5. > content := 'stuff'. > FLSerializer newDefault > serialize: metadata on: aStream; > serialize: content on: aStream ]. > > FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream | > | aMaterializer | > aStream binary. > aMaterializer := FLMaterializer newDefault. > metadata := (aMaterializer materializeFrom: aStream) root. > content := (aMaterializer materializeFrom: aStream) root ]. I like this solution best, since it doesn't change the Fuel "file format". I added some helper methods to my code base to make this usage scenario more obvious: FLMaterializer>>nextMaterializedRootFrom: sourceStream ^ (self materializeFrom: sourceStream) root FLMaterializer>>allMaterializedRootsFrom: sourceStream | contents | contents := OrderedCollection new. [ sourceStream atEnd ] whileFalse: [ contents add: (self nextMaterializedRootFrom: sourceStream) ]. ^ contents Sample usage: FileStream forceNewFileNamed: 'demoMetadata.fuel' do: [:aStream | | serializer | aStream binary. serializer := FLSerializer newDefault. (1 to: 5) do: [ :each | serializer serialize: (Dictionary with: each -> (each * each)) on: aStream ]. ]. FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream | | aMaterializer | aStream binary. aMaterializer := FLMaterializer newDefault. aMaterializer allMaterializedRootsFrom: aStream ]. FileStream oldFileNamed: 'demoMetadata.fuel' do: [:aStream | | aMaterializer | aStream binary. aMaterializer := FLMaterializer newDefault. aMaterializer nextMaterializedRootFrom: aStream. aMaterializer nextMaterializedRootFrom: aStream. ]. |
On Fri, Feb 10, 2012 at 2:45 PM, Yanni Chiu <[hidden email]> wrote:
Great. Maybe we should include #nextMaterializedRootFrom: to fuel. Based on metadata you might decide to not materialize the "real" content. Like:
metadata := (aMaterializer nextMaterializedRootFrom: aStream). [ metadata isWhatINeed ] ifTrue: [ realContent := aMaterializer nextMaterializedRootFrom: aStream ].
|
On 10/02/12 1:03 PM, Martin Dias wrote:
> > Great. Maybe we should include #nextMaterializedRootFrom: to fuel. That would be great. Probably best to leave off #allMaterializedRootsFrom: - it seemed necessary for completeness, but I've no actual need for it. > Based on metadata you might decide to not materialize the "real" > content. Like: > > metadata := (aMaterializer nextMaterializedRootFrom: aStream). > [ metadata isWhatINeed ] ifTrue: [ > realContent := aMaterializer nextMaterializedRootFrom: aStream ]. Yes, that's the usage scenario. |
In reply to this post by Mariano Martinez Peck
On Fri, Feb 10, 2012 at 1:19 AM, Mariano Martinez Peck <[hidden email]> wrote:
e.g. for source code. Imagine storing Monticello packages in Fuel so that the source code is stored in the Fuel file and doesn't have to be written to the changes file. Instead, a more powerful SourceFilesArray (actually a source file manager) can maintain a set of source files, including Fuel files, and fetch source there-from. Hence loading is faster, and the changes file isn't polluted with code loads, containing only one's own code.
e.g. for external resources. imagine loading a Fuel package that also contains some dlls that can be unpacked as required. But if you choose this route let me strongly suggest you use the zip file format. Its very useful with Monticello mcz's to be able to pick them apart using unzip.
best, Eliot |
On 10 February 2012 19:27, Eliot Miranda <[hidden email]> wrote:
> > > But if you choose this route let me strongly suggest you use the zip file > format. Its very useful with Monticello mcz's to be able to pick them apart > using unzip. > yes but then it makes the whole thing completely orthogonal to Fuel. because in zip you can have one file to hold serialized stuff, in another your metadata and whatsoever.. i mean, you can organize your data packaging yourself, without forcing fuel to support anything related to metadata. and i think this is right direction. So. I think it is useless to try extending fuel with metadata. if people want to have metadata, they should make it by own, in the way they see fit. I think fuel should do one thing : serialize/materialize stuff. Lets not turn it into "i can do everything to please you" library. -- Best regards, Igor Stasenko. |
In reply to this post by Yanni Chiu
this is a good idea.
On Feb 10, 2012, at 12:25 AM, Yanni Chiu wrote: > I'd like to store some metadata in a Fuel serialized file. I'd like to be able to read that metadata, without de-serializing the file. So, if there were a fixed header at the start of the file, with some "unused" fields, that would be ideal. > > |
In reply to this post by tinchodias
>
> Great. Maybe we should include #nextMaterializedRootFrom: to fuel. And I would love a chapter for our book :) But after mariano finished his phd draft :) |
In reply to this post by Igor Stasenko
Am 10.02.2012 um 19:51 schrieb Igor Stasenko: > On 10 February 2012 19:27, Eliot Miranda <[hidden email]> wrote: >> >> >> But if you choose this route let me strongly suggest you use the zip file >> format. Its very useful with Monticello mcz's to be able to pick them apart >> using unzip. >> > > yes but then it makes the whole thing completely orthogonal to Fuel. > because in zip you can have one file to hold serialized stuff, in > another your metadata and whatsoever.. > i mean, you can organize your data packaging yourself, without forcing > fuel to support anything related to metadata. > > and i think this is right direction. > > So. I think it is useless to try extending fuel with metadata. if > people want to have metadata, they should make it by own, > in the way they see fit. > I think fuel should do one thing : serialize/materialize stuff. Lets > not turn it into "i can do everything to please you" library. > Fuel does what it does. Having a fuel with metadata is just a customized format. In the end it is all streams. You can open a file read the metadata and then hand over the stream to fuel. I don't see any need to tweak fuel for that. Changing it to be able to read multiple roots sounds great but has the potential for e.g. doubling objects and all sorts (if I understand it correct). Norbert |
Norbert Hartl wrote:
> Am 10.02.2012 um 19:51 schrieb Igor Stasenko: > > >> On 10 February 2012 19:27, Eliot Miranda <[hidden email]> wrote: >> >>> But if you choose this route let me strongly suggest you use the zip file >>> format. Its very useful with Monticello mcz's to be able to pick them apart >>> using unzip. >>> >>> >> yes but then it makes the whole thing completely orthogonal to Fuel. >> because in zip you can have one file to hold serialized stuff, in >> another your metadata and whatsoever.. >> i mean, you can organize your data packaging yourself, without forcing >> fuel to support anything related to metadata. >> >> and i think this is right direction. >> >> So. I think it is useless to try extending fuel with metadata. if >> people want to have metadata, they should make it by own, >> in the way they see fit. >> I think fuel should do one thing : serialize/materialize stuff. Lets >> not turn it into "i can do everything to please you" library. >> >> > +1 > > Fuel does what it does. Having a fuel with metadata is just a customized format. In the end it is all streams. You can open a file read the metadata and then hand over the stream to fuel. I don't see any need to tweak fuel for that. Changing it to be able to read multiple roots sounds great but has the potential for e.g. doubling objects and all sorts (if I understand it correct). > > Norbert > > > > still want to specify exactly the format of a file ending in "*.fuel". Is it _always_ a zip file or do you look for the zipfile magic number and if that is not there revert to it being a single fuel stream. cheers, -ben |
In reply to this post by Stéphane Ducasse
On 10/02/12 3:45 PM, Stéphane Ducasse wrote:
> this is a good idea. Hmmm. I think the .zip file approach is the "correct/better" approach to take to my metadata problem. This discussion did turn up a feature of Fuel that was not obvious to me before - namely, that you can read/write multiple materializations on a single stream. |
In reply to this post by Eliot Miranda-2
Ok, I see. That makes sense. However, that won't be part of Fuel, but rather a tool build on top of it. Now we have the Fuel core a simple serializer. FuelMetalevel, an optional package build on top of Fuel core to be able to correctly serialize classes, methods, traits, etc. FuelPackageLoader, an optional package build on top of FuelMetalevel to experiment import/export of packages in a binary way. So....continue with that infrastructure, if there would be a Monticello integration with Fuel, then such package should take care about serializing different things in different streams. But so far, for Fuel core, and I think the best approach is to let is simple as it is now.
-- Mariano http://marianopeck.wordpress.com |
Free forum by Nabble | Edit this page |