ContentPack

Hello Casey and Juan

Good to see you active on this list.
How to I try out the ContentPack in Cuis 4.1?

Casey mentions in another thread that it might not work anymore.

--Hannes

On 12/30/12, Juan Vuletich <[hidden email]> wrote:
> Hi Hannes,
>
> You might be thinking on Casey's ContentPack, that is part of Cuis. It
> allows us to use only change sets for updating Cuis, while at the same
> time, using external tools for editing resources (like .bmp, .png and
> jpg files, etc).
>
> Cheers,
> Juan Vuletich
>
> H. Hirzel wrote:
.....
>>
>> If I remember well you once did a package for managing resources, right?
>>
>> Where is it?
>>
>> --Hannes
>>

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

Hi Hannes,

H. Hirzel wrote:

In the meantime I found out that


ContentPack seems to be a persistence mechanism for a dictionary which
may contain other dictionaries and instances of Form and ColorForm.

Forms are written out to the file system as *.png files and ColorForms
are written as *.bmp files.

Is this correct?

--Hannes

 

Yes it is correct. It is a bit more than that, though. Forms (and potentially other media types) can exist in three forms:
1) As external files, such as jpg, png, etc. This is the representation we need to use external tools (such as image processing apps, cameras, scanners, web, etc) to work on them.
2) As methods. Non human readable, base-64 encoded binary data. We need this to be able to include such stuff in the update stream, or in packages. After we update an image, we usually delete these methods, just keeping 3).
3) Live objects in the image, for example, stored in class variables. This is to make use of them in Cuis.

Most of the time, we use 3). But we need 2) for the update stream. We also need 1) sometimes to work on them. ContentPack supports the conversion between these 3 formats. The implementation is quite simple. What is really great is that Casey realized we need some tool to move comfortably between these 3 representations. And he also implemented it.

Please grab http://www.jvuletich.org/Cuis/CuisUpdatesUpTo1511.zip and take a look at updates 966, 967 and 968.

Maybe it is time for a bit more documentation, and usage examples...

Cheers,
Juan Vuletich

On 1/2/13, H. Hirzel <[hidden email]> wrote:
 

Hello Juan and Casey

Is the ContentPack something like Fuel?
http://rmod.lille.inria.fr/web/pier/software/Fuel
(BTW it is now available for Squeak as well, see announcement on the
Squeak list)?



I found class 'ContentPack', I copy it in below. The nice thing is
that it only has 11 instance methods and 7 class methods. However most
of the comment I do not understand.

I copy in the class comment below.
I put in comments in uppercase.

Kind regards
Hannes


////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
ContentPack lets you read in and write out the (supported files in
the) contents of a directory on your file system. It also allows you
to trivially create "messenger" subclasses that capture the
information containted (TYPO) in these directory trees, including any
implicit communication that's there in the structure of the directory
hierarchy itself, which are captured in your changes file.

NOTE: I DO NOT UNDERSTAND THIS LANGUAGE. EXAMPLES of supported file types?


You can then file out a change set that contains a representation of
the (supported file/object types and directory structurein) TYPO the
stuff on your disk, or in your image. This subclass is a dummy which
ContentPack compiles methods into containing base 64 encoded data. You
can load this into another image, as long as that image has
ContentPack loaded. The filed in class can then recreate the
ContentPack on the other end with the media files and structure
intact.

The current implementation is based on #storeString, but the plan is
to change that to SmartRefStream in the long run to support
serializing things like morphs.

ContentPack instances hang onto the actual tree of media objects.(I DO
NOT UNDERSTAND)  It has a nice simple EDSL ???? that just interprets
an array of strings from beginning to end as a "path" to a file
(really a series of dictionary lookups to a Smalltalk object, wherin
the dictionaries mirror the structure of what was on the disk, sans
unsupported files.) This mechanism will likely change a little bit at
some point,


ContentPack came into the world a little faster than I expected, as I
ended up using it to send some icons back in time to fix the Cuis
update stream without having to sort my changes all over again. As
such it had some unusual design pressures... it had to be able to
carry information in and out of both the change set stream and the
filesystem, as well as function in a slightly earlier (unreleased)
version of Cuis than it was written in, and not break anything on it's
way back up through the build to head.

SENDING ICONS BACK IN TIME????


The code, in particular the way things are named, has not settled yet,
and that's why this comment contains no code examples. Use with care
and read the code first, for now.

IN THE MEANTIME THE CODE SEEMS TO HAVE SETTLED OTHERWISE IT WOULD NOT
BE IN CORE.


Currently, .bmp import and .png import are implemented, and both can
be exported. Anything you can import, you can also shuffle into a
change set. Plans are in the works to support audio, change sets, and
text files.

PLEASE DO SO. GIVEN THE FACT THAT THIS CLASS ONLY HAS 18 METHODS THIS
SHOULD NOT BE A LARGE EFFORT.

I'll support video if someone has a good importer, exporter, and
player under the MIT license that'll work under Cuis.

Currently, objects are serialized into single methods, which works for
small icons, but likely doesn't work well (if at all) for larger
files. My intent is to add some behavior that breaks up large objects
into smaller chunks so that this becomes a non-issue. I'll likely get
to that when I've removed most of the repetitive subtle variations of
the same recursive tree walking visitor-trick from the code, and
renamed everything. I think in essence this class is slightly smaller
than it is as represented currently.

Hopefully I will be able to explain all of this better once I've
clarified the code a bit so that I can show off some examples.

YES.   EXAMPLES WILL HELP A LOT   :-)

        - cbr

On 1/2/13, Juan Vuletich <[hidden email]> wrote:
   

Hi Folks,

I think it should work ok. I don't recall doing any changes that would
obviously affect it.

BTW, a bit more of documentation wouldn't hurt, but the code is all
there, and there's a reasonable class comment. It is just a matter of
learning and playing a bit with it.

Cheers,
Juan Vuletich

Casey Ransberger wrote:
     

Oh wow and ouch. I wrote it. My fault. I didn't document it. I
expected, after I'd used it to muck with the time stream, that we'd
throw it away once the paradox was resolved, but Juan liked it, and
wanted to keep it.

Anyway, it's my dog and I've been terrible about feeding it. I should
fix that.

On Tue, Jan 1, 2013 at 9:58 PM, H. Hirzel <[hidden email]
<mailto:[hidden email]>> wrote:

    Hello Casey and Juan

    Good to see you active on this list.
    How to I try out the ContentPack in Cuis 4.1?

    Casey mentions in another thread that it might not work anymore.

    --Hannes

    On 12/30/12, Juan Vuletich <[hidden email]
    <mailto:[hidden email]>> wrote:
    > Hi Hannes,
    >
    > You might be thinking on Casey's ContentPack, that is part of
    Cuis. It
    > allows us to use only change sets for updating Cuis, while at
    the same
    > time, using external tools for editing resources (like .bmp,
    .png and
    > jpg files, etc).
    >
    > Cheers,
    > Juan Vuletich
    >
    > H. Hirzel wrote:
    .....
    >>
    >> If I remember well you once did a package for managing
    resources, right?
    >>
    >> Where is it?
    >>
    >> --Hannes
    >>

    _______________________________________________
    Cuis mailing list
    [hidden email] <mailto:[hidden email]>
    http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org




--
Casey Ransberger
------------------------------------------------------------------------

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

       

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

     

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org


 

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org

Hello all

I published a rewrite of ContentPack [1] on

    https://github.com/hhzl/ContentPack

It includes tests and documentation. It can deal with larger instances
of class Form and ColorForm. It splits them so that they are stored in
several compiled methods.

Included is an example of a part of an ABC book, 4MB compressed, 8MB expanded.

My conclusion at the moment

Load time is quite long (>5 min), so a bit of patience is needed. The
interesting fact is that in this test case Cuis can deal with a single
class with 8MB source code; on the other side the speed is not good.

In the future we need to save the content in compressed format and
probably not all in the same class. However at the moment it is useful
for me as-is to move content around and therefore I publish it so that
I can focus on something else next.

The ContentPack idea can be applied in other areas as well.

Comments and code reviews are invited.

Kind regards

Hannes Hirzel



[1]
From:

http://www.jvuletich.org/pipermail/cuis_jvuletich.org/2012-May/000025.html

ContentPack

The idea is to let collaborators of art resources (graphics and sound
designers) use whatever tool they prefer. For them, the resources are
PNG or WAV. But we need to get them in source code, to load them as
updates. ContentPack takes those external files and turns them into
code (that creates live instances). Then the change set is included in
the update stream. As we have the live objects, a later updates
removes all that code. And we are done. Later, if we want to do
another run of edition with external tools, ContentPack lets us export
the resources as files, so our artist updates them. Then the process
is repeated.

The following is copied from the release notes of Cuis 3.3:

ContentPack - A clean solution for a problem Squeak had for over a
decade! (by Casey Ransberger)

    Manages internal/external resources
    Allows import / export to enable use of use existing artifacts and
external tools
    Does not depend on external files for image update
    Updates done with code (enabling change sets of Monticello packages)
    Avoids cruft accumulation, code for resources is removed after update

All these properties are important, and ContentPack solves the issue
really well.

On 1/18/13, H. Hirzel <[hidden email]> wrote:
> Hello Juan and Casey
>
> Thank you Juan, for the elaboration with the added class comment.
> When looking at updates 966, 967 and 968 I realized that I actually
> have to subclass it to use it.
>
>
> The fact that ContentPack takes care of the conversions between
>
> a) Live objects (as of now instances of Form and ColorForm in
> dictionaries of dictionaries)
> b) their representation as Smalltalk methods in a single ContentPack
> subclass
> c) the storage on the disk (as *.png and *.bmp)
>
> makes it very valuable for constructing learning and other games as
> Casey points out. Actually it is a need.
>
> There is actually little code, but as it stands now it is difficult to
> understand because of the naming used and missing convenience methods.
>
> I have started on a rewrite of the class with factoring methods and
> more comments added.
>
> I will present the result soon for review.
>
>
> Kind regards
>
> Hannes Hirzel
>
>
> On 1/4/13, Casey Ransberger <[hidden email]> wrote:
>> Hannes, that's great. JPG support would be awesome to have; eventually,
>> in
>> some wonderful someday, it'd be cool to add support for other media types
>> like mpeg and mp3, it's just a matter of having a way to bring
>> disk-representations in, and some means to reduce them to base64 or some
>> other convenient textual encoding (and also possibly the need to adjust a
>> limitation to the size of a change set, IIRC.)
>>
>> One of the things I'd like to do with Cuis is games, and so multimedia
>> support is something I care about. It's just a matter of finding the
>> time.
>> I will try to work on the documentation part soon.
>>
>> On Thu, Jan 3, 2013 at 6:24 AM, H. Hirzel <[hidden email]>
>> wrote:
>>
>>> Thank you Juan, for giving more details. A follow up question
>>> regarding jpg support below....
>>>
>>> --Hannes
>>>
>>> On 1/3/13, Juan Vuletich <[hidden email]> wrote:
>>> > Hi Hannes,
>>> >
>>> > H. Hirzel wrote:
>>> >> In the meantime I found out that
>>> >>
>>> >>
>>> >> ContentPack seems to be a persistence mechanism for a dictionary
>>> >> which
>>> >> may contain other dictionaries and instances of Form and ColorForm.
>>> >>
>>> >> Forms are written out to the file system as *.png files and
>>> >> ColorForms
>>> >> are written as *.bmp files.
>>> >>
>>> >> Is this correct?
>>> >>
>>> >> --Hannes
>>> >>
>>> >>
>>> >
>>> > Yes it is correct. It is a bit more than that, though. Forms (and
>>> > potentially other media types) can exist in three forms:
>>> > 1) As external files, such as jpg, png, etc. This is the
>>> > representation
>>> > we need to use external tools (such as image processing apps, cameras,
>>> > scanners, web, etc) to work on them.
>>>
>>> Speaking of cameras. I'd like to include jpg files. However I do not
>>> see a support of for them. On the class side of ContentPack we have
>>> the method
>>>
>>>
>>>
>>> mapping
>>>
>>>         ^ {
>>>                 ColorForm -> #bmp .
>>>                 Form -> #png
>>>         }
>>>
>>>
>>> > 2) As methods. Non human readable, base-64 encoded binary data. We
>>> > need
>>> > this to be able to include such stuff in the update stream, or in
>>> > packages. After we update an image, we usually delete these methods,
>>> > just keeping 3).
>>> > 3) Live objects in the image, for example, stored in class variables.
>>> > This is to make use of them in Cuis.
>>> >
>>> > Most of the time, we use 3). But we need 2) for the update stream. We
>>> > also need 1) sometimes to work on them. ContentPack supports the
>>> > conversion between these 3 formats. The implementation is quite
>>> > simple.
>>> > What is really great is that Casey realized we need some tool to move
>>> > comfortably between these 3 representations. And he also implemented
>>> > it.
>>> >
>>> > Please grab http://www.jvuletich.org/Cuis/CuisUpdatesUpTo1511.zip and
>>> > take a look at updates 966, 967 and 968.
>>> >
>>> > Maybe it is time for a bit more documentation, and usage examples...
>>> >
>>> > Cheers,
>>> > Juan Vuletich
>>> >
>>> >> On 1/2/13, H. Hirzel <[hidden email]> wrote:
>>> >>
>>> >>> Hello Juan and Casey
>>> >>>
>>> >>> Is the ContentPack something like Fuel?
>>> >>> http://rmod.lille.inria.fr/web/pier/software/Fuel
>>> >>> (BTW it is now available for Squeak as well, see announcement on the
>>> >>> Squeak list)?
>>> >>>
>>> >>>
>>> >>>
>>> >>> I found class 'ContentPack', I copy it in below. The nice thing is
>>> >>> that it only has 11 instance methods and 7 class methods. However
>>> >>> most
>>> >>> of the comment I do not understand.
>>> >>>
>>> >>> I copy in the class comment below.
>>> >>> I put in comments in uppercase.
>>> >>>
>>> >>> Kind regards
>>> >>> Hannes
>>> >>>
>>> >>>
>>> >>>
>>> ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
>>> >>> ContentPack lets you read in and write out the (supported files in
>>> >>> the) contents of a directory on your file system. It also allows you
>>> >>> to trivially create "messenger" subclasses that capture the
>>> >>> information containted (TYPO) in these directory trees, including
>>> >>> any
>>> >>> implicit communication that's there in the structure of the
>>> >>> directory
>>> >>> hierarchy itself, which are captured in your changes file.
>>> >>>
>>> >>> NOTE: I DO NOT UNDERSTAND THIS LANGUAGE. EXAMPLES of supported file
>>> >>> types?
>>> >>>
>>> >>>
>>> >>> You can then file out a change set that contains a representation of
>>> >>> the (supported file/object types and directory structurein) TYPO the
>>> >>> stuff on your disk, or in your image. This subclass is a dummy which
>>> >>> ContentPack compiles methods into containing base 64 encoded data.
>>> >>> You
>>> >>> can load this into another image, as long as that image has
>>> >>> ContentPack loaded. The filed in class can then recreate the
>>> >>> ContentPack on the other end with the media files and structure
>>> >>> intact.
>>> >>>
>>> >>> The current implementation is based on #storeString, but the plan is
>>> >>> to change that to SmartRefStream in the long run to support
>>> >>> serializing things like morphs.
>>> >>>
>>> >>> ContentPack instances hang onto the actual tree of media objects.(I
>>> >>> DO
>>> >>> NOT UNDERSTAND)  It has a nice simple EDSL ???? that just interprets
>>> >>> an array of strings from beginning to end as a "path" to a file
>>> >>> (really a series of dictionary lookups to a Smalltalk object, wherin
>>> >>> the dictionaries mirror the structure of what was on the disk, sans
>>> >>> unsupported files.) This mechanism will likely change a little bit
>>> >>> at
>>> >>> some point,
>>> >>>
>>> >>>
>>> >>> ContentPack came into the world a little faster than I expected, as
>>> >>> I
>>> >>> ended up using it to send some icons back in time to fix the Cuis
>>> >>> update stream without having to sort my changes all over again. As
>>> >>> such it had some unusual design pressures... it had to be able to
>>> >>> carry information in and out of both the change set stream and the
>>> >>> filesystem, as well as function in a slightly earlier (unreleased)
>>> >>> version of Cuis than it was written in, and not break anything on
>>> >>> it's
>>> >>> way back up through the build to head.
>>> >>>
>>> >>> SENDING ICONS BACK IN TIME????
>>> >>>
>>> >>>
>>> >>> The code, in particular the way things are named, has not settled
>>> >>> yet,
>>> >>> and that's why this comment contains no code examples. Use with care
>>> >>> and read the code first, for now.
>>> >>>
>>> >>> IN THE MEANTIME THE CODE SEEMS TO HAVE SETTLED OTHERWISE IT WOULD
>>> >>> NOT
>>> >>> BE IN CORE.
>>> >>>
>>> >>>
>>> >>> Currently, .bmp import and .png import are implemented, and both can
>>> >>> be exported. Anything you can import, you can also shuffle into a
>>> >>> change set. Plans are in the works to support audio, change sets,
>>> >>> and
>>> >>> text files.
>>> >>>
>>> >>> PLEASE DO SO. GIVEN THE FACT THAT THIS CLASS ONLY HAS 18 METHODS
>>> >>> THIS
>>> >>> SHOULD NOT BE A LARGE EFFORT.
>>> >>>
>>> >>> I'll support video if someone has a good importer, exporter, and
>>> >>> player under the MIT license that'll work under Cuis.
>>> >>>
>>> >>> Currently, objects are serialized into single methods, which works
>>> >>> for
>>> >>> small icons, but likely doesn't work well (if at all) for larger
>>> >>> files. My intent is to add some behavior that breaks up large
>>> >>> objects
>>> >>> into smaller chunks so that this becomes a non-issue. I'll likely
>>> >>> get
>>> >>> to that when I've removed most of the repetitive subtle variations
>>> >>> of
>>> >>> the same recursive tree walking visitor-trick from the code, and
>>> >>> renamed everything. I think in essence this class is slightly
>>> >>> smaller
>>> >>> than it is as represented currently.
>>> >>>
>>> >>> Hopefully I will be able to explain all of this better once I've
>>> >>> clarified the code a bit so that I can show off some examples.
>>> >>>
>>> >>> YES.   EXAMPLES WILL HELP A LOT   :-)
>>> >>>
>>> >>>     - cbr
>>> >>>
>>> >>> On 1/2/13, Juan Vuletich <[hidden email]> wrote:
>>> >>>
>>> >>>> Hi Folks,
>>> >>>>
>>> >>>> I think it should work ok. I don't recall doing any changes that
>>> >>>> would
>>> >>>> obviously affect it.
>>> >>>>
>>> >>>> BTW, a bit more of documentation wouldn't hurt, but the code is all
>>> >>>> there, and there's a reasonable class comment. It is just a matter
>>> >>>> of
>>> >>>> learning and playing a bit with it.
>>> >>>>
>>> >>>> Cheers,
>>> >>>> Juan Vuletich
>>> >>>>
>>> >>>> Casey Ransberger wrote:
>>> >>>>
>>> >>>>> Oh wow and ouch. I wrote it. My fault. I didn't document it. I
>>> >>>>> expected, after I'd used it to muck with the time stream, that
>>> >>>>> we'd
>>> >>>>> throw it away once the paradox was resolved, but Juan liked it,
>>> >>>>> and
>>> >>>>> wanted to keep it.
>>> >>>>>
>>> >>>>> Anyway, it's my dog and I've been terrible about feeding it. I
>>> >>>>> should
>>> >>>>> fix that.
>>> >>>>>
>>> >>>>> On Tue, Jan 1, 2013 at 9:58 PM, H. Hirzel <[hidden email]
>>> >>>>> <mailto:[hidden email]>> wrote:
>>> >>>>>
>>> >>>>>     Hello Casey and Juan
>>> >>>>>
>>> >>>>>     Good to see you active on this list.
>>> >>>>>     How to I try out the ContentPack in Cuis 4.1?
>>> >>>>>
>>> >>>>>     Casey mentions in another thread that it might not work
>>> >>>>> anymore.
>>> >>>>>
>>> >>>>>     --Hannes
>>> >>>>>
>>> >>>>>     On 12/30/12, Juan Vuletich <[hidden email]
>>> >>>>>     <mailto:[hidden email]>> wrote:
>>> >>>>>     > Hi Hannes,
>>> >>>>>     >
>>> >>>>>     > You might be thinking on Casey's ContentPack, that is part
>>> >>>>> of
>>> >>>>>     Cuis. It
>>> >>>>>     > allows us to use only change sets for updating Cuis, while
>>> >>>>> at
>>> >>>>>     the same
>>> >>>>>     > time, using external tools for editing resources (like .bmp,
>>> >>>>>     .png and
>>> >>>>>     > jpg files, etc).
>>> >>>>>     >
>>> >>>>>     > Cheers,
>>> >>>>>     > Juan Vuletich
>>> >>>>>     >
>>> >>>>>     > H. Hirzel wrote:
>>> >>>>>     .....
>>> >>>>>     >>
>>> >>>>>     >> If I remember well you once did a package for managing
>>> >>>>>     resources, right?
>>> >>>>>     >>
>>> >>>>>     >> Where is it?
>>> >>>>>     >>
>>> >>>>>     >> --Hannes
>>> >>>>>     >>
>>> >>>>>
>>> >>>>>     _______________________________________________
>>> >>>>>     Cuis mailing list
>>> >>>>>     [hidden email] <mailto:[hidden email]>
>>> >>>>>     http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> Casey Ransberger
>>> >>>>>
>>> ------------------------------------------------------------------------
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> Cuis mailing list
>>> >>>>> [hidden email]
>>> >>>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>>>>
>>> >>>>>
>>> >>>> _______________________________________________
>>> >>>> Cuis mailing list
>>> >>>> [hidden email]
>>> >>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>>>
>>> >>>>
>>> >>
>>> >> _______________________________________________
>>> >> Cuis mailing list
>>> >> [hidden email]
>>> >> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Cuis mailing list
>>> > [hidden email]
>>> > http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>> >
>>>
>>> _______________________________________________
>>> Cuis mailing list
>>> [hidden email]
>>> http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org
>>>
>>
>>
>>
>> --
>> Casey Ransberger
>>
>

_______________________________________________
Cuis mailing list
[hidden email]
http://jvuletich.org/mailman/listinfo/cuis_jvuletich.org