Hi Kilon,

I think what you are describing as a need for you is the ability to directly load into an image chunks of image memory space, containing already initialized images and code (i.e. live objects stored one way or another). Fuel could be an answer to that (wasn't that Tanker, the Fuel for code?).

Now, whether those are files or not does not concern me. Give me raw blocks on a disk or magic requests to do over a network to get those image chunks and I'll be happy.

Thierry

2015-12-08 1:06 GMT+01:00 Dimitris Chloupis <[hidden email]>:

You could not ask the worst case person even if you wanted :D

But here we go ...

Will depend , let take my ChronosManager for example

To modify the existing code of ChronosManager , is just text that gets compiled to bytecode immediately on each accept, no need to be loaded from a file, even though it gets added to the source and changes files.

To add code to the image whether its the entire code of ChronosManager or just a commit, it will load the code from st files, plus the extra filetree metadata files for class definition, version control and method categorization. I use Github but its a very similar story for those using Smalltalkhub.

To add code that contains data like an image, like I do with the GUI resources , I use a large collection of PNG files which with the help of the IconFactory will compile the binary data to a bytecode string and also pass it as a form to be used immediately. I am thinking instead to replace the iconfactory with direct access to the png files and creation of the form or a need to exist basis. Lazy loading in short.

In the future that I will need a common ground for all the images that use ChronoManager to share data and live state , I will be using fuel files.

For Ephestos (ChronosManager is part of the Ephestos project)  that has to deal with 3d data, the collection of files grow exponentially which I try to keep outside the image because that data can easily reach several GBs and I dont want the image to explode in my face. To give you an example one folder I have with several blender files is around 2 gbs tops, each file being more than 250 MB. Blend files by the way implement their own kind of "object orientation" even though Blender is largely coded in C. 

So if you ask me how many files I have to pass to the opal compiler if all I wanted is to stick just with the image my answer would be "a ton" but its a moot a point since the image cant handle so much data anyway. But even in the current scenario, its still a lot of files if you count the pngs and the repository .

I actually now try to remove my dependency on SmaCC because it makes the load time very slow, apparently compilation of large projects in the image is super slow and I will divert the parsing back to the python side which is way faster and since its python syntax i am parsing, python is really good at it anyway and helps me remove one unnecessary dependency. Though I have to say I am very happy with SmaCC and its great to have it around. 

In the future I predict that I will have to keep more and more things into files to make loading of Ephestos fast and also its execution because image is not really optimized for what I want it to use it for.

So in my case, its files all the way !

On Tue, Dec 8, 2015 at 1:09 AM Ben Coman <[hidden email]> wrote:

Just one simple question...
When you want Pharo to compile some code, which *file* do you pass to
our Opal compiler?

cheers -ben

On Tue, Dec 8, 2015 at 2:02 AM, Dimitris Chloupis <[hidden email]> wrote:
> The devil is in the details ;)
>
> It matters to me, I just came across the need to share data between multiple
> images. So I was pointed by the good people here to the Fuel library that ,
> surprise surprise , it generates binary files that contain objects in their
> live state that helps you move and share code and data between images. Works
> well and I really like its design :)
>
> We are not talking here about something sophisticated, we are talking here
> super basic functionality. Images sharing data and code. What we use ?
> Files. The image by itself has no functionality to even cover this super
> basic scenario because as a format is made to be self contained.
>
> How you cant even care for such basic functionality ? Of course you will at
> some point. Its unavoidable.
>
> The nice thing about files is that they have one very big advantage over the
> image. That is, specialization. When an app find a specific file , just by
> looking at its extension it immediately knows the structure of the data and
> the code that it may contain.
>
> On other hand when you have an object system like the image is, such
> specifications go outside the window meaning you have to deal with the fact
> and trust that those that made those images have adhered to specific
> guidelines so you can make sure that your code wont run in front of some
> very nasty surprises.
>
> But since the image itself allow you hack so deeply as the syntax of the
> language , you can't be sure how the data and code will be presented. Sure
> they will objects, but the format does not really matter so much as the
> structure itself.
>
> In those cases files win hands down because they tend to be far more
> restricted on how they are structured. Not because there is anything special
> to these files, apart from the fact that their authors made sure to follow
> the specific structure to ensure compatibility with third party apps.
>
> So not only Files are not on the Stone Age but they have evolved the level
> of specification to a whole new level that have made the foundation of our
> every day lives.
>
> Sure you could probably replace files with a new way that is more Smalltalk
> friendly and still retain all the advantages of files and file system but ,
> Smalltalk has not presented such solution to my knowledge. Hence we the
> smalltalkers we will still keep relying heavily on files for our every day
> needs until such solution is presented to us. Also with the huge wealth of
> file formats it would be a pain in the ass to replace them with a smalltalk
> solution.
>
> In the mean time there are even more pressing matter that the image file has
> to attend to, which is far more stone age , to use your remark , than files.
> That is the ability to use full memory of the system and the ability to deal
> with large data without any large hits on performance. In short good support
> for 64 bit and big data.
>
>
> "But these are implementation details...implementation of the base system.
> /From the perspective of a programmer writing an application/, none of this
> matters.
>
> As I said earlier, the only reason why Smalltalk has to deal with files at
> all is because we live in a file-based culture. And the reason our culture
> is so entrenched with files is because we are too heavily invested in them,
> and we aren't going to budge. *Files are about as low a storage abstraction
> as you can get*, and they pre-date even Unix. Yes, files belong in the Stone
> Age!"
>
> On Mon, Dec 7, 2015 at 6:45 PM horrido <[hidden email]> wrote:
>>
>> But these are implementation details...implementation of the base system.
>> /From the perspective of a programmer writing an application/, none of
>> this
>> matters.
>>
>> As I said earlier, the only reason why Smalltalk has to deal with files at
>> all is because we live in a file-based culture. And the reason our culture
>> is so entrenched with files is because we are too heavily invested in
>> them,
>> and we aren't going to budge. *Files are about as low a storage
>> abstraction
>> as you can get*, and they pre-date even Unix. Yes, files belong in the
>> Stone
>> Age!
>>
>>
>>
>> kilon.alios wrote
>> > That's the thing you can't take the argument further without diminishing
>> > the value of you argument precisely for the fact that the vm is far
>> > closer
>> > related to the image than it is to 0s and 1s. That tight relation is
>> > fundamental to the behavior and existence of the image. It defines its
>> > functionality, purpose and limitations.
>> >
>> > The image itself is a file and the fact that it can store live state in
>> > a
>> > binary format does not make it unique or any less of a file. In my case
>> > I
>> > use blender files, they store the entire live state of the blender
>> > window
>> > including images and even Python scripts. Similar examples are countless
>> > out there.
>> >
>> > So the answer to the question what makes the image file format unique is
>> > simply.... Nothing
>> > What's the advantage of using the image format compared to other files ?
>> > None
>> >
>> > On Mon, 7 Dec 2015 at 15:14, Ben Coman &lt;
>>
>> > btc@
>>
>> > &gt; wrote:
>> >
>> >> On Mon, Dec 7, 2015 at 3:37 PM, Dimitris Chloupis &lt;
>>
>> > kilon.alios@
>>
>> > &gt;
>> >> wrote:
>> >> > "A Smalltalk Image is your entire system. The Image includes all the
>> >> tools
>> >> > required to interact, customize and add functionality to your system,
>> >> so
>> >> > Smalltalk’s IDE is a very Integrated Development Environment."
>> >> >
>> >> >
>> >> > Thats not the case even for someone like me that has been working
>> >> > with
>> >> > smalltalk for only 2 years. The Image is not even the engine that
>> >> drives
>> >> > smalltalk . Thats the job of the VM that exists in a completely
>> >> different
>> >> > universe than smalltalk. It exists in the same universe than many
>> >> > other
>> >> > languages do exists and thats the C universe, the universe of the OS.
>> >> > Essentially what drives your system is not smalltalk is C. The
>> >> diffirence is
>> >> > that for a part of it that is high level enough, Slang is used, a
>> >> Hybrid
>> >> > language between C and Smalltalk that compiles to C. So while in the
>> >> image
>> >> > everything is , well almost everything, an object all the way down,
>> >> > in
>> >> the
>> >> > VM everything is C all the way down.
>> >>
>> >> To take that argument further, the VM is not even the thing driving
>> >> the image ;).  Essentially what drives it are the 1's and 0's of
>> >> machine code.  Further, what drives that are the electrons flowing
>> >> through the chip.  I think its fair to say that we *code* in Pharo
>> >> without files.  Files relate to Pharo only to the same extent that a
>> >> database like Oracle or Postgres can be said to use files.  That is,
>> >> when you do SQL queries, are you *thinking* in terms of files, even
>> >> though files are used by the server to store the data? Its just a
>> >> matter of where you draw the line of abstraction.
>> >>
>> >> cheers -ben
>> >>
>> >> > Ironically an image misses the most important tool to even generate
>> >> this
>> >> C
>> >> > code and thats the VMMaker that has to be installed separately. And
>> >> > of
>> >> > course there are parts of the system that are coded in pure C, like
>> >> some
>> >> > core functionalities of the VM and of course plugins and external
>> >> libraries
>> >> > that the image has to rely on make things happen.
>> >> >
>> >> > Of course the image is still fairly powerful, you can change the
>> >> syntax,
>> >> > implement high level libraries, IDE tools and much more. But its not
>> >> the
>> >> > core of the system just another essential part of it.
>> >> >
>> >> >
>> >> > On Mon, Dec 7, 2015 at 9:24 AM Dimitris Chloupis &lt;
>>
>> > kilon.alios@
>>
>> > &gt;
>> >> > wrote:
>> >> >>>
>> >> >>>
>> >> >>> well, i wouldn't need or even want it in memory, so on disk is
>> >> >>> fine.
>> >> the
>> >> >>> problem is more likely management of the same. browsing the changes
>> >> is
>> >> >>> not
>> >> >>> really convenient.  ideally i'd like to see versions in the
>> >> class-browser
>> >> >>> and
>> >> >>> in the debugger, where on error i could then take a look at older
>> >> >>> versions for
>> >> >>> comparison, and switch to them to see if maybe the last change was
>> >> the
>> >> >>> cause of
>> >> >>> the error.
>> >> >>>
>> >> >>> greetings, martin.
>> >> >>>
>> >> >>
>> >> >> There are versions already for methods. So the functionality is
>> >> >> there.
>> >> >>
>> >> >> I disagree however with you, I think that changes file was created
>> >> >> for
>> >> the
>> >> >> precise scenarios of an image crash/ lockdown. In that case you may
>> >> want to
>> >> >> go back through the code and dont remember which method was
>> >> >> triggered
>> >> or
>> >> >> what else was defined and created. In the case going chronologically
>> >> which
>> >> >> is how the changes file is already organised is far more useful than
>> >> going
>> >> >> method and class based.
>> >> >>
>> >> >> But I do agree it would be useful to extend the tools working with
>> >> changes
>> >> >> , but then none stop anyone from doing so and is not that hard to
>> >> >> do.
>> >>
>> >>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://forum.world.st/Stop-Thinking-in-Terms-of-Files-tp4865614p4865820.html
>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
>>
>