FileSystem (was: About (backwards) Compatibility)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

FileSystem (was: About (backwards) Compatibility)

Chris Muller-3
>>  I have no interest in chopping on the great work being done by
>> you and my other friends in Pharo, but that doesn't mean it is
>> feasible for me to use it in my business.  While someone in the Pharo
>> community said FileSystem over FileDirectory is "huge", I see it as an
>> incremental API change, and close to being a matter of preference.
>>
> incremental?
> do you think you can implement a memory-based and/or git-based filesystem or
> 'remotely connected database-based file system'
> by just doing incremental changes to FileDirectory?
> good luck with such 'increments' :)

Yes, of course those capabilities could be added simply with factoring
work and little disruption to the API.  That's the promise object
technology, encapsulation.

Colin made a gorgeous domain model of an abstract FileSystem, no doubt
about it.  If we switch to it in Squeak I'm sure I will be happy once
the conversion is done.  I just think its worth asking, though,
whether a "core" system should provide that rich a model
out-of-the-box or instead just a bland, one-layer-above-the-primitives
lightweight model ready to be easily wrapped by the user's _own_ rich
model of a FileSystem.  Is the core system suitable for tiny embedded
programs and will they want a rich model or a basic one?  Maybe Spoon
will allow us to have our cake and eat it too..?

Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

Igor Stasenko
On 13 December 2012 03:59, Chris Muller <[hidden email]> wrote:

>>>  I have no interest in chopping on the great work being done by
>>> you and my other friends in Pharo, but that doesn't mean it is
>>> feasible for me to use it in my business.  While someone in the Pharo
>>> community said FileSystem over FileDirectory is "huge", I see it as an
>>> incremental API change, and close to being a matter of preference.
>>>
>> incremental?
>> do you think you can implement a memory-based and/or git-based filesystem or
>> 'remotely connected database-based file system'
>> by just doing incremental changes to FileDirectory?
>> good luck with such 'increments' :)
>
> Yes, of course those capabilities could be added simply with factoring
> work and little disruption to the API.  That's the promise object
> technology, encapsulation.
>

The main problem with FileDirectory that there's no API.
I can hardly call a single class as API...
as to me, it is clear that file system domain is a bit more complex
and in order to reflect it properly in a language you will need more
that a single role (FileDirectory).

So, i really doubt that you end up with 'little disruption to API'. It
will be substantial.

> Colin made a gorgeous domain model of an abstract FileSystem, no doubt
> about it.  If we switch to it in Squeak I'm sure I will be happy once
> the conversion is done.  I just think its worth asking, though,
> whether a "core" system should provide that rich a model
> out-of-the-box or instead just a bland, one-layer-above-the-primitives
> lightweight model ready to be easily wrapped by the user's _own_ rich
> model of a FileSystem.  Is the core system suitable for tiny embedded
> programs and will they want a rich model or a basic one?  Maybe Spoon
> will allow us to have our cake and eat it too..?
>

To me, design comes first. ALWAYS.
All of the rest is secondary. FileSystem establishing a clear
separation between conceptual model
and implementation detail.
The value of such improvement is hard to underestimate: users can
learn fast & grasp well
how to use it (and even understand better how file systems work).
Moreover, it makes FileSystem nicely portable among dialects.. So, in
future, if you would like to port your application to another dialect,
you'll have to port FileSystem backend, instead of walking over your
whole application and do patches all over the place..
Because once you have sound API, an effort to port application
gravitates down to zero.
Also, porting, in broad sense can mean 'porting from version X to
version Y' (speaking about 'backwards compatimilimulability).

API establishing a set of rules in a domain & models. The point is
that domain does not changes that often, comparing to implementation.
That means , if you have a good design, you can be assured that it
won't require serious changes and will serve you much much longer
without need to change anything in userland code. (Take a Collection
protocols for example).
Also, once you have such set of rules, it is much easier for people to
maintain implementation code,
as well as adding extra functionality if needed.

Clearly, VM interfaces (such as primitives) should be not exposed to
language 'as is', but always come with API.
Because if you don't have rules, then you have very poor
maintainability inside of system, as well as forcing users to reinvent
their own wheel over and over again.. And next thing, if you would
want one day to change primitives or change VM.. you will immediately
hit the wall, with bloody 'backwards compatibility' sign on it.
Because you will discover, that every single application implements
own abstraction layer(s) (or even worse, use low-level functionality
directly) on top of what you provided.

So, to me, your statement looks contradictory:
from one side, you are seriously concerned about backwards compatibility,
but from another one, you trying to ignore/discard the facts that you
cannot have good
backwards compatibility without having a design and API in a first place.

--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

Igor Stasenko
a little followup..

Please, do not even try to say that FileDirectory and FileSystem is
interchangeable :)

Colin laid a foundation in a place where was no foundation at all.
Yes, in Pharo we did some changes in order to 'make it work',
polishing some rough corners,
as well as writing tests and documenting the code.

Without laying foundation, nobody starts to build houses.. (well, some
do.. but they do not stay long).
We even teach kids about that:
http://en.wikipedia.org/wiki/Three_Little_Pigs

--
Best regards,
Igor Stasenko.

Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

timrowledge
In reply to this post by Chris Muller-3

On 12-12-2012, at 6:59 PM, Chris Muller <[hidden email]> wrote:
>
>  I just think its worth asking, though,
> whether a "core" system should provide that rich a model
> out-of-the-box or instead just a bland, one-layer-above-the-primitives
> lightweight model ready to be easily wrapped by the user's _own_ rich
> model of a FileSystem.

My experience suggests that the smart approach is to build a core system that has the parts of the rich model that are needed to support the stuff you want in the core
and to make sure that the rest can be easily and non-destructively added as part of building up the system. That both allows and discourages the roll-your-own-in-a-hurry; which seems a good compromise between encouraging smart re-use of available code and allowing the truly nuts to invent something even better on the odd occasions they don't utterly crater.

There *should* be a way to provide a compatibility layer of fake-FileDirectory nastiness over the top of a nice FileSystem. It really is about time a good clean design got to have a chance; there's been way too much crapulence in FileDirectory over the decades. I should know - I wrote a fairly large part of it, several times in fact, and always because nobody wanted to make a change to a good system.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Death to all fanatics!



Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

Hannes Hirzel
Tim,

I like the points you raise.

On 12/13/12, tim Rowledge <[hidden email]> wrote:
>
> On 12-12-2012, at 6:59 PM, Chris Muller <[hidden email]> wrote:
>>
>>  I just think its worth asking, though,
>> whether a "core" system should provide that rich a model
>> out-of-the-box or instead just a bland, one-layer-above-the-primitives
>> lightweight model ready to be easily wrapped by the user's _own_ rich
>> model of a FileSystem.

You mean,

a) Colin's FileSystem should be wrapped around FileDirectory?
    (The is no reference to FileDirectory from any of the new FS
packages, This would maintain backward compatibility and at the same
time give a more abstract view.
)

Or

b) Colin's FileSystem wrapped around a low-level replacement of FileDirectory?
plus a compatibility layer of top which gives back FileDirectory


> My experience suggests that the smart approach is to build a core system
> that has the parts of the rich model that are needed to support the stuff
> you want in the core
> and to make sure that the rest can be easily and non-destructively added as
> part of building up the system.


Yes, indeed.

> That both allows and discourages the
> roll-your-own-in-a-hurry; which seems a good compromise between encouraging
> smart re-use of available code and allowing the truly nuts to invent
> something even better on the odd occasions they don't utterly crater.
>
> There *should* be a way to provide a compatibility layer of
> fake-FileDirectory nastiness over the top of a nice FileSystem.


Yes,

Marcus Denker mentions on the Pharo list that this might be an option
for Pharo to do

http://forum.world.st/To-try-out-FileSystem-in-2-0-which-download-td4649945.html#a4650038
(full citation see below)


> It really is
> about time a good clean design got to have a chance;

Yes, and the inclusion of Colin's FileSystem into Squeak is planned
for 4.5 as I
understand the minutes of the board meetings.


> there's been way too
> much crapulence in FileDirectory over the decades.

Yes.

>I should know - I wrote a
> fairly large part of it, several times in fact, and always because nobody
> wanted to make a change to a good system.
Grate to have you still around here.

In the meantime the attitude has changed and for now (or better early
2013) the addition
of FileSystem is planned.

As of now just a SqueakMap Squeak 4.4 entry for the load script of
FileSystem might do the job. Could anybody take this on please?


Colin writes that the FileSystem implementation he has been working on
for many years  (

http://wiresong.ca/filesystem/ )
is pretty close to the Pharo version which is a fork of it.

Colin and the Pharo programmers recently did a sync as Colin writes in
http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-November/166519.html

A easyt to read tutorial for FileSystem is here
 http://pharobooks.gforge.inria.fr/PharoByExampleTwo-Eng/latest/FileSystem.pdf


To load FileSystem into Squeak

(Installer wiresong project: 'mc')
  addPackage: 'FS-Core';
  addPackage: 'FS-Disk';
  addPackage: 'FS-Memory';
  addPackage: 'FS-AnsiStreams';
  addPackage: 'FS-Zip';
 "addPackage: 'FS-Xtreams';"
  addPackage: 'FS-FileStream';
  addPackage: 'FS-Tests-Core';
  addPackage: 'FS-Tests-AnsiStreams';
 " addPackage: 'FS-Tests-Xtreams'; "
  addPackage: 'FS-Tests-Zip';
  addPackage: 'FS-Tests-Disk';
  addPackage: 'FS-Tests-FileStream';
  install.

does the job
(script taken from http://lists.squeakfoundation.org/pipermail/squeak-dev/2012-

September/165822.html)

Two packages which depend on the presence of Xtreams are not loaded.
As for this thread Xtreams is a different issue.

FileSystem Testrunner results
300 tests, 297 passes, 3 errors.


--Hannes

Reply | Threaded
Open this post in threaded view
|

re: FileSystem

ccrraaiigg
In reply to this post by Chris Muller-3

Hi Chris--

> I just think its worth asking, though, whether a "core" system should
> provide that rich a model out-of-the-box or instead just a bland,
> one-layer-above-the-primitives lightweight model ready to be easily
> wrapped by the user's _own_ rich model of a FileSystem.  Is the core
> system suitable for tiny embedded programs and will they want a rich
> model or a basic one?  Maybe Spoon will allow us to have our cake and
> eat it too..?

     Yes, they're both straightforward use cases with Naiad modules. And
the Flow filesystem support is an option, too.


-C

--
Craig Latta
www.netjam.org/resume
+31   6 2757 7177
+ 1 415 287 3547 (no SMS)


Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

Hannes Hirzel
In reply to this post by Hannes Hirzel
P.S. Here the citation of Marcus' post

about "libraries as theories"

>
> http://forum.world.st/To-try-out-FileSystem-in-2-0-which-download-td4649945.html#a4650038

> (full citation see below)


<citation>
Libraries are in the end theories. They model something, e.g. Filesystem
access.

The idea here is the following: Libraries are in the end theories.
They model something, e.g. Filesystem
access. There was an old theory, and it turned out it was not really
good. So Colin made a new Theory: FileSystem.

Now there are two philosophies

1) we load the new in addition. It even is compatible to many
different versions of the System, and everyone uses it
in new code. But the system can not use it, as the developer of course
wants to load the latest version of this independently
developed library. Some people requested that for FileSystem, and we
renamed all classes so that they can continue
to load the code themselves.

2) We replace the old. Why?

 - to prove the theory. It is amazing, but in all cases, the new needs
to be either slightly adapted / improved or bug fixed
   when you *really* replace the old. It's the ultimate test.
 - to only have to maintain one code base.
 - to have a better used and therefore more bug free code base (see
e.g. the FilePlugin fixes that had to be done)

and, actually not last but very important: Have examples in the image
that show how the library is used. In the end, the system is it's own
best documentation. There should only be code in there that shows the
beginner how he is supossed to write new code.

</citation>

Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

Colin Putney-3
In reply to this post by Hannes Hirzel


On Thu, Dec 13, 2012 at 6:34 AM, H. Hirzel <[hidden email]> wrote:
Yes, and the inclusion of Colin's FileSystem into Squeak is planned
for 4.5 as I
understand the minutes of the board meetings.

Well, it was proposed, and a few board members were receptive to the idea, but that's not a final decision. Just to be clear, the Board's job is to manage the administrivia of community infrastructure, like servers and mailing lists. Setting the development agenda falls the community at large. 

I think the best way to introduce Filesystem in the base image would be to include just the core packages (FS-Core, FS-Disk, FS-Tests-Core and FS-Tests-Disk), and leave the others as optional, loadable additions. That core would be sufficient to cover all the functionality provided by FileDirectory, and parts of the system that deal with files can be gradually migrated over. 

I wrote a couple of paragraphs on why it wouldn't make sense to have an implementation FileDirectory as a layer on top of Filesystem, but then I came to the conclusion that it might not be a bad idea after all. The devil is in the details, and we ought to look into the details. Either way, the current implementation of FileDirectory should eventually be available as a loadable package, so that we can still build images with legacy code that depends on it.

Also, I wanted to point out that FS-AnsiStreams is basically obsolete. I wrote it because FileStream and friends couldn't be adapted to work with non-disk filesystems, and I wanted to have a single, uniform interface for reading and writing files, regardless of the type of filesystem that contained them. But then I discovered that Xtreams already does a way better job of that than anything I could have written. There might be some value in FS-AnsiStreams as an ANSI-compliant portability interface, but AFAICT, no dialect supports this part of the ANSI standard correctly anyway, so it wouldn't be much use.

FS-FileStream is more useful, but it's only meant as a stepping stone. It lets existing applications use Filesystem for navigating around the directory hierarchy (which should be a fairly easy port from FileDirectory) while continuing to use FileStream for IO. It provides a FileStream-like interface for IO operations on non-disk files. (Ultimately, a modern application that does a lot of IO should use Xtreams, though. It's a tremendous improvement over FileStream, SocketStream etc.)

So really, the optional bits are FS-Memory, FS-Zip, FS-Git (by Max Leske and Camillo Buni), and FS-Xtreams. 

Also, to address Chris' point about tiny images, it would be possible to split FS-Core into two packages, one with just the minimal functionality and then another with the advanced features. I don't think that's a high priority, since we don't have a core image of the size where FS-Core would make a difference. But If/when we do get there, I don't think Filesystem will be an obstacle.

Colin


Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

Frank Shearar-3
On 13 December 2012 14:52, Colin Putney <[hidden email]> wrote:

>
>
> On Thu, Dec 13, 2012 at 6:34 AM, H. Hirzel <[hidden email]> wrote:
>>
>> Yes, and the inclusion of Colin's FileSystem into Squeak is planned
>> for 4.5 as I
>> understand the minutes of the board meetings.
>
>
> Well, it was proposed, and a few board members were receptive to the idea,
> but that's not a final decision. Just to be clear, the Board's job is to
> manage the administrivia of community infrastructure, like servers and
> mailing lists. Setting the development agenda falls the community at large.
>
> I think the best way to introduce Filesystem in the base image would be to
> include just the core packages (FS-Core, FS-Disk, FS-Tests-Core and
> FS-Tests-Disk), and leave the others as optional, loadable additions. That
> core would be sufficient to cover all the functionality provided by
> FileDirectory, and parts of the system that deal with files can be gradually
> migrated over.
>
> I wrote a couple of paragraphs on why it wouldn't make sense to have an
> implementation FileDirectory as a layer on top of Filesystem, but then I
> came to the conclusion that it might not be a bad idea after all. The devil
> is in the details, and we ought to look into the details. Either way, the
> current implementation of FileDirectory should eventually be available as a
> loadable package, so that we can still build images with legacy code that
> depends on it.
>
> Also, I wanted to point out that FS-AnsiStreams is basically obsolete. I
> wrote it because FileStream and friends couldn't be adapted to work with
> non-disk filesystems, and I wanted to have a single, uniform interface for
> reading and writing files, regardless of the type of filesystem that
> contained them. But then I discovered that Xtreams already does a way better
> job of that than anything I could have written. There might be some value in
> FS-AnsiStreams as an ANSI-compliant portability interface, but AFAICT, no
> dialect supports this part of the ANSI standard correctly anyway, so it
> wouldn't be much use.

And that's pretty much most of the reason for Martin & Mark writing
Xtreams - noone implemented streams in quite the same way (and there
were things that were just wrong in the spec, but can't recall the
details).

frank

Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

timrowledge
In reply to this post by Hannes Hirzel

On 13-12-2012, at 3:34 AM, H. Hirzel <[hidden email]> wrote:

> You mean,
>
> a) Colin's FileSystem should be wrapped around FileDirectory?
>    (The is no reference to FileDirectory from any of the new FS
> packages, This would maintain backward compatibility and at the same
> time give a more abstract view.
> )
>
> Or
>
> b) Colin's FileSystem wrapped around a low-level replacement of FileDirectory?
> plus a compatibility layer of top which gives back FileDirectory
>

Not quite -
a) use FileSystem
b) write a new FileDirectory class that provides the same API but redirects everything through FileSystem. Plus, obviously, doing occasional transformations, mangling of filename strings, etc to keep the experience authentic.
c) beat up users of FileDirectory to teach them sense.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: ESR: Emulate Slide Rule



Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

timrowledge
In reply to this post by Hannes Hirzel
Minor critique of one statement in this -


On 13-12-2012, at 4:37 AM, H. Hirzel <[hidden email]> wrote:

> P.S. Here the citation of Marcus' post
> [snip]
> - to have a better used and therefore more bug free code base (see
> e.g. the FilePlugin fixes that had to be done)

I'd suggest a completely new file related plugin - or even split it up into several, possibly? - and leave the old one alone. It could be expensively confusing to have a plugin of the same name intended for a dramatically different system. Far easier to explain to some suffering user that they need to fetch a new version of FileSystemPlugin than FilePlugin but don't forget it has to be from this directory on this site and check this detail….

Whilst the subject is at the front of my mind, let's remember that there is no real need to have identical plugins and low-level image code for all platforms. There is no particular good in having a single class calling a plugin and that plugin having to do large amounts of platform specific mangling in order to conform. Consider the potential for class Blargh having proxies (or bridges, or subclasses, to taste, they can all work) BlarghForWindowsFAT32, BlarghForUnixZFS, BlarghForRISCOSADFS, BlarghForRISCOSNetFS, etc etc. You get to do complicated things in a language that is good for complicated things and pass calls down to simpler, platform/OS/filesystem specific, plugins.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: JSP: Jump on Sexy Programmer



Reply | Threaded
Open this post in threaded view
|

Re: FileSystem (was: About (backwards) Compatibility)

Andreas.Raab
In reply to this post by Colin Putney-3
Colin Putney-3 wrote
I wrote a couple of paragraphs on why it wouldn't make sense to have an
implementation FileDirectory as a layer on top of Filesystem, but then I
came to the conclusion that it might not be a bad idea after all. The devil
is in the details, and we ought to look into the details. Either way, the
current implementation of FileDirectory should eventually be available as a
loadable package, so that we can still build images with legacy code that
depends on it.
+1. Sounds much better than trying to build a compatible version.

Cheers,
  - Andreas