Smalltalk › Squeak › Squeak - Dev

[ANN] Filesystem 1.0

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

30 messages Options

Colin Putney

[ANN] Filesystem 1.0

Hi folks,

I've just made available for download a new project of mine. It's an
alternative interface to the filesystem, which grew out of my
frustration with trying to use FileDirectory for some particularly
file-intensive code. It's built upon the FilePlugin primitives, but
all the image-side code is written from scratch.

The interface offers:
- Convenient path manipulation
- Transparent access to the contents of zip files
- Late-bound references to files and directories
- Support for working with entire directory trees

A SAR file is available for download here:

http://www.wiresong.ca/downloads/Filesystem-1.0.0.sar

A short tutorial is available here:

http://www.wiresong.ca/filesystem/

Feedback appreciated!

Colin

Eliot Miranda-2

Re: [ANN] Filesystem 1.0

Colin,

you say nothing in the tutorial about Windows drive letters. I imagine 26 upper or lower case selectors could provide e.g.

FSPath C / 'plonk' / 'feep'       => C:\plonk\feep

On Fri, Nov 20, 2009 at 6:08 PM, Colin Putney <[hidden email]> wrote:

Hi folks,

I've just made available for download a new project of mine. It's an alternative interface to the filesystem, which grew out of my frustration with trying to use FileDirectory for some particularly file-intensive code. It's built upon the FilePlugin primitives, but all the image-side code is written from scratch.

The interface offers:
- Convenient path manipulation
- Transparent access to the contents of zip files
- Late-bound references to files and directories
- Support for working with entire directory trees

A SAR file is available for download here:

http://www.wiresong.ca/downloads/Filesystem-1.0.0.sar

A short tutorial is available here:

http://www.wiresong.ca/filesystem/

Feedback appreciated!

Colin

Colin Putney

Re: [ANN] Filesystem 1.0

On 20-Nov-09, at 8:23 PM, Eliot Miranda wrote:

> Colin,
>
> you say nothing in the tutorial about Windows drive letters. I
> imagine 26 upper or lower case selectors could provide e.g.
>
> FSPath C / 'plonk' / 'feep' => C:\plonk\feep

You're right, I should have a section on how things work on Windows.

The above example isn't quite right, because Windows drives are
actually separate filesystems, in contrast to Unix's single-rooted
filesystem. So the twenty six selectors should be on FSReference. It's
a great idea though, thanks!

Colin

Eliot Miranda-2

Re: [ANN] Filesystem 1.0

On Fri, Nov 20, 2009 at 8:37 PM, Colin Putney <[hidden email]> wrote:

On 20-Nov-09, at 8:23 PM, Eliot Miranda wrote:

Colin,

you say nothing in the tutorial about Windows drive letters. I imagine 26 upper or lower case selectors could provide e.g.

FSPath C / 'plonk' / 'feep' => C:\plonk\feep

You're right, I should have a section on how things work on Windows.

The above example isn't quite right, because Windows drives are actually separate filesystems, in contrast to Unix's single-rooted filesystem. So the twenty six selectors should be on FSReference. It's a great idea though, thanks!

I wonder how useful that distinction is. Remember in unix different filesystems can occur within the same namespace, e.g. mounting something on /usr or /home or /mnt etc. Even though the Windows drive letters do map to different file systems they are still a single namespace. Yes there will be operations that will fail across file systems (rename) but there will still be operations within the same filesystem that will fail (e.g. copy to a write-protected directory entry).

If FSReference can be done without and added as a whistle for special purpose use then all the better. Something minimal, effective and easy to use is the sweet spot.

BTW, one thing that always frustrated me about the VW implementation was not being able to proceed through certain exceptions. The classic example is implementing find in the presence of unreadable directories where you want to be able to proceed across the read protect rather than aborting the entire find. i.e. there's some utility on a path such as allChildrenDo: upon which is built find like facilities. At the find client level one wants to be able to wrap an exception handler around the allChildrenDo: invocation that will e.g. report errors to the transcript but proceed without aborting.

So a good set of use cases is making it easy to implement some basic unix utilities, mv, cp, find, cat etc.

I'm probably attempting to teach you to suck eggs, forgive me :)

Colin

Andreas.Raab

Re: [ANN] Filesystem 1.0

In reply to this post by Colin Putney

Colin Putney wrote:
> Feedback appreciated!

I guess the big question here is: To what extent is this a feasible
replacement of FileDirectory and/or [Standard|CrLf|MultiByte]FileStream?

Besides that, a couple of smaller comments and questions:

* How does one convert 'D:\Squeak\3.10' to the equivalent to a
FileDirectory? I.e., what's the equivalent to

fd := FileDirectory on: 'D:\Squeak\3.10'.

and then (to make the example more concrete):

(fd fileNamesMatching:'*.txt') do:[:oldName|
newName := (oldName allButLast: 4),'.bak'.
[oldFile := fd readOnlyFileNamed: oldName.
newFile := fd forceNewFileNamed: newName.
fd copyFile: oldFile toFile: newFile] ensure:[
oldFile close.
newFile close.
].
].

* What about Windows \\shares\? Many (most?) network file systems aren't
mapped to drives any longer so the access to \\shares\ is fairly important.

* Nitpick: On Windows, FSPlatformTest>>testHome will generally fail.
That's because the home location is different between Windows versions
(it's C:\Documents and Settings\ on XP, C:\Users on Vista and Windows 7)
but also because that path is localized (i.e., "Dokumente und
Einstellungen" on a German XP variant). Your best guess (short of a
dedicated primitive) is currenty

(FileDirectory on: SecurityManager default untrustedUserDirectory)
containingDirectory containingDirectory

since the untrustedUserDirectory nowadays defaults to <home>\My
Documents\My Squeak (unless overridden).

Cheers,
- Andreas

Colin Putney

Re: [ANN] Filesystem 1.0

In reply to this post by Eliot Miranda-2

On 20-Nov-09, at 9:05 PM, Eliot Miranda wrote:

> I wonder how useful that distinction is. Remember in unix different
> filesystems can occur within the same namespace, e.g. mounting
> something on /usr or /home or /mnt etc. Even though the Windows
> drive letters do map to different file systems they are still a
> single namespace. Yes there will be operations that will fail
> across file systems (rename) but there will still be operations
> within the same filesystem that will fail (e.g. copy to a write-
> protected directory entry).

I think the distinction is useful in practice. In unix, yes, you can
have separate filesystems mounted inside the global root. But that's a
pretty low-level detail. Unless he uses some pretty obscure commands,
the mount points are hidden. But on Windows, the separate filesystem
are presented to the user. For example, each drive has it's own
current directory. You can use a path like C:plonk\feep and it'll be
resolved relative to the current directory on C:. There's also the
notion of the "current" drive," which is meaningless on Unix, but
pretty basic on Windows.

> If FSReference can be done without and added as a whistle for
> special purpose use then all the better. Something minimal,
> effective and easy to use is the sweet spot.

FSReference wasn't created for dealing with multiple filesystems on
Windows; I find it quite useful on the Mac. It's sugar, sure.
Collapsing two objects into one may seem like small gain, but it
actually makes things a lot simpler... YMMV.

> BTW, one thing that always frustrated me about the VW implementation
> was not being able to proceed through certain exceptions. The
> classic example is implementing find in the presence of unreadable
> directories where you want to be able to proceed across the read
> protect rather than aborting the entire find. i.e. there's some
> utility on a path such as allChildrenDo: upon which is built find
> like facilities. At the find client level one wants to be able to
> wrap an exception handler around the allChildrenDo: invocation that
> will e.g. report errors to the transcript but proceed without
> aborting.

Ah, interesting. I'll look into that too.

Colin

Colin Putney

Re: Re: [ANN] Filesystem 1.0

In reply to this post by Andreas.Raab

On 20-Nov-09, at 9:21 PM, Andreas Raab wrote:

> Colin Putney wrote:
>> Feedback appreciated!
>
> I guess the big question here is: To what extent is this a feasible
> replacement of FileDirectory and/or [Standard|CrLf|
> MultiByte]FileStream?

Well, my intention is to create a complete replacement. I've found
FileDirectory, in particular to be a major stumbling block. What I
released today is, I think, a good replacement for FileDirectory. It's
not yet a complete replacement for FileStream. The stream
implementations are pretty basic, and don't do much beyond moving
bytes in and out of the image. That's all I needed for my immediate
needs.

I expect that to improve, but in the meantime, it's possible to use
StandardFileStream with FSDiskFilesystem subclasses, and there are
compatibility streams for FSMemoryFilesystem and FSZipFilesystem. So
it's possible to start using Filesystem to replace FileDirectory, and
keep using FileStream until the FS stream implementations mature. To
put it another way, compatibility with FileStream is a non-goal for
the design of the native FS streams, but having a smooth migration
path *is* a goal. I've got legacy code that I need to convert!

> Besides that, a couple of smaller comments and questions:
>
> * How does one convert 'D:\Squeak\3.10' to the equivalent to a
> FileDirectory? I.e., what's the equivalent to
>
> fd := FileDirectory on: 'D:\Squeak\3.10'.

ref := FSWindowsFilesystem stringToReference: 'D:\Squeak\3.10'.

> and then (to make the example more concrete):
>
> (fd fileNamesMatching:'*.txt') do:[:oldName|
> newName := (oldName allButLast: 4),'.bak'.
> [oldFile := fd readOnlyFileNamed: oldName.
> newFile := fd forceNewFileNamed: newName.
> fd copyFile: oldFile toFile: newFile] ensure:[
> oldFile close.
> newFile close.
> ].
> ].

(ref children select: [:child | child isFile and : [child basename
endsWith: '.txt']]) do:
[:oldRef |
newRef := oldRef withExtension: 'bak'.
oldRef copyTo: newRef].

Hmm, a few convenience methods would clean up that top line a bit.

> * What about Windows \\shares\? Many (most?) network file systems
> aren't mapped to drives any longer so the access to \\shares\ is
> fairly important.

Not yet implemented. Not being a Windows user, I used VMware to test
the Windows stuff, but of course I'm not on a network. I don't see any
reason it would be difficult.

> * Nitpick: On Windows, FSPlatformTest>>testHome will generally fail.
> That's because the home location is different between Windows
> versions (it's C:\Documents and Settings\ on XP, C:\Users on Vista
> and Windows 7) but also because that path is localized (i.e.,
> "Dokumente und Einstellungen" on a German XP variant). Your best
> guess (short of a dedicated primitive) is currenty
>
> (FileDirectory on: SecurityManager default untrustedUserDirectory)
> containingDirectory containingDirectory
>
> since the untrustedUserDirectory nowadays defaults to <home>\My
> Documents\My Squeak (unless overridden).

Nice catch, thank you. I actually have an implementation of
FSWindowsResolver>>home that does the equivalent of the above, but it
seems to have gone missing. I'll fix in the next release.

Colin

Andreas.Raab

Re: [ANN] Filesystem 1.0

Colin Putney wrote:
> I expect that to improve, but in the meantime, it's possible to use
> StandardFileStream with FSDiskFilesystem subclasses, and there are
> compatibility streams for FSMemoryFilesystem and FSZipFilesystem. So
> it's possible to start using Filesystem to replace FileDirectory, and
> keep using FileStream until the FS stream implementations mature. To put
> it another way, compatibility with FileStream is a non-goal for the
> design of the native FS streams, but having a smooth migration path *is*
> a goal. I've got legacy code that I need to convert!

Thanks. I thought I'd give it a quick try and port the first thing that
came into my hands which was FileList. Here is some feedback:

From a Windows perspective, the handling of drives and file systems is
quite awkward. I would strongly suggest that you drop the idea that
drives (or shares) are part of the file system and rather move them into
the path representation. Otherwise you end up with code like here:

"Figure out the #localName of the directory to display"
localName := aDirectory path basename.
"On windows, the local name may be empty and we must look at the
embedded filesystem's drive field instead"
(Smalltalk platformName = 'Win32' and:[localName isEmpty])
ifTrue:[localName := aDirectory filesystem printString].

I mean, eeek. There are several other places where one needs to do stand
on one's head to do things mixed with drives and paths on Windows which
will make it significantly harder to write cross-platform code. Just
consider for example the root directories that we'd like to display in
the file list. Here is how this looks today:

roots := FileDirectory root directoryNames.

and now with Filesystem:
Smalltalk platformName = "Win32"
ifTrue:[^FSWindowsFilesystem disks collect:[:wfs| wfs root]]
ifFalse:[^FSDiskFileSystem root entries select:[:d| d
isDirectory]].

Etc. The problem is fairly persistent and needlessly so given that
treating the drives and shares like virtual directories gives one
everything that's needed for dealing nicely with the cross-platform
aspects while being able to do the Windows-specific stuff just as well.
What I would propose here is that you keep the filesystem distinction
(so that someone is able to ask a question like: is this on the same
file system?) but move the drive letter / share name into the path.

Another issue I found bothersome was FSReference vs. FSPath. It is very
unclear to me when you use one or the other and given the difficulty to
distinguish FSPath from FSReference visually I would strongly vote for
making them one and the same to reduce confusion (see FSFilesystem
workingDirectory which returns an FSPath vs. FSFilesystem root which
returns an FSReference etc).

Some smaller issues:
* FSFileSystem root prints incorrectly 'C:/' instead of 'C:\'
* FSWindowsFilesystem>>disks should return an ordered array (C: should
come before H:)
* There is no #size on FSReadStream?

Other than that I got a basic port of FileList working. It's definitely
work to port things over, it doesn't come for free. I think there are a
few things that could help migration significantly, most importantly
implementing #localName and #fullName on FSReference and
#modificationTime (in addition to #modificationSeconds) on
FSDirectoryEntry because they are so commonly used.

Cheers,
- Andreas

Colin Putney

Re: [ANN] Filesystem 1.0

On 21-Nov-09, at 12:29 AM, Andreas Raab wrote:

> Thanks. I thought I'd give it a quick try and port the first thing
> that came into my hands which was FileList. Here is some feedback:

Awesome. Thanks for trying it out. Would you send me that code, so I
can see the areas that were problematic with a bit more context? I'd
like to make this simpler.

I rearranged your comments below, so make my responses more
intelligible.

> Another issue I found bothersome was FSReference vs. FSPath. It is
> very unclear to me when you use one or the other and given the
> difficulty to distinguish FSPath from FSReference visually I would
> strongly vote for making them one and the same to reduce confusion
> (see FSFilesystem workingDirectory which returns an FSPath vs.
> FSFilesystem root which returns an FSReference etc).

Yeah, I can see this is an issue. The idea is to use FSReference
everywhere. Filesystems and paths are meant to be the low-level
implementation mechanism for references, and you shouldn't have to
deal with them unless you're doing something exotic. Clearly the
tutorial is actually confusing more than it's enlightening; I need to
put references front and centre, and maybe mention paths and
filesystems in passing.

> From a Windows perspective, the handling of drives and file systems
> is quite awkward. I would strongly suggest that you drop the idea
> that drives (or shares) are part of the file system and rather move
> them into the path representation.

> Etc. The problem is fairly persistent and needlessly so given that
> treating the drives and shares like virtual directories gives one
> everything that's needed for dealing nicely with the cross-platform
> aspects while being able to do the Windows-specific stuff just as
> well. What I would propose here is that you keep the filesystem
> distinction (so that someone is able to ask a question like: is this
> on the same file system?) but move the drive letter / share name
> into the path.

One fo the advantages of the current arrangement is that all of the
filesystem-specific code is a subclass of FSFilesystem. If I start
having FSWindowsPath and FSUnixPath and FSZipPath, it gets harder to
make things uniform and transparent across different filesystem
implementations and harder to extend the system with new filesystems.

I realize that this insistence on supporting multiple filesystems with
a single interface does come with a cost. It makes the simple case of
files on disk a little more complex. To me, though, being able to use
the same interface for an alternate filesystem is an important feature.

Take FSMemoryFilesystem, for example. The implementation is pretty
trivial - it's a toy filesystem, not a real RAM disk. But it's good
enough for running tests against. Over the years, I've found that the
single biggest source issues with unit tests is tests that touch the
filesystem. It slows tests down, causes them to fail even when the
code is correct, and requires elaborate setUp and tearDown code to
avoid these problems as much as possible. Running the tests against an
in-image filesystem makes all that a whole lot easier.

The same is true for ZipArchives. It's valuable to be able to write
code that doesn't care if the directory structure it's setting up is
inside a ZipArchive or not. Or on a Ftp server... if FSFtpFilesystem
ever gets implemented.

I know you probably appreciate all this, but I want to make clear my
motivation for some of these design choices that may not make sense
when looking only at the simple files-on-disk case.

> Just consider for example the root directories that we'd like to
> display in the file list. Here is how this looks today:

> roots := FileDirectory root directoryNames.
>
> and now with Filesystem:
> Smalltalk platformName = "Win32"
> ifTrue:[^FSWindowsFilesystem disks collect:[:wfs| wfs root]]
> ifFalse:[^FSDiskFileSystem root entries select:[:d| d
> isDirectory]].

Yes. I now see that there needs to be a better way to get references
to the roots of all the available filesystems. (FileDirectory on: '')
provides this, but there's no way to do the same with Filesystem. It's
more obvious on Windows systems, but it's a general issue. If the idea
is to support multiple filesystems, there needs to be a way to find
out what filesystems are available.

> Otherwise you end up with code like here:
>
> "Figure out the #localName of the directory to display"
> localName := aDirectory path basename.
> "On windows, the local name may be empty and we must look at the
> embedded filesystem's drive field instead"
> (Smalltalk platformName = 'Win32' and:[localName isEmpty])
> ifTrue:[localName := aDirectory filesystem printString].
>
> I mean, eeek. There are several other places where one needs to do
> stand on one's head to do things mixed with drives and paths on
> Windows which will make it significantly harder to write cross-
> platform code.

I suspect this would be easier if you were using references uniformly.
Assuming aDirectory above is a reference to the root of a filesystem,
you could just do 'aDirectory printString'. That would get you 'C:\'
instead of 'C:', but that's not so bad, is it? (The only circumstance
I can think of for basename to be empty is for a root path. If that's
happening for another reason, it's a bug.)

> Some smaller issues:
> * FSFileSystem root prints incorrectly 'C:/' instead of 'C:\'

Fixed.

> * FSWindowsFilesystem>>disks should return an ordered array (C:
> should come before H:)

Fixed.

> * There is no #size on FSReadStream?

Apparently not. :-) I just went through the ANSI spec and implemented
everything there. There's probably a few more things that should
obviously be there, but aren't.

> Other than that I got a basic port of FileList working. It's
> definitely work to port things over, it doesn't come for free. I
> think there are a few things that could help migration
> significantly, most importantly implementing #localName and
> #fullName on FSReference and #modificationTime (in addition to
> #modificationSeconds) on FSDirectoryEntry because they are so
> commonly used.

Again, thanks for the detailed feedback. I'll make a new release soon
that fixes a bunch of this stuff, and then we can argue about the
design choices more productively.

Colin

Jason Rogers-4

Re: Re: [ANN] Filesystem 1.0

Trolling... Just wanted to say that I am enjoying this discussion and
Colin's effort to "clean up" the file handling -- having a single,
consistent interface over many different file systems is a laudable
goal.

Colin -- I'm curious what you think of Rio (http://wiki.squeak.org/squeak/5929).

On Sat, Nov 21, 2009 at 12:19 PM, Colin Putney <[hidden email]> wrote:

>
> On 21-Nov-09, at 12:29 AM, Andreas Raab wrote:
>
>> Thanks. I thought I'd give it a quick try and port the first thing that
>> came into my hands which was FileList. Here is some feedback:
>
> Awesome. Thanks for trying it out. Would you send me that code, so I can see
> the areas that were problematic with a bit more context? I'd like to make
> this simpler.
>
> I rearranged your comments below, so make my responses more intelligible.
>
>> Another issue I found bothersome was FSReference vs. FSPath. It is very
>> unclear to me when you use one or the other and given the difficulty to
>> distinguish FSPath from FSReference visually I would strongly vote for
>> making them one and the same to reduce confusion (see FSFilesystem
>> workingDirectory which returns an FSPath vs. FSFilesystem root which returns
>> an FSReference etc).
>
> Yeah, I can see this is an issue. The idea is to use FSReference everywhere.
> Filesystems and paths are meant to be the low-level implementation mechanism
> for references, and you shouldn't have to deal with them unless you're doing
> something exotic. Clearly the tutorial is actually confusing more than it's
> enlightening; I need to put references front and centre, and maybe mention
> paths and filesystems in passing.
>
>> From a Windows perspective, the handling of drives and file systems is
>> quite awkward. I would strongly suggest that you drop the idea that drives
>> (or shares) are part of the file system and rather move them into the path
>> representation.
>
>> Etc. The problem is fairly persistent and needlessly so given that
>> treating the drives and shares like virtual directories gives one everything
>> that's needed for dealing nicely with the cross-platform aspects while being
>> able to do the Windows-specific stuff just as well. What I would propose
>> here is that you keep the filesystem distinction (so that someone is able to
>> ask a question like: is this on the same file system?) but move the drive
>> letter / share name into the path.
>
>
> One fo the advantages of the current arrangement is that all of the
> filesystem-specific code is a subclass of FSFilesystem. If I start having
> FSWindowsPath and FSUnixPath and FSZipPath, it gets harder to make things
> uniform and transparent across different filesystem implementations and
> harder to extend the system with new filesystems.
>
> I realize that this insistence on supporting multiple filesystems with a
> single interface does come with a cost. It makes the simple case of files on
> disk a little more complex. To me, though, being able to use the same
> interface for an alternate filesystem is an important feature.
>
> Take FSMemoryFilesystem, for example. The implementation is pretty trivial -
> it's a toy filesystem, not a real RAM disk. But it's good enough for running
> tests against. Over the years, I've found that the single biggest source
> issues with unit tests is tests that touch the filesystem. It slows tests
> down, causes them to fail even when the code is correct, and requires
> elaborate setUp and tearDown code to avoid these problems as much as
> possible. Running the tests against an in-image filesystem makes all that a
> whole lot easier.
>
> The same is true for ZipArchives. It's valuable to be able to write code
> that doesn't care if the directory structure it's setting up is inside a
> ZipArchive or not. Or on a Ftp server... if FSFtpFilesystem ever gets
> implemented.
>
> I know you probably appreciate all this, but I want to make clear my
> motivation for some of these design choices that may not make sense when
> looking only at the simple files-on-disk case.
>
>> Just consider for example the root directories that we'd like to display
>> in the file list. Here is how this looks today:
>
>> roots := FileDirectory root directoryNames.
>>
>> and now with Filesystem:
>> Smalltalk platformName = "Win32"
>> ifTrue:[^FSWindowsFilesystem disks collect:[:wfs| wfs root]]
>> ifFalse:[^FSDiskFileSystem root entries select:[:d| d isDirectory]].
>
> Yes. I now see that there needs to be a better way to get references to the
> roots of all the available filesystems. (FileDirectory on: '') provides
> this, but there's no way to do the same with Filesystem. It's more obvious
> on Windows systems, but it's a general issue. If the idea is to support
> multiple filesystems, there needs to be a way to find out what filesystems
> are available.
>
>> Otherwise you end up with code like here:
>>
>> "Figure out the #localName of the directory to display"
>> localName := aDirectory path basename.
>> "On windows, the local name may be empty and we must look at the
>> embedded filesystem's drive field instead"
>> (Smalltalk platformName = 'Win32' and:[localName isEmpty])
>> ifTrue:[localName := aDirectory filesystem printString].
>>
>> I mean, eeek. There are several other places where one needs to do stand
>> on one's head to do things mixed with drives and paths on Windows which will
>> make it significantly harder to write cross-platform code.
>
> I suspect this would be easier if you were using references uniformly.
> Assuming aDirectory above is a reference to the root of a filesystem, you
> could just do 'aDirectory printString'. That would get you 'C:\' instead of
> 'C:', but that's not so bad, is it? (The only circumstance I can think of
> for basename to be empty is for a root path. If that's happening for another
> reason, it's a bug.)
>
>> Some smaller issues:
>> * FSFileSystem root prints incorrectly 'C:/' instead of 'C:\'
>
> Fixed.
>
>> * FSWindowsFilesystem>>disks should return an ordered array (C: should
>> come before H:)
>
> Fixed.
>
>> * There is no #size on FSReadStream?
>
> Apparently not. :-) I just went through the ANSI spec and implemented
> everything there. There's probably a few more things that should obviously
> be there, but aren't.
>
>> Other than that I got a basic port of FileList working. It's definitely
>> work to port things over, it doesn't come for free. I think there are a few
>> things that could help migration significantly, most importantly
>> implementing #localName and #fullName on FSReference and #modificationTime
>> (in addition to #modificationSeconds) on FSDirectoryEntry because they are
>> so commonly used.
>
>
> Again, thanks for the detailed feedback. I'll make a new release soon that
> fixes a bunch of this stuff, and then we can argue about the design choices
> more productively.
>
> Colin
>
>
>

--
Jason Rogers

"I am crucified with Christ: nevertheless I live;
yet not I, but Christ liveth in me: and the life
which I now live in the flesh I live by the faith of
the Son of God, who loved me, and gave
himself for me."
Galatians 2:20

Colin Putney

Re: [ANN] Filesystem 1.0

In reply to this post by Colin Putney

Stéphane Ducasse posted this to Pharo-dev, but since I'm not on that
list, I'll reply here.

On 21-Nov-09, at 2:07 AM, Stéphane Ducasse wrote:

> Hi colin
>
> good that you did that because I was starting to do the same. This
> file stuff was too frustrating.
>
> - I read the tutorial and I scanned the code (too early this
> morning) and I could not see how to rename a file.
> without copying it.

You're right. That needs to be implemented. :-)

> - why having this in the tests and not in the classes?
> createDirectory: aString
> filesystem createDirectory: (filesystem stringToPath: aString)
> createFile: aString
> filesystem createFile: (filesystem stringToPath: aString)

That's just a convenience methods for setting up that test. You'll see
that FSReference>>createDirectory does exist, and you can create a
file with #writeStreamDo:.

> - It was not clear to me how we map the old interface
> readyOnlyFileNamed and friends

If you open a read stream on a file, it's automatically opened read-
only. It's only writable if you open a write stream. For
#forceNewFileNamed:, you open a write stream, then send it #truncate.
#oldFileNamed: isn't necessary because you get an error if you open a
read stream on a file that doesn't exist.

> - ref := FSWindowsFilesystem stringToReference: 'D:\Squeak\3.10'.
> I was wondering if reference should be part of the user domain
> because why not
> ref := FSWindowsFilesystem on: 'D:\Squeak\3.10'. or
> ref := FSWindowsFilesystem folder: 'D:\Squeak\3.10'.
> I was confused by FSReference vs. FSPath

I really need to rewrite the tutorial to de-emphasize paths. You
should use FSReference for everything. Once I get Eliot's suggestion
implemented, the above will be:

ref := FSReference D / 'Squeak' / '3.10'.

> - I would really like to see how we can get rid of FileDirectory and
> use your lib and its improved version.

Slowly and carefully, I think. For now, Filesystem can be a loadable
package that other apps can depend on. That will let us work out the
kinks and flesh out the protocols through real-world use. At some
point, it might make sense to start converting old code that uses
FileDirectory, write new primitives and so on. But first I just want
to work through all the issues that people are bringing up.

Thanks,

Colin

Colin Putney

Re: Re: [ANN] Filesystem 1.0

In reply to this post by Jason Rogers-4

On 21-Nov-09, at 10:10 AM, Jason Rogers wrote:

> Colin -- I'm curious what you think of Rio (http://wiki.squeak.org/squeak/5929
> ).

I took a look at Rio a while back, and I remember being turned off by
the whole DSL-mania thing from Ruby. IMO, trying to layer another
language on top of Smalltalk syntax makes for very twisted code. It's
a bit easier in Ruby because of the implicit self, but even so, I
think simple, powerful abstractions that combine in predictable ways
is the best a way to achieve clear, understandable code.

That said, looking at the page you link to above, Filesystem and Rio
have a lot in common. Rio may have evolved since I looked at it, and I
probably stole a bunch of Keith's ideas without realizing it.

Colin

Andreas.Raab

Re: [ANN] Filesystem 1.0

In reply to this post by Colin Putney

Colin Putney wrote:
> On 21-Nov-09, at 12:29 AM, Andreas Raab wrote:
> Awesome. Thanks for trying it out. Would you send me that code, so I can
> see the areas that were problematic with a bit more context? I'd like to
> make this simpler.

Sure. Attached are the changes (relative to FileList in trunk).

>> Another issue I found bothersome was FSReference vs. FSPath. It is
>> very unclear to me when you use one or the other and given the
>> difficulty to distinguish FSPath from FSReference visually I would
>> strongly vote for making them one and the same to reduce confusion
>> (see FSFilesystem workingDirectory which returns an FSPath vs.
>> FSFilesystem root which returns an FSReference etc).
>
> Yeah, I can see this is an issue. The idea is to use FSReference
> everywhere. Filesystems and paths are meant to be the low-level
> implementation mechanism for references, and you shouldn't have to deal
> with them unless you're doing something exotic. Clearly the tutorial is
> actually confusing more than it's enlightening; I need to put references
> front and centre, and maybe mention paths and filesystems in passing.

I think you should change the terminology slightly. Having the
implementation specific thing called "path" and the user visible entity
called "reference" is confusing - when I was looking at the code I saw
"oh, there's a path, yeah, that's what I'm looking for". The term
"reference" didn't ring a bell with me. In other words, I would have
users deal with FSPath objects and rename the current FSPath to
something obviously obscure, for example "FSPathSequenceImpl" or
something to make obviously clear that from the user perspective you're
going to deal with the FSPath only.

FWIW, I didn't find the usage of file system as confusing; the only
confusion arose from having to deal with them due to the windows drive
letters (but see below).

>> From a Windows perspective, the handling of drives and file systems is
>> quite awkward. I would strongly suggest that you drop the idea that
>> drives (or shares) are part of the file system and rather move them
>> into the path representation.
>
>> Etc. The problem is fairly persistent and needlessly so given that
>> treating the drives and shares like virtual directories gives one
>> everything that's needed for dealing nicely with the cross-platform
>> aspects while being able to do the Windows-specific stuff just as
>> well. What I would propose here is that you keep the filesystem
>> distinction (so that someone is able to ask a question like: is this
>> on the same file system?) but move the drive letter / share name into
>> the path.
>
> One fo the advantages of the current arrangement is that all of the
> filesystem-specific code is a subclass of FSFilesystem. If I start
> having FSWindowsPath and FSUnixPath and FSZipPath, it gets harder to
> make things uniform and transparent across different filesystem
> implementations and harder to extend the system with new filesystems.

Absolutely. But I don't think you'd have to change the arrangement at
all. The idea is that on Windows there is a virtual root of the file
system and all shares and drive are logically subdirectories of that
virtual root. This works fine and really the only difference is printing.

The easiest way to experiment with that is by just removing all of
FSWindowsFilsystem stringToPath:, stringToReference:,
forReferencePrintOn: and fix up pathToString: to not print the disk. At
this point you can already do stuff like:

FSDiskFilesystem current stringToReference: 'D:\Squeak\3.10'.

Which shows the idea. After which things break mostly because there are
a few tricky Unix assumptions in the code, for example
FSPath>>isAbsolute. The assumption that you can tell in a context-free
manner whether a path is absolute or not will cause grief if you need to
be able to distinguish /foo/bar from \\share\dir from c:\windows etc.
Even more tricky, \Windows\system32 is not an absolute path in any
meaningful way on Windows. I would vote for being explicit here and have
FSRelativePath that the file system can construct appropriately.

[... two hours later ...]

Ah, what the heck. I just spend two hours fixing things into shape the
way I think it should work. Attached you'll find my patches which
implement the above. Check it out and you'll see why I'm saying there is
no real difference in arrangement. Let me know what you think about that
direction. The patches allow one to use '\\server\path\to\file' as well
as 'c:\windows\system' etc. The tests pass, too :-)

Cheers,
- Andreas

FSFileList.3.cs.gz (3K) Download Attachment

FSWinPatches.3.cs.gz (5K) Download Attachment

Andreas.Raab

Re: [ANN] Filesystem 1.0

In reply to this post by Colin Putney

Colin Putney wrote:
> I took a look at Rio a while back, and I remember being turned off by
> the whole DSL-mania thing from Ruby. IMO, trying to layer another
> language on top of Smalltalk syntax makes for very twisted code. It's a
> bit easier in Ruby because of the implicit self, but even so, I think
> simple, powerful abstractions that combine in predictable ways is the
> best a way to achieve clear, understandable code.

+1. I like what I see in Filesystem. It's good Smalltalk code, with
recognizable patterns, simple usage and clear structure.

Cheers,
- Andreas

Colin Putney

Re: Re: [ANN] Filesystem 1.0

In reply to this post by Andreas.Raab

On 21-Nov-09, at 3:22 PM, Andreas Raab wrote:

> I think you should change the terminology slightly. Having the
> implementation specific thing called "path" and the user visible
> entity called "reference" is confusing - when I was looking at the
> code I saw "oh, there's a path, yeah, that's what I'm looking for".
> The term "reference" didn't ring a bell with me. In other words, I
> would have users deal with FSPath objects and rename the current
> FSPath to something obviously obscure, for example
> "FSPathSequenceImpl" or something to make obviously clear that from
> the user perspective you're going to deal with the FSPath only.

Yes. Terminology is both important and difficult. I'm not very happy
with the term "reference" either, but I can't think of anything
better. The thing I called "path" really is a path, where as
"references" are something more than a path. That said, it may be
wiser to bow to the intuition of users, even though their intuition is
wrong. :-)

>> One fo the advantages of the current arrangement is that all of the
>> filesystem-specific code is a subclass of FSFilesystem. If I start
>> having FSWindowsPath and FSUnixPath and FSZipPath, it gets harder
>> to make things uniform and transparent across different filesystem
>> implementations and harder to extend the system with new filesystems.
>
> Absolutely. But I don't think you'd have to change the arrangement
> at all. The idea is that on Windows there is a virtual root of the
> file system and all shares and drive are logically subdirectories of
> that virtual root. This works fine and really the only difference is
> printing.

I'm not sure I understand the problem with multiple filesystems on
Windows. Sure, you stumbled over some exposed girders trying to
enumerate the available drives, but that doesn't mean the building
isn't structurally sound. Given that there are going to be multiple
filesystem floating around even on unix, it seems to me that the
Windows paradigm of multiple roots is a better fit than the unix
policy of maintaining the illusion of a single root.

I really like your idea of a virtual root directory, but I'm thinking
that instead of having that built into a single FSWindowsFilesystem,
it would be better to reify it and give it the ability to embed other
filesystems within it. That way Windows drive letters can live
alongside in-image filesystems, remote filesystems etc. We already do
that sort of thing when we display superswikis in the FileList. This
solves your problem of how to enumerate the local drives without
Windows-specific code, and allows Unix systems to play as well.

> The assumption that you can tell in a context-free manner whether a
> path is absolute or not will cause grief if you need to be able to
> distinguish /foo/bar from \\share\dir from c:\windows etc. Even more
> tricky, \Windows\system32 is not an absolute path in any meaningful
> way on Windows.

Yeah, this is what convinced me that it's better not to try to paper
over the fact that Windows explicitly has multiple filesystems.

BTW, I disagree with your assertion above. \Windows\system32 *is* an
absolute path; it's just interpreted in the context of the current
drive. Maybe you don't think about the current drive much when using
the Windows Explorer, but that doesn't mean it isn't there.

Here's an experiment that proves it:

1. I opened an Explorer window on my squeak image. In the address bar
it says

C:\Documents and Settings\colin\My Documents\Mason

2. I highlight that and enter 'package-cache'

3. Now the windows shows my MC repository and the address bar says:

C:\Documents and Settings\colin\My Documents\Mason\package-cache

4. That was a relative path! I highlight that and enter '\windows
\system32'

5. Now the window shows a bunch of system files and the address bar
says:

C:\WINDOWS\system32

6. That was an absolute path!

The console behaves the same way when you use cd. The flip side of the
coin is that C:WINDOWS\system32 is a relative path, despite beginning
with a drive letter. If you make D: the current drive in the console,
then do "cd C:WINDOWS\system32" the current directory on drive C: gets
changed but the current drive remains D:. The new current directory on
C: is determined by resolving "WINDOWS\system32" against the old
current directory.

The current implementation gets all this behavior more-or-less for
free, by acknowledging that each drive is really a separate filesystem.

> Ah, what the heck. I just spend two hours fixing things into shape
> the way I think it should work. Attached you'll find my patches
> which implement the above. Check it out and you'll see why I'm
> saying there is no real difference in arrangement. Let me know what
> you think about that direction. The patches allow one to use '\
> \server\path\to\file' as well as 'c:\windows\system' etc. The tests
> pass, too :-)

Thanks. It seems reasonable from quick perusal. I'll respond when I
understand it better.

Colin

James Hayes

Re: [ANN] Filesystem 1.0

In reply to this post by Colin Putney

Hi Colin,

Please take a look at:

http://commons.apache.org/vfs/

It is Java but is brilliant.

Cheers,

James

Colin Putney wrote:

> Hi folks,
>
> I've just made available for download a new project of mine. It's an
> alternative interface to the filesystem, which grew out of my
> frustration with trying to use FileDirectory for some particularly
> file-intensive code. It's built upon the FilePlugin primitives, but all
> the image-side code is written from scratch.
>
> The interface offers:
> - Convenient path manipulation
> - Transparent access to the contents of zip files
> - Late-bound references to files and directories
> - Support for working with entire directory trees
>
> A SAR file is available for download here:
>
> http://www.wiresong.ca/downloads/Filesystem-1.0.0.sar
>
> A short tutorial is available here:
>
> http://www.wiresong.ca/filesystem/
>
> Feedback appreciated!
>
> Colin
>
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 8.5.425 / Virus Database: 270.14.76/2519 - Release Date: 11/22/09 07:38:00
>

No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.5.425 / Virus Database: 270.14.76/2519 - Release Date: 11/22/09 07:38:00

Göran Krampe

Re: Re: [ANN] Filesystem 1.0

In reply to this post by Colin Putney

Hi!

Colin Putney wrote:
> On 21-Nov-09, at 10:10 AM, Jason Rogers wrote:
>
>> Colin -- I'm curious what you think of Rio
>> (http://wiki.squeak.org/squeak/5929).
>
> I took a look at Rio a while back, and I remember being turned off by
> the whole DSL-mania thing from Ruby.

Amen. :)

I have NOT looked closely at Rio, and I know Keith can do good stuff,
but I have seen Rio snippets and I am sorry but I also do NOT like this
coding style.

Also, thank you, thank you Colin for taking on this BRUTE of a task! :)
Let's all support Colin in pushing this all the way through. And of
course, it would be really nice if the Pharo community could step up
behind this initiative too. Being compatible at this level would be nice.

regards, Göran

Colin Putney

Re: [ANN] Filesystem 1.0

In reply to this post by James Hayes

On 22-Nov-09, at 8:30 AM, James Hayes wrote:

> Hi Colin,
>
> Please take a look at:
>
> http://commons.apache.org/vfs/
>
> It is Java but is brilliant.

Hey, thanks for this link. It *is* quite interesting; they're
definitely trying accomplish the same sort of thing that I am. I had
hoped to steal some of their terminology, but... Reference maybe sort
of meaningless but it's better than FileObject. :-)

One thing that is interesting is that they use URIs, with made-up
schema that map to their filesystem providers. So a Reference to a
file in a MemoryFilesystem would have a url like

ram:/path/to/my/file.

And files inside zip archives look like:

file:///path/to/archive.zip!/path/to/file

Java folks expect to us URIs everywhere. I wonder how well that would
work in Smalltalk.

Colin

johnmci

Re: Re: [ANN] Filesystem 1.0

On 2009-11-23, at 7:26 AM, Colin Putney wrote:

> One thing that is interesting is that they use URIs, with made-up schema that map to their filesystem providers. So a Reference to a file in a MemoryFilesystem would have a url like
>
> ram:/path/to/my/file.
>
> And files inside zip archives look like:
>
> file:///path/to/archive.zip!/path/to/file
>
> Java folks expect to us URIs everywhere. I wonder how well that would work in Smalltalk.

I'll play broken record, and say yes we did this in Sophie. All resource references to file like things was via URI.
We also did the "made-up schema" so that we could refer to cached memory files versus http: or file:

--
===========================================================================
John M. McIntosh <[hidden email]> Twitter: squeaker68882
Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com
===========================================================================

Colin Putney

Re: Re: [ANN] Filesystem 1.0

On 23-Nov-09, at 8:42 AM, John M McIntosh wrote:

>> Java folks expect to us URIs everywhere. I wonder how well that
>> would work in Smalltalk.
>
> I'll play broken record, and say yes we did this in Sophie. All
> resource references to file like things was via URI.
> We also did the "made-up schema" so that we could refer to cached
> memory files versus http: or file:

Ok, but that doesn't answer my question. How well did it work? What
was good about it? What was bad about it? Was it a natural fit in
Smalltalk, or did it take some shoehorning to make it work?

Colin