FileSystem
Locked
12
12
 
|
So Metacello is being forced, along with many other packages, to
re-implement a compatibility layer because Squeak uses FileDirectory and Pharo uses FileSystem. Now I completely understand that Pharo's understanding of FileDirectory is about 5 years out of date. By now, countless arguments on pharo-dev have shown that the two packages are roughly isomorphic in functionality and API. Camillo Bruni's even shown this by implementing a shim exposing a FIleDirectory API on top of FileSystem to ease migration of early Pharo projects to later versions of Pharo. So I have a pair of questions: * how far have wiresong's FileSystem and Pharo's FileSystem diverged? (And how do we bring the two back together again, if necessary?) * How do we fix the problem? Pharo has nearly emptied the open source Smalltalk room of oxygen, so there's an argument for saying that FileSystem has won and we should use it. If we did move to FileSystem we _would_ want to keep Cami's shim, because we care about backwards compatibility a whole lot more than Pharo. And of course there's Cuis to consider. Thoughts? (Note: please _don't_ talk about the spilt more than absolutely necessary. It's done, it can't be undone, and we should concentrate our energies on moving Squeak forward, and minimising pain for those brave souls who try keep their packages cross platform.) frank |
|
|
Quoting Frank Shearar <[hidden email]>:
> So Metacello is being forced, along with many other packages, to > re-implement a compatibility layer because Squeak uses FileDirectory > and Pharo uses FileSystem. > > Now I completely understand that Pharo's understanding of > FileDirectory is about 5 years out of date. By now, countless > arguments on pharo-dev have shown that the two packages are roughly > isomorphic in functionality and API. > > ... > > And of course there's Cuis to consider. > > Thoughts? > ... > frank Thanks for remembering about Cuis. In Cuis, we haven't payed much attention to FileSystem functionality yet. Most likely our code is rather outdated here. If Squeak decides to switch to FileSystem, I think Cuis can do the same. In any case, we need to review alternatives and pick one for Cuis, but we're not in a hurry. Cheers, Juan Vuletich |
|
|
In reply to this post by Frank Shearar-3
On 24-05-2013, at 1:58 AM, Frank Shearar <[hidden email]> wrote: > > > > So I have a pair of questions: > * how far have wiresong's FileSystem and Pharo's FileSystem diverged? > (And how do we bring the two back together again, if necessary?) > * How do we fix the problem? Without even knowing anything specifically about FileSystem I would claim a very high likelihood that it is better than the old code, because:- Colin wrote at least a good chunk of it it was written recently with experience of just how awful FileDirectory et al have become over years of largely unprincipled hacking And thus I would strongly suggest it in with a chance of being an improvement. My only caveat would be that I feel reasonably sure it was not written with much of an eye to 'odd' filing systems like RISC OS and thus it may need a bit more work to drag it away from the depressingly mono-cultural ersatz-unix-style that seems to have pervaded the world. I wouldn't mind if were actually a good style... tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim Klingon Code Warrior:- 4) "This machine is a piece of GAGH! I need dual ARMv8 processors if I am to do battle with this code!" |
|
|
> tim Rowledge
a
> > > On 24-05-2013, at 1:58 AM, Frank Shearar <[hidden email]> wrote: > > > > > > > > So I have a pair of questions: > > * how far have wiresong's FileSystem and Pharo's FileSystem diverged? > > (And how do we bring the two back together again, if necessary?) > > * How do we fix the problem? > > > Without even knowing anything specifically about FileSystem I would claim > very high likelihood that it is better than the old code, because:- Colin wrote at > least a good chunk of it it was written recently with experience of just how > awful FileDirectory et al have become over years of largely unprincipled hacking > > And thus I would strongly suggest it in with a chance of being an improvement. > My only caveat would be that I feel reasonably sure it was not written with > much of an eye to 'odd' filing systems like RISC OS and thus it may need a bit > more work to drag it away from the depressingly mono-cultural ersatz-unix-style > that seems to have pervaded the world. I wouldn't mind if were actually a good > style... Also when / if the change is done I recommend that someone that goes through the conversion on production code produce a conversion guide to help people that need to update their systems. Ron > > tim > -- > tim Rowledge; [hidden email]; http://www.rowledge.org/tim Klingon Code > Warrior:- 4) "This machine is a piece of GAGH! I need dual ARMv8 processors if I > am to do battle with this code!" > > > |
|
|
In reply to this post by Frank Shearar-3
Hi Frank,
Can you tell me which one has better SUnit coverage? (I'm on a phone.) If FileSystem doesn't have just some enormous complexity, if the code is at least as good (however we're going to be measuring that) and there are good tests, I can't think of any reason we shouldn't do what the hipsters are doing, except of course the work involved. <ironic type="hipster">mustache</ironic> On May 24, 2013, at 1:58 AM, Frank Shearar <[hidden email]> wrote: > So Metacello is being forced, along with many other packages, to > re-implement a compatibility layer because Squeak uses FileDirectory > and Pharo uses FileSystem. > > Now I completely understand that Pharo's understanding of > FileDirectory is about 5 years out of date. By now, countless > arguments on pharo-dev have shown that the two packages are roughly > isomorphic in functionality and API. > > Camillo Bruni's even shown this by implementing a shim exposing a > FIleDirectory API on top of FileSystem to ease migration of early > Pharo projects to later versions of Pharo. > > So I have a pair of questions: > * how far have wiresong's FileSystem and Pharo's FileSystem diverged? > (And how do we bring the two back together again, if necessary?) > * How do we fix the problem? > > Pharo has nearly emptied the open source Smalltalk room of oxygen, so > there's an argument for saying that FileSystem has won and we should > use it. If we did move to FileSystem we _would_ want to keep Cami's > shim, because we care about backwards compatibility a whole lot more > than Pharo. And of course there's Cuis to consider. > > Thoughts? > > (Note: please _don't_ talk about the spilt more than absolutely > necessary. It's done, it can't be undone, and we should concentrate > our energies on moving Squeak forward, and minimising pain for those > brave souls who try keep their packages cross platform.) > > frank > |
|
Administrator
|
In reply to this post by Frank Shearar-3
Wow, I'm impressed by the thoughtfulness of the request, as well as the responses - all business :) I've seen similar attitudes on the Pharo side regarding adopting improvements in Squeak. In fact, I've felt the relationship between Squeak and Pharo warming for a while and this confirms that yet again. Thanks to all for being so cool! - Sean p.s. I still dream about a day when Squeak and Pharo cooperate more directly, possibly even [gasp] unifying (e.g. Pharo core + Squeak/etoys on top), but I try not to talk about it as not to scare people ;)
Cheers,
Sean |
|
|
In reply to this post by Frank Shearar-3
On Fri, May 24, 2013 at 1:58 AM, Frank Shearar <[hidden email]> wrote: So I have a pair of questions: I doubt there's much divergence. I synced with the Pharo folks as they were putting it in to their base image, and that was a pretty mature implementation, so I'd be surprised if there was anything difficult to deal with.
* How do we fix the problem? Well, I'd be happy to see Filesystem in Squeak. :-) I know Chris is worried about breaking Magma, so we'll need to be careful, but we can do that.
Colin |
|
|
In reply to this post by timrowledge
On Fri, May 24, 2013 at 9:49 AM, tim Rowledge <[hidden email]> wrote: ould strongly suggest it in with a chance of being an improvement. My only caveat would be that I feel reasonably sure it was not written with much of an eye to 'odd' filing systems like RISC OS and thus it may need a bit more work to drag it away from the depressingly mono-cultural ersatz-unix-style that seems to have pervaded the world. I wouldn't mind if were actually a good style... Yeah, Andreas smacked me over forcing unix style on Windows users. Fortunately, he smacked me with code, so there's a bit of agnosticism in there already. Is it just a matter of generating correct path strings to pass to the primitives, or is there more to it? Colin |
|
|
In reply to this post by Colin Putney-3
> Well, I'd be happy to see Filesystem in Squeak. :-)
> > I know Chris is worried about breaking Magma, so we'll need to be careful, > but we can do that. That's appreciated, thanks. You're right, and also about Banyan. And not just breakage but also performance. I'm sure I won't be left behind entirely but I also hope it won't be too painful to switch over. Banyan makes use of that method in FileDirectory I committed to trunk yesterday (directoryTreeDo:). Important to Banyan would be for FileSystem to be able to do that without creating much (if any) garbage. Is that doable? |
|
|
On Sun, May 26, 2013 at 7:28 PM, Chris Muller <[hidden email]> wrote: Banyan makes use of that method in FileDirectory I committed to trunk Sure. Filesystem does have methods for walking a directory tree, although they're not optimized for memory usage. |
|
|
In reply to this post by Colin Putney-3
On 27 May 2013 03:04, Colin Putney <[hidden email]> wrote:
> > > > On Fri, May 24, 2013 at 1:58 AM, Frank Shearar <[hidden email]> > wrote: > >> >> So I have a pair of questions: >> * how far have wiresong's FileSystem and Pharo's FileSystem diverged? >> (And how do we bring the two back together again, if necessary?) > > > I doubt there's much divergence. I synced with the Pharo folks as they were > putting it in to their base image, and that was a pretty mature > implementation, so I'd be surprised if there was anything difficult to deal > with. > >> >> * How do we fix the problem? > > > Well, I'd be happy to see Filesystem in Squeak. :-) > > I know Chris is worried about breaking Magma, so we'll need to be careful, > but we can do that. OK, so the first step is to add a CI build for FileSystem. I'm away from my image at the moment, but doesn't Installer have a #wiresong? If you supply me with an Installer script for FileSystem I'll wire up Jenkins. One thing I've put off doing is getting Magma under CI. It has special ways of setting up tests that don't quite fit our existing CI test infrastructure. But once that's done, Magma would be an excellent guinea pig. (Of course, one could run the Magma test suite manually. I like the CI route because it keeps me honest: you can't fool a CI with accidental local state changes.) frank > Colin |
|
|
In reply to this post by Sean P. DeNigris
On 5/26/13 12:13 PM, "Sean P. DeNigris" <[hidden email]> wrote: > p.s. I still dream about a day when Squeak and Pharo cooperate more > directly, possibly even [gasp] unifying (e.g. Pharo core + Squeak/etoys on > top), but I try not to talk about it as not to scare people ;) Yes, this could be good. As Pharo 2.0 and 3.0 diverge too much... What about we took Pharo 1.4 Kernel and build on top? Edgar |
|
|
In reply to this post by Colin Putney-3
On Sun, May 26, 2013 at 07:14:19PM -0700, Colin Putney wrote:
> On Fri, May 24, 2013 at 9:49 AM, tim Rowledge <[hidden email]> wrote: > > > > ould strongly suggest it in with a chance of being an improvement. My only > > caveat would be that I feel reasonably sure it was not written with much of > > an eye to 'odd' filing systems like RISC OS and thus it may need a bit more > > work to drag it away from the depressingly mono-cultural ersatz-unix-style > > that seems to have pervaded the world. I wouldn't mind if were actually a > > good style... > > > > Yeah, Andreas smacked me over forcing unix style on Windows users. > Fortunately, he smacked me with code, so there's a bit of agnosticism in > there already. > > Is it just a matter of generating correct path strings to pass to the > primitives, or is there more to it? > There are other things as well. I'm not sure if these apply to FileSystem (sorry I can't look right now) but examples might be: - Unix views the file system as a tree, regardless of how many file systems and physical devices are involved. Windows models this differently as volumes (C:, D: etc). Other operating systems may have different models. - The notions of file creation time, last accessed time, last modified time and so forth are done differently (and incompatibly) on different kinds of file system. - One operating system may simulaneously host multiple types of file system, so some of the differences are associated with the file system and not with the operating system per se. I'm not sure what might be required to have FileSystem work smoothly with RISC OS as well as Windows/Unix, but I think the effort would be very worthwhile. Windows and Unix file systems are already very similar, so the best way to ensure a good level of abstraction is to make sure that the model also fits file systems such as RISC OS that are not fundamentally unix-based. So far I have looked at FileSystem only to the extent needed to get OSProcess/CommandShell working on Pharo, so I don't know if the above are really relevant. In any case, I'll be happy to help where I can. Dave |
|
|
On Mon, May 27, 2013 at 6:03 AM, David T. Lewis <[hidden email]> wrote: - Unix views the file system as a tree, regardless of how many file systems The first version of FS that I did tried to model that pretty closely. FS its self supports multiple volumes, and so I created one for each Windows drive, each with it's own CWD, just as Windows shells do. People who run Squeak on Windows (particularly Andreas) didn't like this set up, and so we switched it over to a more Unix-like model where there's a single root, single CWD and special handling for the drive letters in absolute paths. I thought it was cool that we could match the Windows filesystem semantics correctly, but <shrug> I'm not a Windows user, what do I know.
- The notions of file creation time, last accessed time, last modified The FilePlugin primitives handle the platform differences for us. I haven't run into any issues with this yet. - One operating system may simulaneously host multiple types of file Filesystem can handle multiple "file systems" at once, so at the image level this isn't an issue (and FS already includes support for file systems other than the one supplied by the operating system—in memory, inside a zip file etc). If the FilePlugin primitives and/or the OS file API don't unify the interface to different filesystems for us, we could use a different set of primitives for each filesystem. I did have an idea for a new set of primitives to go with the new image-level code, but I figured it'd be better to wait and see what issues, if any, surface from real-world usage.
I'm not sure what might be required to have FileSystem work smoothly I think the differences in the path syntax should be easy to deal with, but we'll probably need some way to manipulate the file metadata. (eg. to set the filetype correctly).
So far I have looked at FileSystem only to the extent needed to get Cool! Colin |
|
|
In reply to this post by Colin Putney-3
>> Banyan makes use of that method in FileDirectory I committed to trunk
>> yesterday (directoryTreeDo:). Important to Banyan would be for >> FileSystem to be able to do that without creating much (if any) >> garbage. Is that doable? > > > Sure. Filesystem does have methods for walking a directory tree, although > they're not optimized for memory usage. That's what I thought I remember from briefly looking at it before, that the way it did that created lots of garbage. I hope I'm wrong. IMO, FileSystem should be at least equal to FileDirectory across the board, so we leave no reasons at all for someone to want to have FileDirectory in their image (which means they'd have to have both). |
|
|
On Mon, May 27, 2013 at 12:15 PM, Chris Muller <[hidden email]> wrote:
I don't think you can expect a general API that's designed for flexibility and usefulness across a broad range of applications to perform as well as an optimized implementation that you hand-tuned for your specific application. In fact, I'd argue that #directoryTreeDo: shouldn't be part of the trunk, it should be an extension method in Banyan.
Look at it this way: FileDirectory was missing a lot of features that you needed. Instead of using Filesystem, which provides those features, you added them to FileDirectory in a way that's highly specific to the needs of your application. Now you're worried that Filesystem isn't "equal" to FileDirectory, meaning that it's not tuned for your application. But if you put the same effort into optimizing Banyan+Filesystem, you'd be fine.
If you don't want to put in that effort, that's fine too. FileDirectory will be a loadable package, so you can just make Banyan depend on it, and add it to your build script. Colin |
|
|
>> That's what I thought I remember from briefly looking at it before,
>> that the way it did that created lots of garbage. I hope I'm wrong. >> IMO, FileSystem should be at least equal to FileDirectory across the >> board, so we leave no reasons at all for someone to want to have >> FileDirectory in their image (which means they'd have to have both). > > I don't think you can expect a general API that's designed for flexibility > and usefulness across a broad range of applications to perform as well as an > optimized implementation that you hand-tuned for your specific > application. In fact, I'd argue that #directoryTreeDo: shouldn't be part of > the trunk, it should be an extension method in Banyan. Colin, may we please not be at odds here? I'm on your side. Before you said "Sure, FileSystem does that too" but above you're saying this method should not be in trunk? But it should be in trunk as part of FileSystem? I'm confused. > Look at it this way: FileDirectory was missing a lot of features that you > needed. Instead of using Filesystem, which provides those features, you > added them to FileDirectory in a way that's highly specific to the needs of > your application. "A lot of features?" What are you talking about? I've presented just one short public method (with one supporting private) from 2007, long before FileSystem, which performs a /very general/ operation, an efficient internal Iterator of a directory tree. > Now you're worried that Filesystem isn't "equal" to > FileDirectory, meaning that it's not tuned for your application. But if you > put the same effort into optimizing Banyan+Filesystem, you'd be fine. > If you don't want to put in that effort, that's fine too. FileDirectory will > be a loadable package, so you can just make Banyan depend on it, and add it > to your build script. I don't understand this defensiveness. I'm trying to open up to FileSystem by asking you if this one important function can be made to perform well as in FD. I figured you'd look at it and, based on your FS knowledge, say "yes, an equivalent method could be spanked out in under 5 minutes" (even if the "efficient" version had to be an extension), so we could be off to the races. Is it not the case? I was hoping to be encouraged by your reply but surprised instead to hear back that I should lower my expectations about performance and appreciate the flexibility..? How can someone be expected to allocate time to convert legacy code if it's known certain functions will end up compromised on performance? |
|
|
On Mon, May 27, 2013 at 2:45 PM, Chris Muller <[hidden email]> wrote:
I'm sorry, I didn't mean for that post to be so confrontational. I misread your post as shifting the goal-posts so that FileSystem wouldn't be acceptable unless it performs like your tuned-for-Banyan additions to FileDirectory.
There's certainly room to optimize the tree-walking code in Filesystem, so we may be able to meet your needs that way. On the other hand, there are layers of indirection in Filesystem that aren't present in FileDirectory. Filesystem works with many kinds of directory trees—on disk, in-memory, inside a zip file, inside a git repository etc. It also has whole-tree operations that need to be able to customize the tree walking algorithm. For example, copying a tree needs to visit directories before their contents, while deleting a tree needs to visit directories after their contents. So from the point of view of Filesystem, #directoryTreeDo: is *not* a very general operation, it's quite specific, and tuned for a particular use-case. It may not be possible to optimize Filesystem's tree-walking code to the same level of memory efficiency without sacrificing generality.
But that's not a show-stopper! We could make a method like #directoryTreeDo: just for Banyan, or Banyan could keep on using FileDirectory. It's not like FileDirectory would be removed in 4.5, and even when it finally does get removed, it would still be available as a compatibility package.
Again, my apologies for the over-reaction. Colin |
|
|
Could you point me to what's the best way to install latest FS into
trunk? Thanks! On Mon, May 27, 2013 at 7:27 PM, Colin Putney <[hidden email]> wrote: > > > > On Mon, May 27, 2013 at 2:45 PM, Chris Muller <[hidden email]> wrote: > >> >> Colin, may we please not be at odds here? I'm on your side. > > > I'm sorry, I didn't mean for that post to be so confrontational. I misread > your post as shifting the goal-posts so that FileSystem wouldn't be > acceptable unless it performs like your tuned-for-Banyan additions to > FileDirectory. > > There's certainly room to optimize the tree-walking code in Filesystem, so > we may be able to meet your needs that way. > > On the other hand, there are layers of indirection in Filesystem that aren't > present in FileDirectory. Filesystem works with many kinds of directory > trees—on disk, in-memory, inside a zip file, inside a git repository etc. It > also has whole-tree operations that need to be able to customize the tree > walking algorithm. For example, copying a tree needs to visit directories > before their contents, while deleting a tree needs to visit directories > after their contents. So from the point of view of Filesystem, > #directoryTreeDo: is *not* a very general operation, it's quite specific, > and tuned for a particular use-case. It may not be possible to optimize > Filesystem's tree-walking code to the same level of memory efficiency > without sacrificing generality. > > But that's not a show-stopper! We could make a method like #directoryTreeDo: > just for Banyan, or Banyan could keep on using FileDirectory. It's not like > FileDirectory would be removed in 4.5, and even when it finally does get > removed, it would still be available as a compatibility package. > > Again, my apologies for the over-reaction. > > Colin > > > |
|
|
In reply to this post by Casey Ransberger-2
Hi Casey,
It looks like FileDirectory has 26% methods covered, which honestly is higher than I'd expected. It looks like FileSystem has 80% code coverage. "Looks like" because FSFilesystemTest class >> #packageNamesUnderTest returns #('Filesystem') which seems to make TestRunner think there's nothing to cover. Hacking that to return #('FS') and adding an 'FS' package to the image (so that one package "captures" all the FS-* packages) seems to fix the problem. frank On 25 May 2013 23:51, Casey Ransberger <[hidden email]> wrote: > Hi Frank, > > Can you tell me which one has better SUnit coverage? (I'm on a phone.) > > If FileSystem doesn't have just some enormous complexity, if the code is at least as good (however we're going to be measuring that) and there are good tests, I can't think of any reason we shouldn't do what the hipsters are doing, except of course the work involved. > > <ironic type="hipster">mustache</ironic> > > On May 24, 2013, at 1:58 AM, Frank Shearar <[hidden email]> wrote: > >> So Metacello is being forced, along with many other packages, to >> re-implement a compatibility layer because Squeak uses FileDirectory >> and Pharo uses FileSystem. >> >> Now I completely understand that Pharo's understanding of >> FileDirectory is about 5 years out of date. By now, countless >> arguments on pharo-dev have shown that the two packages are roughly >> isomorphic in functionality and API. >> >> Camillo Bruni's even shown this by implementing a shim exposing a >> FIleDirectory API on top of FileSystem to ease migration of early >> Pharo projects to later versions of Pharo. >> >> So I have a pair of questions: >> * how far have wiresong's FileSystem and Pharo's FileSystem diverged? >> (And how do we bring the two back together again, if necessary?) >> * How do we fix the problem? >> >> Pharo has nearly emptied the open source Smalltalk room of oxygen, so >> there's an argument for saying that FileSystem has won and we should >> use it. If we did move to FileSystem we _would_ want to keep Cami's >> shim, because we care about backwards compatibility a whole lot more >> than Pharo. And of course there's Cuis to consider. >> >> Thoughts? >> >> (Note: please _don't_ talk about the spilt more than absolutely >> necessary. It's done, it can't be undone, and we should concentrate >> our energies on moving Squeak forward, and minimising pain for those >> brave souls who try keep their packages cross platform.) >> >> frank >> > |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |