FileDirectory fails

> On 14.06.2020, at 12:19, Marcel Taeumel <[hidden email]> wrote:
>
> > It actually forces them when necessary…
>
> You mean "also", not "actually". :-)

? no. I meant "actually". but "also" is correct, too…

> On 14.06.2020, at 12:33, Marcel Taeumel <[hidden email]> wrote:
>
> > ? no. I meant "actually". but "also" is correct, too…
>
> Ah, then it's a language thing. For me, it read like "only forced" because of that "actually". My bad. Sorry for the noise. ^__^

All fine :D

On 14/06/20 1:19 pm, K K Subbu wrote:
>
> On the other hand, the path 'SourcesV50.sources' is incomplete (dirname
> missing) and should not be passed to plugin. The image has to resolve
> this file in imagePath, vmPath or getcwd() directories before passing it
> to the plugin.

On thinking further, I feel path name completion (fullPathFor:) is best
delegated to plugin since the specific set of paths to lookup are
platform-specific. For legacy plugins, the image can fallback to the
first valid path in the list {name, imagePath/name or vmPath/name}.

FileDirectory should just be a tuple (container, entry). A FileDirectory
instance is a directory if its entry is a directory or a file otherwise.
A root is a directory with empty strings for both.

Does this proposal make sense?

Regards .. Subbu

There are several layers actually.
I think while not completely nice, the Primitive part is ok-ish.
The FileDirectory-Abstraction is a bit due for refurbishment, but the FileSystem-project is  due for Trunk-merge a while now, right?
-t

On 14/06/20 7:27 pm, Jakob Reschke wrote:
> In my understanding and in most of the libraries I have seen, a path is
> a sequence of parts that mostly make up the directories and the last
> element is the name of either a file or a directory. You may represent
> the whole thing in a single string with a separator between the
> elements.
This hierarchical name space breaks down these days because of links and
bind mounts. On unix, names are just links to a file object. There can
be multiple links (within or even external to a filesystem). Therefore,
the interpretation '..' differs from system to system.

> An agreement between image and plugin never to pass relative paths
> between each other could make sense to me. But I still don't see the
> value of distinguishing relative paths and incomplete paths, or how
> ./foo/bar would be any more or less ambiguous for further processing
> than just foo/bar. In both cases you need to know the startpoint (.) on
> both sides (image, plugin), or one side will not know what it is talking
> about.

If I pass "startup.st" as a document name to the VM,
primitiveFileOpen("startup.st", ...) has the option of resolving paths
by looking up a set of directories whereas if the path is given as
./startup.st primitiveFileOpen("./startup.st", ...) has to look in CWD
only. Currently, FileDirectory forces "startup.st" to mean
imagePath/startup.st only, even if the vm is not started from the image
directory.

e.g. Suppose app/ contains startup.st bin/ and shared/. Then this will fail:

    app$ bin/squeak shared/squeak.image startup.st

The FileDirectory should just be a tuple (container, entry) and move all
  platform-specific stuff into FilePlugin.

> Subbu, you write about a platform-specific set of paths to look up. Do
> you mean resolving against environment variables such as $PATH?
Yes. Plugins can use env variables to guide path resolution. The unix vm
already uses env variables like SQUEAK_VM, SQUEAK_IMAGE, SQUEAK_USERDIR
to avoid hard-coded paths in the image. This means Squeak applications
could place a startup document in, say, $HOME/.squeak/startup.st or
/etc/squeak/startup.st and have the image open it with just 'startup.st'
without hard-coding the full path.

Regards .. Subbu

Am So., 14. Juni 2020 um 20:21 Uhr schrieb K K Subbu <[hidden email]>:
So some of the path parts in the middle could actually be links,
reparse points, or whatever. But should our VM or Squeak even care,
unless we use a primitive for something like readlink? The OS resolves
such things for us.

Wait a minute, is that what you think it should be like, or is it how
it is at the moment? The primitive could do anything of course, but in
my opinion it must not use this "option", see further down. Also the
primitive should ideally not get to see the path with ./ still in it.

> Currently, FileDirectory forces "startup.st" to mean
> imagePath/startup.st only, even if the vm is not started from the image
> directory.

As far as I understood, this is because the VM actually changes the
CWD to the image's parent directory during startup. If it would not do
that, this could behave differently, for example resolving startup.st
relative to the invoking process's CWD (which is inherited).

I think we should not make ./startup.st and startup.st mean different
things in Squeak because it would be highly confusing to everyone who
has ever used a shell.

>
> e.g. Suppose app/ contains startup.st bin/ and shared/. Then this will fail:
>
>     app$ bin/squeak shared/squeak.image startup.st

...which is unexpected for the uninitiated user/developer as we saw in
Christoph's question. I don't think that it is good behavior, it
should work, not fail.

>
> The FileDirectory should just be a tuple (container, entry) and move all
>   platform-specific stuff into FilePlugin.

Sorry, I still fail to see how this representation would change
anything about the situation or how it is related at all to the
primitive interface. Could you explain what this representation
implies in practice?

>
> > Subbu, you write about a platform-specific set of paths to look up. Do
> > you mean resolving against environment variables such as $PATH?
> Yes. Plugins can use env variables to guide path resolution.

In my opinion, a FilePlugin implementation must not do this. Neither
Linux open(2) nor Win32 OpenFile use a search path. Why should the
VM/plugin introduce such intransparent behavior for basic file access?

env(1) or exec(2) and friends use PATH, and so does ShellExecute under
Windows, but CreateProcess does not (only looks in the current drive
and directory). These functions are used to launch other processes,
but that is in the domain of OSProcess, not in the domain of file
access primitives.

> The unix vm
> already uses env variables like SQUEAK_VM, SQUEAK_IMAGE, SQUEAK_USERDIR
> to avoid hard-coded paths in the image. This means Squeak applications
> could place a startup document in, say, $HOME/.squeak/startup.st or
> /etc/squeak/startup.st and have the image open it with just 'startup.st'
> without hard-coding the full path.

Is this already the case and applies to FileStream readOnlyFileNamed:?
If yes, oh my...

I would expect that you can get these locations/paths via some
primitive, just like in the FileSystem locators, but not that they are
magically preprended to any paths that are passed to the primitive. If
at all, this should happen with the CWD, at the discretion of the OS.
If a particular platform has no concept equivalent to a working
directory, we must either forbid passing relative paths to primitives,
or use a fallback instead on that platform, such as the image's
directory.

From my image:
FSLocator home asReference "=> C:\Users\Jakob"
FSLocator home resolve: './foo' "=> {home}\foo (this is portable, as
in: changes the actual location depending on where you run it)"
FSLocator home resolve: 'foo' "=> {home}\foo (so, the same as with ./)"
(FSLocator home resolve: './foo') class "=> FSLocator"
(FSLocator home resolve: './foo') asReference "=> C:\Users\Jakob\foo"
FSLocator vmDirectory asReference "=> C:\Squeak\VMs\Squeak64"
FSLocator image asReference "=> C:\Users\Jakob\HiDrive\Squot-trunk-64bit.image"
FileSystem disk workingDirectory "=> C:\Users\Jakob\HiDrive"
FileDirectory default "=> DosFileDirectory on 'C:\Users\Jakob\HiDrive'"
(FileSystem disk * 'foo') asAbsolute "=> C:\Users\Jakob\HiDrive\foo"
(FileSystem disk resolve: 'foo') "=> FSPath / 'C:' / 'Users' / 'Jakob'
/ 'HiDrive' / 'foo'"
(FileSystem disk resolve: './foo') "=> FSPath / 'C:' / 'Users' /
'Jakob' / 'HiDrive' / 'foo'"

Paths are converted to strings just before the primitives are called.
The conversion works in a platform-specific manner, or rather
FileSystem-specific. The paths are already resolved before being
converted to strings, where resolved means they are absolute and
contain no more ./ or ../.

On 15/06/20 3:08 am, Jakob Reschke wrote:
Precisely, path resolution should be in the OS.

The startup document path comes from the command line, so it is
OS-specific. Therefore, it's path resolution will also be OS-specific.

>> Currently, FileDirectory forces "startup.st" to mean
>> imagePath/startup.st only, even if the vm is not started from the image
>> directory.
>
> As far as I understood, this is because the VM actually changes the
> CWD to the image's parent directory during startup. If it would not do
> that, this could behave differently, for example resolving startup.st
> relative to the invoking process's CWD (which is inherited).
This is a myth. I have no idea how this myth originated. Does the
Windows port do it? The unix port doesn't change its working directory.
You can check the process properties at /proc/$pid/environ to verify
this. The image can also be saved in any directory. The VM does not
switch its CWD to track the new path.
$ pwd
/opt
$ /opt/share/bin/sqlinux.68019/bin/squeak
share/squeak/images/Squeak6.0alpha-19603-64bit.image
share/squeak/beeper.st &
[2] 15159
$ tr '\0' '\n' </proc/15159/environ | grep PWD
OLDPWD=/opt/share/squeak
PWD=/opt
$

> I think we should not make ./startup.st and startup.st mean different
> things in Squeak because it would be highly confusing to everyone who
> has ever used a shell.

Any beginner who has tried to compile and run test.c will understand the
difference anymore ;-). It was often used to prank unix beginners :-).

XDG has a whole specification to supply default paths for desktop,
documents, pictures etc.

>>
>> e.g. Suppose app/ contains startup.st bin/ and shared/. Then this will fail:
>>
>>      app$ bin/squeak shared/squeak.image startup.st
>
> ...which is unexpected for the uninitiated user/developer as we saw in
> Christoph's question. I don't think that it is good behavior, it
> should work, not fail.
The document path is an argument. It could come from anywhere not
necessarily from someone typing it on the command line. Insisting that
it always be /.../startup.st or $IMAGEDIR/startup.st is burdensome. In
fact, I burnt my fingers while testing the above code. It was loading an
leftover startup.st in the image directory, not the one I specified on
the command line.

>> The FileDirectory should just be a tuple (container, entry) and move all
>>    platform-specific stuff into FilePlugin.
>
> Sorry, I still fail to see how this representation would change
> anything about the situation or how it is related at all to the
> primitive interface. Could you explain what this representation
> implies in practice?

The FileDirectory just encapsulates a native store. A native store is
modeled as a pair - an entry (which can be streamed) and an array of
entries (container) accessed by entriesDo:. An entry can also be an
array (returns true to isDirectory). That's it. This is the same pattern
followed for reifications of other native resources.

I used the words entry and container so that these are not confused with
OS objects like file, directory etc. These are not terms used in Squeak.

An analogy is how Display is modeled by DisplayScreen and plugins map
this into the host's display implementation. Squeak doesn't pull in X11
display server, framebuffer or Win32 GUI classes into the image.

I am not saying this is how it should be. It is just a proposal inspired
by Squeak's design principles of simplicity and generality. They are
open to debate, discussion and disposal.

>>> Subbu, you write about a platform-specific set of paths to look up. Do
>>> you mean resolving against environment variables such as $PATH?
>> Yes. Plugins can use env variables to guide path resolution.
>
> In my opinion, a FilePlugin implementation must not do this. Neither
> Linux open(2) nor Win32 OpenFile use a search path. Why should the
> VM/plugin introduce such intransparent behavior for basic file access

Because Squeak is a virtual machine, not a shell, and depends on the
host for storage, display and other i/o services without pulling in
implementations into the image.

>> The unix vm
>> already uses env variables like SQUEAK_VM, SQUEAK_IMAGE, SQUEAK_USERDIR
>> to avoid hard-coded paths in the image. This means Squeak applications
>> could place a startup document in, say, $HOME/.squeak/startup.st or
>> /etc/squeak/startup.st and have the image open it with just 'startup.st'
>> without hard-coding the full path.
>
> Is this already the case and applies to FileStream readOnlyFileNamed:?

No. But lazy binding of pathnames in executables is a long standing unix
convention. Lazy binding is essential part of live programming. Stephen
Kell talks about "Liberating the Smalltalk lurking in C and Unix" at
https://youtu.be/LwicN2u6Dro (41 mins!).

>  From my image:
> FSLocator home asReference "=> C:\Users\Jakob"
> FSLocator home resolve: './foo' "=> {home}\foo (this is portable, as

This is essentially a command shell function. Pharo has become a command
shell on the host, handling file globbing, tracking process properties
(HOME, PWD etc.). It is a totally different topic.

Regards .. Subbu

---