Smalltalk › Pharo › Pharo Smalltalk Developers

File path encoding

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

8 messages Options

Sven Van Caekenberghe-2

File path encoding

Hi,

Some people raise issues with file path encoding with non-ASCII characters.

The following works fine for me (macOS, Pharo 7, using new File class):

"works"

(FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].

(FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].

(FileLocator temp / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].

"works"

(FileLocator temp / 'benoît') ensureCreateDirectory.

(FileLocator temp / 'benoît' / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].

(FileLocator temp / 'benoît' / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].

Can Linux/Windows users confirm that it works on their platform ?

Sven

Ben Coman

Re: File path encoding

On 16 April 2018 at 17:51, Sven Van Caekenberghe <[hidden email]> wrote:

> Hi,
>
> Some people raise issues with file path encoding with non-ASCII characters.
>
> The following works fine for me (macOS, Pharo 7, using new File class):
>
> "works"
>
> (FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>
> (FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].
>
> "works"
>
> (FileLocator temp / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>
> (FileLocator temp / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>
> "works"
>
> (FileLocator temp / 'benoît') ensureCreateDirectory.
>
> (FileLocator temp / 'benoît' / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>
> (FileLocator temp / 'benoît' / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>
>
> Can Linux/Windows users confirm that it works on their platform ?

Thanks for attacking this. All three work on Windows 10 English locale.
I'd hope it wasn't different with French locale, but wait for a report on that.

cheers -ben

Ben Coman

Re: File path encoding

On 16 April 2018 at 18:52, Ben Coman <[hidden email]> wrote:

> On 16 April 2018 at 17:51, Sven Van Caekenberghe <[hidden email]> wrote:
>> Hi,
>>
>> Some people raise issues with file path encoding with non-ASCII characters.
>>
>> The following works fine for me (macOS, Pharo 7, using new File class):
>>
>> "works"
>>
>> (FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>
>> (FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>
>> "works"
>>
>> (FileLocator temp / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>
>> (FileLocator temp / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>
>> "works"
>>
>> (FileLocator temp / 'benoît') ensureCreateDirectory.
>>
>> (FileLocator temp / 'benoît' / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>
>> (FileLocator temp / 'benoît' / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>
>>
>> Can Linux/Windows users confirm that it works on their platform ?
>
> Thanks for attacking this. All three work on Windows 10 English locale.
> I'd hope it wasn't different with French locale, but wait for a report on that.
>
> cheers -ben

I unzipped a fresh download of http://files.pharo.org/platform/Pharo6.1-win.zip
* into C:\Temp\Pharo6.1
and immediately copied that directory
* into C:\Temp\Pharo6.1.benoît

Double-clicking on Pharo.exe
* in the former, opens Pharo an expected.
* in the latter, does nothing.

cheers -ben

Sven Van Caekenberghe-2

Re: File path encoding

Ben,

On my machine, macOS 10.13.4,

$ pwd
/tmp/benoît

$ curl get.pharo.org/70+vm | bash
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3036 100 3036 0 0 38694 0 --:--:-- --:--:-- --:--:-- 38923
Downloading the latest 70 Image:
http://files.pharo.org/get-files/70/pharo.zip
Pharo.image
Downloading the latest pharoVM:
http://files.pharo.org/get-files/70/pharo-mac-stable.zip
pharo-vm/Pharo.app/Contents/MacOS/Pharo
Creating starter scripts pharo and pharo-ui

$ ./pharo Pharo.image printVersion
[version] 'Pharo7.0' 'Pharo-7.0+alpha.build.769.sha.244850d3ffc1b49ab716c8093f4bcd9af3917be6 (32 Bit)'

$ ./pharo Pharo.image eval 'FileLocator image resolve'
File @ /private/tmp/benoît/Pharo.image

And the UI version works too.

So we're back at 'it is a Windows problem' (but maybe it is only visible on Windows).

Can you see what is going on ?

Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

> On 16 Apr 2018, at 13:05, Ben Coman <[hidden email]> wrote:
>
> On 16 April 2018 at 18:52, Ben Coman <[hidden email]> wrote:
>> On 16 April 2018 at 17:51, Sven Van Caekenberghe <[hidden email]> wrote:
>>> Hi,
>>>
>>> Some people raise issues with file path encoding with non-ASCII characters.
>>>
>>> The following works fine for me (macOS, Pharo 7, using new File class):
>>>
>>> "works"
>>>
>>> (FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>>
>>> (FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>>
>>> "works"
>>>
>>> (FileLocator temp / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>>
>>> (FileLocator temp / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>>
>>> "works"
>>>
>>> (FileLocator temp / 'benoît') ensureCreateDirectory.
>>>
>>> (FileLocator temp / 'benoît' / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>>
>>> (FileLocator temp / 'benoît' / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>>
>>>
>>> Can Linux/Windows users confirm that it works on their platform ?
>>
>> Thanks for attacking this. All three work on Windows 10 English locale.
>> I'd hope it wasn't different with French locale, but wait for a report on that.
>>
>> cheers -ben
>
> I unzipped a fresh download of http://files.pharo.org/platform/Pharo6.1-win.zip
> * into C:\Temp\Pharo6.1
> and immediately copied that directory
> * into C:\Temp\Pharo6.1.benoît
>
> Double-clicking on Pharo.exe
> * in the former, opens Pharo an expected.
> * in the latter, does nothing.
>
> cheers -ben
>

alistairgrant

Re: File path encoding

Hi Sven,

On 16 April 2018 at 13:57, Sven Van Caekenberghe <[hidden email]> wrote:
>
> Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

I'm not going to have time to look at this in the next few days, sorry.

But I have a vague memory that some of the other primitives outside
FilePlugin don't add the "\\?\" prefix, so they possibly also don't do
wide character conversion.

Cheers,
Alistair

Sven Van Caekenberghe-2

Re: File path encoding

> On 16 Apr 2018, at 14:51, Alistair Grant <[hidden email]> wrote:
>
> Hi Sven,
>
> On 16 April 2018 at 13:57, Sven Van Caekenberghe <[hidden email]> wrote:
>>
>> Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.
>
> I'm not going to have time to look at this in the next few days, sorry.
>
> But I have a vague memory that some of the other primitives outside
> FilePlugin don't add the "\\?\" prefix, so they possibly also don't do
> wide character conversion.

Then I hope some other Windows user/developer can have a look.
Hell, I believe we got a few French people working on Pharo, n'est pas ?

> Cheers,
> Alistair
>

Ben Coman

Re: File path encoding

In reply to this post by Sven Van Caekenberghe-2

>> On 16 Apr 2018, at 13:05, Ben Coman <[hidden email]> wrote:
>>
>> On 16 April 2018 at 18:52, Ben Coman <[hidden email]> wrote:
>>> On 16 April 2018 at 17:51, Sven Van Caekenberghe <[hidden email]> wrote:
>>>> Hi,
>>>>
>>>> Some people raise issues with file path encoding with non-ASCII characters.
>>>>
>>>> The following works fine for me (macOS, Pharo 7, using new File class):
>>>>
>>>> "works"
>>>>
>>>> (FileLocator temp / 'foo' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>>>
>>>> (FileLocator temp / 'foo' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>>>
>>>> "works"
>>>>
>>>> (FileLocator temp / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>>>
>>>> (FileLocator temp / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>>>
>>>> "works"
>>>>
>>>> (FileLocator temp / 'benoît') ensureCreateDirectory.
>>>>
>>>> (FileLocator temp / 'benoît' / 'benoît' , 'txt') writeStreamDo: [ :out | out << 'Les élèves Français' ; crlf ].
>>>>
>>>> (FileLocator temp / 'benoît' / 'benoît' , 'txt') readStreamDo: [ :in | in upToEnd ].
>>>>
>>>>
>>>> Can Linux/Windows users confirm that it works on their platform ?
>>>
>>> Thanks for attacking this. All three work on Windows 10 English locale.
>>> I'd hope it wasn't different with French locale, but wait for a report on that.
>>>
>>> cheers -ben
>>
>> I unzipped a fresh download of http://files.pharo.org/platform/Pharo6.1-win.zip
>> * into C:\Temp\Pharo6.1
>> and immediately copied that directory
>> * into C:\Temp\Pharo6.1.benoît
>>
>> Double-clicking on Pharo.exe
>> * in the former, opens Pharo an expected.
>> * in the latter, does nothing.

On 16 April 2018 at 19:57, Sven Van Caekenberghe <[hidden email]> wrote:

> Ben,
>
> On my machine, macOS 10.13.4,
>
> $ pwd
> /tmp/benoît
>
> $ curl get.pharo.org/70+vm | bash
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 3036 100 3036 0 0 38694 0 --:--:-- --:--:-- --:--:-- 38923
> Downloading the latest 70 Image:
> http://files.pharo.org/get-files/70/pharo.zip
> Pharo.image
> Downloading the latest pharoVM:
> http://files.pharo.org/get-files/70/pharo-mac-stable.zip
> pharo-vm/Pharo.app/Contents/MacOS/Pharo
> Creating starter scripts pharo and pharo-ui
>
> $ ./pharo Pharo.image printVersion
> [version] 'Pharo7.0' 'Pharo-7.0+alpha.build.769.sha.244850d3ffc1b49ab716c8093f4bcd9af3917be6 (32 Bit)'
>
> $ ./pharo Pharo.image eval 'FileLocator image resolve'
> File @ /private/tmp/benoît/Pharo.image
>
> And the UI version works too.
>
> So we're back at 'it is a Windows problem' (but maybe it is only visible on Windows).
>
> Can you see what is going on ?
>
> Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.

I tried logging the startup like this...

WorkingSession >> runStartup: isImageStarting

'C:\Temp\StartupLog.txt' asFileReference writeStreamDo: [ :log |

log nextPutAll: log className; flush.

self

runList: manager startupList

do: [ :each |

log

nextPutAll: each printString;

nextPutAll: String crlf;

flush.

each startup: isImageStarting ].

self executeDeferredStartupActions: isImageStarting

]

which worked in the former and with the latter(benoît) nothing.

But poking around I bumped into an interesting symptom.

The following does nothing....

C:\Temp\Pharo6.1.benoît> pharo

but the following successfully opens the Image...

C:\Temp\Pharo6.1.benoît> pharo Pharo6.1.image

Not sure where to go from there. Maybe needs a Windows Debug-VM,

but I'm out of time for now.

cheers -ben

Sven Van Caekenberghe-2

Re: File path encoding

In reply to this post by Sven Van Caekenberghe-2

Actually, (from another thread) I think that it is related to how environment variables are accessed.

Consider:

$ FOO=benoît ./pharo Pharo.image eval 'OSEnvironment current associations'
{'TERM_PROGRAM'->'Apple_Terminal'. 'TERM'->'xterm-256color'. 'SHELL'->'/bin/bash'. 'TMPDIR'->'/var/folders/sy/sndrtj9j1tq06j0lfnshmrl80000gn/T/'. 'FOO'->'benoÃ®t'. 'Apple_PubSub_Socket_Render'->'/private/tmp/com.apple.launchd.uWk7pivcLT/Render'. 'TERM_PROGRAM_VERSION'->'404'. 'TERM_SESSION_ID'->'845BECCD-0AB0-4686-B7F9-3A0FF84BDCB7'. 'USER'->'sven'. 'SSH_AUTH_SOCK'->'/private/tmp/com.apple.launchd.y5oCwdUyaG/Listeners'. 'PATH'->'/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/texbin:/opt/X11/bin'. 'PWD'->'/tmp/benoiÌt'. 'XPC_FLAGS'->'0x0'. 'XPC_SERVICE_NAME'->'0'. 'HOME'->'/Users/sven'. 'SHLVL'->'2'. 'LOGNAME'->'sven'. 'LC_CTYPE'->'UTF-8'. 'DISPLAY'->'/private/tmp/com.apple.launchd.lsgASYFiWW/org.macosforge.xquartz:0'. 'SECURITYSESSIONID'->'186a9'. 'OLDPWD'->'/tmp/benoiÌt'. '_'->'/tmp/benoiÌt/pharo-vm/Pharo.app/Contents/MacOS/Pharo'. '__CF_USER_TEXT_ENCODING'->'0x1F5:0x0:0x0'}

As you can see, several values are wrongly decoded - and this is on macOS as well.

> On 16 Apr 2018, at 15:10, Sven Van Caekenberghe <[hidden email]> wrote:
>
>
>
>> On 16 Apr 2018, at 14:51, Alistair Grant <[hidden email]> wrote:
>>
>> Hi Sven,
>>
>> On 16 April 2018 at 13:57, Sven Van Caekenberghe <[hidden email]> wrote:
>>>
>>> Technically, at the File (FilePlugin) level we seem to be able to handle paths with strange characters, even on Windows, but somehow, somewhere a bogus call is made during image startup.
>>
>> I'm not going to have time to look at this in the next few days, sorry.
>>
>> But I have a vague memory that some of the other primitives outside
>> FilePlugin don't add the "\\?\" prefix, so they possibly also don't do
>> wide character conversion.
>
> Then I hope some other Windows user/developer can have a look.
> Hell, I believe we got a few French people working on Pharo, n'est pas ?
>
>> Cheers,
>> Alistair