Smalltalk › Pharo › Pharo Smalltalk Users

issue with large file path on windows

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

9 messages Options

Nicolai Hess

issue with large file path on windows

For those who had problems with pharo on windows and github based repositories,

I built a windows vm with support for long paths:

https://drive.google.com/file/d/0B8yEahnuIem2bmxwdzJuUXFxVGM/view?usp=sharing

For browsing directories with large paths (FileList or Inspect),
you may need one additional change in the image (But I am not really sure about that) :

DiskStore>>initialize
super initialize.
maxFileNameLength := Smalltalk vm maxFilenameLength ifNil: [ 32767 ].

please test and give feedback.

This wasn't as easy as I thought, and I had to make some more changes

for the file permissions (the stat-functions don't work for files with long paths).

Please test other file/folder operations.

nicolai

Tudor Girba-2

Re: [Pharo-dev] issue with large file path on windows

This is great news!

I had tons of problems last year with that and I did not know what to do about this.

Doru

On Sun, Aug 23, 2015 at 1:44 PM, Nicolai Hess <[hidden email]> wrote:

For those who had problems with pharo on windows and github based repositories,
I built a windows vm with support for long paths:

https://drive.google.com/file/d/0B8yEahnuIem2bmxwdzJuUXFxVGM/view?usp=sharing

For browsing directories with large paths (FileList or Inspect),
you may need one additional change in the image (But I am not really sure about that) :

DiskStore>>initialize
super initialize.
maxFileNameLength := Smalltalk vm maxFilenameLength ifNil: [ 32767 ].

please test and give feedback.

This wasn't as easy as I thought, and I had to make some more changes
for the file permissions (the stat-functions don't work for files with long paths).
Please test other file/folder operations.

nicolai

www.tudorgirba.com

"Every thing has its own flow"

Thierry Goubier

Re: [Pharo-dev] issue with large file path on windows

Le 23/08/2015 14:28, Tudor Girba a écrit :
> This is great news!
>
> I had tons of problems last year with that and I did not know what to do
> about this.

A possible solution would have been to rewrite a bit the filetree
support so that it could load packages directly from the zip archive
downloaded from github, instead of unpackaging it to disk.

But... pushing for the windows vm so that it would use the right API is
a lot better :)

Thanks Nicolai,

Thierry

> Doru
>
> On Sun, Aug 23, 2015 at 1:44 PM, Nicolai Hess <[hidden email]
> <mailto:[hidden email]>> wrote:
>
> For those who had problems with pharo on windows and github based
> repositories,
> I built a windows vm with support for long paths:
>
> https://drive.google.com/file/d/0B8yEahnuIem2bmxwdzJuUXFxVGM/view?usp=sharing
>
>
> For browsing directories with large paths (FileList or Inspect),
> you may need one additional change in the image (But I am not really
> sure about that) :
>
> DiskStore>>initialize
> super initialize.
> maxFileNameLength := Smalltalk vm maxFilenameLength ifNil: [
> 32767 ].
>
>
> please test and give feedback.
>
> This wasn't as easy as I thought, and I had to make some more changes
> for the file permissions (the stat-functions don't work for files
> with long paths).
> Please test other file/folder operations.
>
>
> nicolai
>
>
>
>
>
>
> --
> www.tudorgirba.com <http://www.tudorgirba.com>
>
> "Every thing has its own flow"

Nicolai Hess

Re: issue with large file path on windows

In reply to this post by Nicolai Hess

And If you want to review the changes:

https://github.com/nicolaihess/pharo-vm/compare/master...nicolaihess:win-long-filename

2015-08-23 13:44 GMT+02:00 Nicolai Hess <[hidden email]>:

For those who had problems with pharo on windows and github based repositories,
I built a windows vm with support for long paths:

https://drive.google.com/file/d/0B8yEahnuIem2bmxwdzJuUXFxVGM/view?usp=sharing

For browsing directories with large paths (FileList or Inspect),
you may need one additional change in the image (But I am not really sure about that) :

DiskStore>>initialize
super initialize.
maxFileNameLength := Smalltalk vm maxFilenameLength ifNil: [ 32767 ].

please test and give feedback.

This wasn't as easy as I thought, and I had to make some more changes
for the file permissions (the stat-functions don't work for files with long paths).
Please test other file/folder operations.

nicolai

Henrik Sperre Johansen

Re: issue with large file path on windows

 On 23 Aug 2015, at 6:09 , Nicolai Hess <[hidden email]> wrote:

And If you want to review the changes:

https://github.com/nicolaihess/pharo-vm/compare/master...nicolaihess:win-long-filename


2015-08-23 13:44 GMT+02:00 Nicolai Hess <[hidden email]>:
For those who had problems with pharo on windows and github based repositories,
I built a windows vm with support for long paths:

https://drive.google.com/file/d/0B8yEahnuIem2bmxwdzJuUXFxVGM/view?usp=sharing


For browsing directories with large paths (FileList or Inspect),
you may need one additional change in the image (But I am not really sure about that) :

DiskStore>>initialize
    super initialize.
    maxFileNameLength := Smalltalk vm maxFilenameLength ifNil: [ 32767 ].


please test and give feedback.

This wasn't as easy as I thought, and I had to make some more changes
for the file permissions (the stat-functions don't work for files with long paths).
Please test other file/folder operations.


nicolai







+  #define CONVERT_MULTIBYTE_TO_WIDECHAR_PATH(buffer, size, fileNameString, fileNameLength)  {	\
+    buffer = (WCHAR*)alloca((size+4+1)*sizeof(WCHAR));\
+    buffer[0] = L'\\';buffer[1] = L'\\'; buffer[2] = L'?'; buffer[3] = L'\\';\
+    MultiByteToWideChar(CP_UTF8, 0, fileNameString, fileNameLength, buffer + 4, size);\
+    buffer[size + 4] = 0;\
+    size += 4;}


- Is an alloca version really worth it for the potential problems?
- Also, LONG_PATH_PREFIX_SIZE is defined above in the header, should probably use this instead of 4 in the last two lines?
- The comment on lines 111/164/265/381, while not modified, are somewhat incorrect, null-terminated C-string indicates UTF8, which we are converting from, not into.

I'm curious what the "<a href="smb://?/" class="">\\?\" prefix does to hasCaseSensitiveDuplicate (adding a new ? segment to the path), but I cba to understand that code, and considering
int caseSensitiveFileMode = 0;
if(!caseSensitiveFileMode) return 0;

I guess it doesn't really matter.

Otherwise, looks good to me!

Cheers,
Henry
++  #define CONVERT_MULTIBYTE_TO_WIDECHAR_PATH(buffer, size, fileNameString, fileNameLength)  {	\
++    buffer = (WCHAR*)alloca((size+4+1)*sizeof(WCHAR));\
++    buffer[0] = L'\\';buffer[1] = L'\\'; buffer[2] = L'?'; buffer[3] = L'\\';\
++    MultiByteToWideChar(CP_UTF8, 0, fileNameString, fileNameLength, buffer + 4, size);\
++    buffer[size + 4] = 0;\
++    size += 4;}

Nicolai Hess

Re: issue with large file path on windows

2015-08-24 13:34 GMT+02:00 Henrik Johansen <[hidden email]>:

On 23 Aug 2015, at 6:09 , Nicolai Hess <[hidden email]> wrote:

And If you want to review the changes:

https://github.com/nicolaihess/pharo-vm/compare/master...nicolaihess:win-long-filename

2015-08-23 13:44 GMT+02:00 Nicolai Hess <[hidden email]>:
For those who had problems with pharo on windows and github based repositories,
I built a windows vm with support for long paths:

https://drive.google.com/file/d/0B8yEahnuIem2bmxwdzJuUXFxVGM/view?usp=sharing

For browsing directories with large paths (FileList or Inspect),
you may need one additional change in the image (But I am not really sure about that) :

DiskStore>>initialize
super initialize.
maxFileNameLength := Smalltalk vm maxFilenameLength ifNil: [ 32767 ].

please test and give feedback.

This wasn't as easy as I thought, and I had to make some more changes
for the file permissions (the stat-functions don't work for files with long paths).
Please test other file/folder operations.

nicolai

Thank you Henrik, for looking at this

+ #define CONVERT_MULTIBYTE_TO_WIDECHAR_PATH(buffer, size, fileNameString, fileNameLength) { \
+ buffer = (WCHAR*)alloca((size+4+1)*sizeof(WCHAR));\
+ buffer[0] = L'\\';buffer[1] = L'\\'; buffer[2] = L'?'; buffer[3] = L'\\';\
+ MultiByteToWideChar(CP_UTF8, 0, fileNameString, fileNameLength, buffer + 4, size);\
+ buffer[size + 4] = 0;\
+ size += 4;}

- Is an alloca version really worth it for the potential problems?

It is what Marcel used for the long path support in the squeak vm (I would have copy and paste the whole change, but I couldn't find the appropriate change for directory functions).

- Also, LONG_PATH_PREFIX_SIZE is defined above in the header, should probably use this instead of 4 in the last two lines?

Yes

- The comment on lines 111/164/265/381, while not modified, are somewhat incorrect, null-terminated C-string indicates UTF8, which we are converting from, not into.

Right, I ll chang that

I'm curious what the "\\?\" prefix does to hasCaseSensitiveDuplicate (adding a new ? segment to the path), but I cba to understand that code, and considering
int caseSensitiveFileMode = 0;
if(!caseSensitiveFileMode) return 0;

I guess it doesn't really matter.

Yes, I tried to make this change in a way that does not break the case sensitive duplicate check, but even without my change,

enabling the caseSenstiveFileMode makes the windows vm unusable. So, at least, it is no worse.

Otherwise, looks good to me!

Cheers,
Henry

Ben Coman

Re: [Pharo-dev] issue with large file path on windows

In reply to this post by Nicolai Hess

(CC'd vm-dev)

On Mon, Aug 24, 2015 at 12:09 AM, Nicolai Hess <[hidden email]> wrote:
> And If you want to review the changes:
>
> https://github.com/nicolaihess/pharo-vm/compare/master...nicolaihess:win-long-filename

I don't have a Windows machine to test on, but I perused the changes.
one spelling defect ../platforms/win32/vm/sqWin32.h, line 452 "functiosn"

Now my curiosity is aroused regarding sqFileDeleteNameSize() in [1].
Where does its parameter "char* fileNameIndex" get its null termination?
I can only guess maybe the following definition in primitiveFileDelete.st [2]
<var: 'nameIndex' type: 'char *'>
automatically appends a null to the Smalltalk string. My understanding
is that the Smalltalk string's internal representation is not null
terminated.

I ask since the sqFileDeleteNameSize()'s comment "convert the file
name into a null-terminated C string" indicates the null is added by
MultiByteToWideChar(), but [3] indicates that is not true -- that the
output of MultiByteToWideChar() is only null terminated if the input
is null terminated.

Also, anyone know if the concern "BIG WARNING about the return value
and the `cbMultiByte` parameter" expressed near the end of [3] is
relevant?

[1] ../pharo-vm/platforms/win32/plugins/FilePlugin/sqWin32FilePrims.c
[2] ../pharo-vm/mc/VMMaker.oscog.package/FilePlugin.class/instance/primitiveFileDelete.st
[3] https://msdn.microsoft.com/en-us/library/windows/desktop/dd319072(v=vs.85).aspx

Henrik Sperre Johansen

Re: [Vm-dev] [Pharo-dev] issue with large file path on windows

On 24 Aug 2015, at 4:59 , Ben Coman <[hidden email]> wrote:

(CC'd vm-dev)

On Mon, Aug 24, 2015 at 12:09 AM, Nicolai Hess <[hidden email]> wrote:
And If you want to review the changes:

https://github.com/nicolaihess/pharo-vm/compare/master...nicolaihess:win-long-filename

I don't have a Windows machine to test on, but I perused the changes.
one spelling defect ../platforms/win32/vm/sqWin32.h, line 452 "functiosn"

Now my curiosity is aroused regarding sqFileDeleteNameSize() in [1].
Where does its parameter "char* fileNameIndex" get its null termination?
I can only guess maybe the following definition in primitiveFileDelete.st [2]
<var: 'nameIndex' type: 'char *'>
automatically appends a null to the Smalltalk string. My understanding
is that the Smalltalk string's internal representation is not null
terminated.

I ask since the sqFileDeleteNameSize()'s comment "convert the file
name into a null-terminated C string" indicates the null is added by
MultiByteToWideChar(), but [3] indicates that is not true -- that the
output of MultiByteToWideChar() is only null terminated if the input
is null terminated.

That's not what 3 indicates, the way I read it:

"cbMultiByte [in]

Size, in bytes, of the string indicated by the lpMultiByteStr parameter. Alternatively, this parameter can be set to -1 if the string is null-terminated. "

"cchWideChar [in]

Size, in characters, of the buffer indicated by lpWideCharStr. If this value is 0, the function returns the required buffer size, in characters, including any terminating null character"

So we first make a call using cbMultiByte > 0 and cchWideChar = 0, which then returns the size of null-terminated WCHAR-buffer needed to hold the conversion of non-null terminated (but sized) parameter. (sz = size)

Then create that buffer, assign it to win32Path, and convert into it the CONVERT_MULTIBYTE_TO_WIDECHAR_PATH macro.

The Win APIs are written in a way that they seldom (never?) allocate memory, instead you'll see this pattern of "call me once with size = NULL to get the required size, then allocate and call me again with the actual pointer/size"

Cheers,

Henry

signature.asc (859 bytes) Download Attachment

Henrik Sperre Johansen

Re: [Vm-dev] [Pharo-dev] issue with large file path on windows

On 24 Aug 2015, at 5:19 , Henrik Johansen <[hidden email]> wrote:

On 24 Aug 2015, at 4:59 , Ben Coman <[hidden email]> wrote:

(CC'd vm-dev)

On Mon, Aug 24, 2015 at 12:09 AM, Nicolai Hess <[hidden email]> wrote:
And If you want to review the changes:

https://github.com/nicolaihess/pharo-vm/compare/master...nicolaihess:win-long-filename

I don't have a Windows machine to test on, but I perused the changes.
one spelling defect ../platforms/win32/vm/sqWin32.h, line 452 "functiosn"

Now my curiosity is aroused regarding sqFileDeleteNameSize() in [1].
Where does its parameter "char* fileNameIndex" get its null termination?
I can only guess maybe the following definition in primitiveFileDelete.st [2]
<var: 'nameIndex' type: 'char *'>
automatically appends a null to the Smalltalk string. My understanding
is that the Smalltalk string's internal representation is not null
terminated.

I ask since the sqFileDeleteNameSize()'s comment "convert the file
name into a null-terminated C string" indicates the null is added by
MultiByteToWideChar(), but [3] indicates that is not true -- that the
output of MultiByteToWideChar() is only null terminated if the input
is null terminated.

That's not what 3 indicates, the way I read it:
"cbMultiByte [in]
Size, in bytes, of the string indicated by the lpMultiByteStr parameter. Alternatively, this parameter can be set to -1 if the string is null-terminated. "
"cchWideChar [in]
Size, in characters, of the buffer indicated by lpWideCharStr. If this value is 0, the function returns the required buffer size, in characters, including any terminating null character"

So we first make a call using cbMultiByte > 0 and cchWideChar = 0, which then returns the size of null-terminated WCHAR-buffer needed to hold the conversion of non-null terminated (but sized) parameter. (sz = size)
Then create that buffer, assign it to win32Path, and convert into it the CONVERT_MULTIBYTE_TO_WIDECHAR_PATH macro.

The Win APIs are written in a way that they seldom (never?) allocate memory, instead you'll see this pattern of "call me once with size = NULL to get the required size, then allocate and call me again with the actual pointer/size"

Cheers,
Henry

And I guess I would be wrong, reading a bit further:

"MultiByteToWideChar does not null-terminate an output string if the input string length is explicitly specified without a terminating null character. To null-terminate an output string for this function, the application should pass in -1 or explicitly count the terminating null character for the input string."

It is handled in the macro though, which adds the null terminator:

buffer = (WCHAR*)malloc((size+ LONG_PATH_PREFIX_SIZE +1)*sizeof(WCHAR));\
buffer[size + 4] = 0;\

Cheers,Henry

signature.asc (859 bytes) Download Attachment