InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Andreas.Raab
This function is news to me - can somebody explain what it's good for,
where it should be used and (most importantly) why we'd export this for
plugins but don't use in the VM itself (e.g., no senders)?

Thanks,
   - Andreas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Andreas.Raab
Oh, and what is the return value?

Cheers,
   - Andreas

Andreas Raab wrote:
> This function is news to me - can somebody explain what it's good for,
> where it should be used and (most importantly) why we'd export this for
> plugins but don't use in the VM itself (e.g., no senders)?
>
> Thanks,
>   - Andreas
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

timrowledge
In reply to this post by Andreas.Raab

On 25-Mar-06, at 10:33 PM, Andreas Raab wrote:

> This function is news to me - can somebody explain what it's good  
> for, where it should be used and (most importantly) why we'd export  
> this for plugins but don't use in the VM itself (e.g., no senders)?

See mantis 2044 for some details and the original rationale. It also  
relates somewhat to :-
>>

>> From: tim Rowledge <[hidden email]>
>> Subject: Re: New VMMaker/svn release
>> Date: Fri, 30 Dec 2005 17:14:06 -0800
>> To: [hidden email]
>>
>> Hokay,
>> After taking Andreas' suggestion to dump the tacky old
>> sqFilenameFromString macro to heart, SM has a new release 11 -
>> 3.8b5-64B and SVN has level 1284 for you to play with. Read the SM
>> page comments.  It may well break things but I think it gets the
>> basic facility cleaner. I also cleaned out some ugly arg-typing
>> floobs while I was at it.
>>
>> Note that it looks to me as if AsynchFilePlugin and DeflatePlugin
>> could do with some looking at wrt 64bit cleanliness.

The resolveAliases bit at the end is an artifact of Apple having  
their head up their rectum when it comes to aliases in filepaths.  
Like the old macro sqFilenameFromStringOpen()



tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Design simplicity: It was developed on a shoe-string budget.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Andreas.Raab
tim Rowledge wrote:
>
> On 25-Mar-06, at 10:33 PM, Andreas Raab wrote:
>
>> This function is news to me - can somebody explain what it's good for,
>> where it should be used and (most importantly) why we'd export this
>> for plugins but don't use in the VM itself (e.g., no senders)?
>
> See mantis 2044 for some details and the original rationale. It also
> relates somewhat to :-

I guess you mean http://bugs.impara.de/view.php?id=2404 yes? One
question about this: Do we have a way of doing that kind of file name
conversion from the image itself? It strikes me as somewhat odd that a
plugin needs to do that conversion given that the image can just pass a
well-formed version of the string as the argument, no?

Cheers,
   - Andreas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Bert Freudenberg-3

Am 26.03.2006 um 09:11 schrieb Andreas Raab:

> tim Rowledge wrote:
>> On 25-Mar-06, at 10:33 PM, Andreas Raab wrote:
>>> This function is news to me - can somebody explain what it's good  
>>> for, where it should be used and (most importantly) why we'd  
>>> export this for plugins but don't use in the VM itself (e.g., no  
>>> senders)?
>> See mantis 2044 for some details and the original rationale. It  
>> also relates somewhat to :-
>
> I guess you mean http://bugs.impara.de/view.php?id=2404 yes? One  
> question about this: Do we have a way of doing that kind of file  
> name conversion from the image itself? It strikes me as somewhat  
> odd that a plugin needs to do that conversion given that the image  
> can just pass a well-formed version of the string as the argument, no?

Not if the VM creates the illusion of some other naming than the  
actual system uses. Like the use of HFS file names in the Mac VM  
(with : separators), where the VM translates to unix path names  
nowadays. The problem with the macro was that it called (on the Mac  
at least) a non-exported VM function. But in the freetype plugin I  
needed to be able to convert from the path name the image uses to the  
actual system path name. So just using the macro didn't work, which  
is why we added that function.

- Bert -

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

David T. Lewis
On Sun, Mar 26, 2006 at 11:35:35AM +0200, Bert Freudenberg wrote:

> Am 26.03.2006 um 09:11 schrieb Andreas Raab:
> > tim Rowledge wrote:
> >> On 25-Mar-06, at 10:33 PM, Andreas Raab wrote:
> >>> This function is news to me - can somebody explain what it's good  
> >>> for, where it should be used and (most importantly) why we'd  
> >>> export this for plugins but don't use in the VM itself (e.g., no  
> >>> senders)?
> >> See mantis 2044 for some details and the original rationale. It  
> >> also relates somewhat to :-
> >
> > I guess you mean http://bugs.impara.de/view.php?id=2404 yes? One  
> > question about this: Do we have a way of doing that kind of file  
> > name conversion from the image itself? It strikes me as somewhat  
> > odd that a plugin needs to do that conversion given that the image  
> > can just pass a well-formed version of the string as the argument, no?
>
> Not if the VM creates the illusion of some other naming than the  
> actual system uses. Like the use of HFS file names in the Mac VM  
> (with : separators), where the VM translates to unix path names  
> nowadays. The problem with the macro was that it called (on the Mac  
> at least) a non-exported VM function. But in the freetype plugin I  
> needed to be able to convert from the path name the image uses to the  
> actual system path name. So just using the macro didn't work, which  
> is why we added that function.

This is as good a time as any to mention that there is a potential for
buffer overrun problems in ioFileName:fromString:ofLength:resolveAliases.
By definition, the sender does not know what the file name will be translated
to, hence cannot know how large the pre-allocated aCharBuffer should be.

The sender in this case may be any arbitrary primitive written by some
clueless twit such as myself. That means that sqGetFilenameFromString()
needs to protect against buffer overruns.

Dave

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

timrowledge
In reply to this post by Andreas.Raab

On 25-Mar-06, at 11:11 PM, Andreas Raab wrote:

> tim Rowledge wrote:
>> On 25-Mar-06, at 10:33 PM, Andreas Raab wrote:
>>> This function is news to me - can somebody explain what it's good  
>>> for, where it should be used and (most importantly) why we'd  
>>> export this for plugins but don't use in the VM itself (e.g., no  
>>> senders)?
>> See mantis 2044 for some details and the original rationale. It  
>> also relates somewhat to :-
>
> I guess you mean http://bugs.impara.de/view.php?id=2404 yes?

Yup. Meant to fix that before hitting send but screwed up, obviously.

> One question about this: Do we have a way of doing that kind of  
> file name conversion from the image itself? It strikes me as  
> somewhat odd that a plugin needs to do that conversion given that  
> the image can just pass a well-formed version of the string as the  
> argument, no?
Actually, no. Until and unless we ever get some decent file handling  
code in the image we simply can't rely on a passed in filename being  
even vaguely decent. My preference is to call a VM routine to  
canonicalise the passed in string for every usage simply because you  
can rely (for some value of 'rely') on the OS getting it right in  
working out that '../../foo/bar/bim.bmp' is actually '/usr/mounts/
someOtherComputer/viaALink/mumble/frotz.jpeg'.

It also helps in case where the filenames in the image are not seen  
the same way the OS sees them - RISC OS uses unixy paths in Squeak  
because the ROS rules are so difficult to fit into the nasty habits  
people have of  assembling strings as a path and then forcing them  
down the primitives. The canonicaliser call gets a chance to swap the  
$/ to $. before passing to the OS.

So the idea was to provide an access point to this 'service' within  
the core that plugins could use at need. For use in the core itself,  
well I guess only image loading/saving is a possible user and that  
may or may not be useful. For example in RISC OS you wouldn't need it  
for the image name to load since it would already be guaranteed to  
have been canonicalised *unless*  you were to run from a commandline  
and deliberately pass in a dodgy string.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
When all else fails, let a = 7.  If that doesn't help, then read the  
manual.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Andreas.Raab
tim Rowledge wrote:

>> One question about this: Do we have a way of doing that kind of file
>> name conversion from the image itself? It strikes me as somewhat odd
>> that a plugin needs to do that conversion given that the image can
>> just pass a well-formed version of the string as the argument, no?
> Actually, no. Until and unless we ever get some decent file handling
> code in the image we simply can't rely on a passed in filename being
> even vaguely decent. My preference is to call a VM routine to
> canonicalise the passed in string for every usage simply because you can
> rely (for some value of 'rely') on the OS getting it right in working
> out that '../../foo/bar/bim.bmp' is actually
> '/usr/mounts/someOtherComputer/viaALink/mumble/frotz.jpeg'.

All right, you guys know this better than I do (since I don't have this
problem ;-) So, just for verification, the following is a legal
implementation for the method, yes?

int sqGetFilenameFromString(char *buf, char *fname, int length, int b) {
   memcpy(buf, fname, length);
   return 1;
}

> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> When all else fails, let a = 7.  If that doesn't help, then read the
> manual.

How fitting ;-) It was just how I felt when writing the above. Are you
sure this is random?

Cheers,
   - Andreas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

johnmci
I believe fname here and length is technically the smalltalk object,  
and it's length. If you copy over from the fname it may not have
a terminating 0x00.

buf[length] = 0x00

Also if you run the image as UTF-8 would copying the file name  
directly and assume you've get a file name that is usable by Windows,  
is that correct?

On 26-Mar-06, at 3:17 PM, Andreas Raab wrote:

> tim Rowledge wrote:
>>> One question about this: Do we have a way of doing that kind of  
>>> file name conversion from the image itself? It strikes me as  
>>> somewhat odd that a plugin needs to do that conversion given that  
>>> the image can just pass a well-formed version of the string as  
>>> the argument, no?
>> Actually, no. Until and unless we ever get some decent file  
>> handling code in the image we simply can't rely on a passed in  
>> filename being even vaguely decent. My preference is to call a VM  
>> routine to canonicalise the passed in string for every usage  
>> simply because you can rely (for some value of 'rely') on the OS  
>> getting it right in working out that '../../foo/bar/bim.bmp' is  
>> actually '/usr/mounts/someOtherComputer/viaALink/mumble/frotz.jpeg'.
>
> All right, you guys know this better than I do (since I don't have  
> this problem ;-) So, just for verification, the following is a  
> legal implementation for the method, yes?
>
> int sqGetFilenameFromString(char *buf, char *fname, int length, int  
> b) {
>   memcpy(buf, fname, length);
>   return 1;
> }
>
>> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
>> When all else fails, let a = 7.  If that doesn't help, then read  
>> the manual.
>
> How fitting ;-) It was just how I felt when writing the above. Are  
> you sure this is random?
>
> Cheers,
>   - Andreas

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Andreas.Raab
John M McIntosh wrote:
> I believe fname here and length is technically the smalltalk object, and
> it's length. If you copy over from the fname it may not have
> a terminating 0x00.

So. Where. Is. This. Used??? The comment in sq.h merely states:

   "New filename converting function; used by the interpreterProxy
function ioFilenamefromStringofLengthresolveAliases. Most platforms can
ignore the resolveAlias boolean - it seems to only be of use by OSX but
is crucial there."

and unfortunately, this doesn't give much guidance about arguments or
return values. Without an example (which I have asked for, remember?) it
is *impossible* to implement that function correctly based on the
information I have.

Cheers,
   - Andreas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

johnmci
The mpeg plugin uses it to take the file name and to open it, also  
the freetype 2 plugin where it's the path to the font file.

The api  then converts the given file name in VM platform character  
type (macroman/utf-8/latin-1) into a string that the host operating  
system can use to act upon using host operating system file calls.

Usage of it in the standard unix based VMs can be seen in the

Cross/sqFilePluginBasicPrims.c


sqInt sqFileOpen(SQFile *f, char* sqFileName, sqInt sqFileNameSize,  
sqInt writeFlag) {
        /* Opens the given file using the supplied sqFile structure
           to record its state. Fails with no side effects if f is
           already open. Files are always opened in binary mode;
           Squeak must take care of any line-end character mapping.
        */

        char cFileName[1001];

        /* don't open an already open file */
        if (sqFileValid(f)) return interpreterProxy->success(false);

        /* copy the file name into a null-terminated C string */
        if (sqFileNameSize > 1000) {
                return interpreterProxy->success(false);
        }
        interpreterProxy->ioFilenamefromStringofLengthresolveAliases
(cFileName, sqFileName, sqFileNameSize, true);
        if (writeFlag) {
                /* First try to open an existing file read/write: */
                setFile(f, fopen(cFileName, "r+b"));




On 26-Mar-06, at 4:02 PM, Andreas Raab wrote:

> John M McIntosh wrote:
>> I believe fname here and length is technically the smalltalk  
>> object, and it's length. If you copy over from the fname it may  
>> not have
>> a terminating 0x00.
>
> So. Where. Is. This. Used??? The comment in sq.h merely states:
>
>   "New filename converting function; used by the interpreterProxy  
> function ioFilenamefromStringofLengthresolveAliases. Most platforms  
> can ignore the resolveAlias boolean - it seems to only be of use by  
> OSX but is crucial there."
>
> and unfortunately, this doesn't give much guidance about arguments  
> or return values. Without an example (which I have asked for,  
> remember?) it is *impossible* to implement that function correctly  
> based on the information I have.
>
> Cheers,
>   - Andreas

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Andreas.Raab
John M McIntosh wrote:
> The mpeg plugin uses it to take the file name and to open it, also the
> freetype 2 plugin where it's the path to the font file.

Not in any version of the MPEG plugin that I have. Which reminds me: Did
you see my message about the linker error due to the undefined reference
to "sqFilenameFromStringOpen"? Could this be related?

> The api  then converts the given file name in VM platform character type
> (macroman/utf-8/latin-1) into a string that the host operating system
> can use to act upon using host operating system file calls.

Which, again, makes me want to ask why we actually burden the plugin
with figuring that out instead of giving it a filename that works?

> Usage of it in the standard unix based VMs can be seen in the
>
> Cross/sqFilePluginBasicPrims.c
>
>     /* copy the file name into a null-terminated C string */
>     if (sqFileNameSize > 1000) {
>         return interpreterProxy->success(false);
>     }
>     interpreterProxy->ioFilenamefromStringofLengthresolveAliases(cFileName,
> sqFileName, sqFileNameSize, true);

I see. If the other usages are the same then the one thing that's
missing from my implementation is a terminating zero character, yes?

And like David mentioned, the above is prone to buffer overflow attacks
whenever the VMs encoding is larger than the image's (which is easily
possible if you consider, e.g., latin1 -> utf8 conversion).

Cheers,
   - Andreas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

johnmci
On 26-Mar-06, at 4:35 PM, Andreas Raab wrote:

> John M McIntosh wrote:
>> The mpeg plugin uses it to take the file name and to open it, also  
>> the freetype 2 plugin where it's the path to the font file.
>
> Not in any version of the MPEG plugin that I have. Which reminds  
> me: Did you see my message about the linker error due to the  
> undefined reference to "sqFilenameFromStringOpen"? Could this be  
> related?

The mpeg plugin uses
sqFilenameFromStringOpen()

which I declare as

#include "sqVirtualMachine.h"

extern struct VirtualMachine * interpreterProxy;

void sqFilenameFromStringOpen(char *buffer,long fileIndex, long  
fileLength, int flag) {
        interpreterProxy->ioFilenamefromStringofLengthresolveAliases
(buffer,fileIndex, fileLength, flag);
}



>
>> The api  then converts the given file name in VM platform  
>> character type (macroman/utf-8/latin-1) into a string that the  
>> host operating system can use to act upon using host operating  
>> system file calls.
>
> Which, again, makes me want to ask why we actually burden the  
> plugin with figuring that out instead of giving it a filename that  
> works?

Well we need a primitive that would translate the image file string  
to the host api file name.
ioFilenamefromStringofLengthresolveAliases comes to mind.

Then pass in the translated results, however now it's an issue of  
backward/forward compatibility.


>
>> Usage of it in the standard unix based VMs can be seen in the
>> Cross/sqFilePluginBasicPrims.c
>>     /* copy the file name into a null-terminated C string */
>>     if (sqFileNameSize > 1000) {
>>         return interpreterProxy->success(false);
>>     }
>>     interpreterProxy->ioFilenamefromStringofLengthresolveAliases
>> (cFileName, sqFileName, sqFileNameSize, true);
>
> I see. If the other usages are the same then the one thing that's  
> missing from my implementation is a terminating zero character, yes?

Yes

>
> And like David mentioned, the above is prone to buffer overflow  
> attacks whenever the VMs encoding is larger than the image's (which  
> is easily possible if you consider, e.g., latin1 -> utf8 conversion).

Ya, likely we should pass in the max size of the target buffer.

>
> Cheers,
>   - Andreas

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Andreas.Raab
John M McIntosh wrote:

> The mpeg plugin uses
> sqFilenameFromStringOpen()
>
> which I declare as
>
> #include "sqVirtualMachine.h"
>
> extern struct VirtualMachine * interpreterProxy;
>
> void        sqFilenameFromStringOpen(char *buffer,long fileIndex, long
> fileLength, int flag) {
>     interpreterProxy->ioFilenamefromStringofLengthresolveAliases(buffer,fileIndex,
> fileLength, flag);
> }

So where is this declaration? I can't find it anywhere.

>>> The api  then converts the given file name in VM platform character
>>> type (macroman/utf-8/latin-1) into a string that the host operating
>>> system can use to act upon using host operating system file calls.
>>
>> Which, again, makes me want to ask why we actually burden the plugin
>> with figuring that out instead of giving it a filename that works?
>
> Well we need a primitive that would translate the image file string to
> the host api file name.
> ioFilenamefromStringofLengthresolveAliases comes to mind.

Well, my point is that if we expose that primitive to the image, we will
never have the need to do it from a plugin because you can always pass a
valid file name into it. It's really strange to me that the VM does
translation for stuff which is (supposedly) handled in the image
already. I could understand this if we would shoot for a file/directory
abstraction inside the image but the way it's right now we're hiding
stuff from the image that the image supposedly is responsible for
(platform file name handling) and that's Just Wrong (tm).

>> And like David mentioned, the above is prone to buffer overflow
>> attacks whenever the VMs encoding is larger than the image's (which is
>> easily possible if you consider, e.g., latin1 -> utf8 conversion).
>
> Ya, likely we should pass in the max size of the target buffer.

Why not just return "char*" and let the implementor figure out the size
of the buffer? On most systems that size is bounded (by MAX_PATH or
similar) which means that a "static char[MAX_PATH]" does the job just
fine on the implementors side.

Cheers,
   - Andreas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Yoshiki Ohshima
  Hello,

> Well, my point is that if we expose that primitive to the image, we will
> never have the need to do it from a plugin because you can always pass a
> valid file name into it. It's really strange to me that the VM does
> translation for stuff which is (supposedly) handled in the image
> already. I could understand this if we would shoot for a file/directory
> abstraction inside the image but the way it's right now we're hiding
> stuff from the image that the image supposedly is responsible for
> (platform file name handling) and that's Just Wrong (tm).

  Since Squeak3.8, FilePath class is supposed to do the conversion.  I
wonder if how ioFilename:fromString:ofLength:resolveAliases: interacts
with FilePath...

-- Yoshiki
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

johnmci
Ah, poorly.

 From what I've seen the in image encoding assumes latin-1. It gets  
macroman from the VM and mangles, demangles that ok, however
displays the characters then wrong in the file browser.  If you  
switch the VM to use latin-1 then characters are displayed correctly,  
however
it is possible to create file names that contains characters that  
UTF-8 allows but has not latin-1 mapping, thus the internal VM  
mapping from that character to latin-1 fails.

Keeping the VM in UTF-8 seems to work, of course FilePath mangles  
that as latin-1 and the names are not displayed correctly.


On 29-Mar-06, at 6:42 PM, Yoshiki Ohshima wrote:

>   Hello,
>
>> Well, my point is that if we expose that primitive to the image,  
>> we will
>> never have the need to do it from a plugin because you can always  
>> pass a
>> valid file name into it. It's really strange to me that the VM does
>> translation for stuff which is (supposedly) handled in the image
>> already. I could understand this if we would shoot for a file/
>> directory
>> abstraction inside the image but the way it's right now we're hiding
>> stuff from the image that the image supposedly is responsible for
>> (platform file name handling) and that's Just Wrong (tm).
>
>   Since Squeak3.8, FilePath class is supposed to do the conversion.  I
> wonder if how ioFilename:fromString:ofLength:resolveAliases: interacts
> with FilePath...
>
> -- Yoshiki

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Yoshiki Ohshima
  John,

  It may be done poorly.  But the idea is to switch the encoding in
the image by providing the right fileNameConverter.

  I'm not sure what you mean by "in image encoding assumes latin-1".
If the file browser doesn't show the right file name, the right fix is
to change the fileNameConverter, not changing the VM.

-- Yoshiki

At Wed, 29 Mar 2006 21:35:47 -0800,
John M McIntosh wrote:

>
> Ah, poorly.
>
>  From what I've seen the in image encoding assumes latin-1. It gets  
> macroman from the VM and mangles, demangles that ok, however
> displays the characters then wrong in the file browser.  If you  
> switch the VM to use latin-1 then characters are displayed correctly,  
> however
> it is possible to create file names that contains characters that  
> UTF-8 allows but has not latin-1 mapping, thus the internal VM  
> mapping from that character to latin-1 fails.
>
> Keeping the VM in UTF-8 seems to work, of course FilePath mangles  
> that as latin-1 and the names are not displayed correctly.
>
>
> On 29-Mar-06, at 6:42 PM, Yoshiki Ohshima wrote:
>
> >   Hello,
> >
> >> Well, my point is that if we expose that primitive to the image,  
> >> we will
> >> never have the need to do it from a plugin because you can always  
> >> pass a
> >> valid file name into it. It's really strange to me that the VM does
> >> translation for stuff which is (supposedly) handled in the image
> >> already. I could understand this if we would shoot for a file/
> >> directory
> >> abstraction inside the image but the way it's right now we're hiding
> >> stuff from the image that the image supposedly is responsible for
> >> (platform file name handling) and that's Just Wrong (tm).
> >
> >   Since Squeak3.8, FilePath class is supposed to do the conversion.  I
> > wonder if how ioFilename:fromString:ofLength:resolveAliases: interacts
> > with FilePath...
> >
> > -- Yoshiki
>
> --
> ========================================================================
> ===
> John M. McIntosh <[hidden email]> 1-800-477-2659
> Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
> ========================================================================
> ===
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

johnmci
Well the VM is given a info.plist entry or cmd line that defines what  
encoding to use. This information needs to be gotten from
the VM in order to set the proper filenameConverter. My understanding  
was that the default for the mac carbon vm was
MacRoman, but the fileNameConverter used in 3.8? is latin-1. If I  
choose latin-1 there are characters in the utf-8 character set  
allowed in
os-x that can't be translated to latin-1 so latin-1 technically is  
not a viable choice. I understand UTF-8 is viable, but somehow the  
image should configure that from the Vm without manual intervention.


On 1-Apr-06, at 6:38 PM, Yoshiki Ohshima wrote:

>   John,
>
>   It may be done poorly.  But the idea is to switch the encoding in
> the image by providing the right fileNameConverter.
>
>   I'm not sure what you mean by "in image encoding assumes latin-1".
> If the file browser doesn't show the right file name, the right fix is
> to change the fileNameConverter, not changing the VM.
>
> -- Yoshiki
>
> At Wed, 29 Mar 2006 21:35:47 -0800,
> John M McIntosh wrote:
>>
>> Ah, poorly.
>>
>>  From what I've seen the in image encoding assumes latin-1. It gets
>> macroman from the VM and mangles, demangles that ok, however
>> displays the characters then wrong in the file browser.  If you
>> switch the VM to use latin-1 then characters are displayed correctly,
>> however
>> it is possible to create file names that contains characters that
>> UTF-8 allows but has not latin-1 mapping, thus the internal VM
>> mapping from that character to latin-1 fails.
>>
>> Keeping the VM in UTF-8 seems to work, of course FilePath mangles
>> that as latin-1 and the names are not displayed correctly.
>>
>>
>> On 29-Mar-06, at 6:42 PM, Yoshiki Ohshima wrote:
>>
>>>   Hello,
>>>
>>>> Well, my point is that if we expose that primitive to the image,
>>>> we will
>>>> never have the need to do it from a plugin because you can always
>>>> pass a
>>>> valid file name into it. It's really strange to me that the VM does
>>>> translation for stuff which is (supposedly) handled in the image
>>>> already. I could understand this if we would shoot for a file/
>>>> directory
>>>> abstraction inside the image but the way it's right now we're  
>>>> hiding
>>>> stuff from the image that the image supposedly is responsible for
>>>> (platform file name handling) and that's Just Wrong (tm).
>>>
>>>   Since Squeak3.8, FilePath class is supposed to do the  
>>> conversion.  I
>>> wonder if how ioFilename:fromString:ofLength:resolveAliases:  
>>> interacts
>>> with FilePath...
>>>
>>> -- Yoshiki
>>
>> --
>> =====================================================================
>> ===
>> ===
>> John M. McIntosh <[hidden email]> 1-800-477-2659
>> Corporate Smalltalk Consulting Ltd.  http://
>> www.smalltalkconsulting.com
>> =====================================================================
>> ===
>> ===
>>

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Bert Freudenberg-3
In reply to this post by Yoshiki Ohshima
The image defaults to Latin1Environment on Mac, but John's VM did not  
support Latin1 until very recently. It does support UTF-8, however,  
there is no UTF8Environment that could be set as default.

A hack that I used was to manually set the filename converter class  
to  UTF-8 after startup, but that's cumbersome.

- Bert -

Am 02.04.2006 um 04:38 schrieb Yoshiki Ohshima:

>   John,
>
>   It may be done poorly.  But the idea is to switch the encoding in
> the image by providing the right fileNameConverter.
>
>   I'm not sure what you mean by "in image encoding assumes latin-1".
> If the file browser doesn't show the right file name, the right fix is
> to change the fileNameConverter, not changing the VM.
>
> -- Yoshiki
>
> At Wed, 29 Mar 2006 21:35:47 -0800,
> John M McIntosh wrote:
>>
>> Ah, poorly.
>>
>>  From what I've seen the in image encoding assumes latin-1. It gets
>> macroman from the VM and mangles, demangles that ok, however
>> displays the characters then wrong in the file browser.  If you
>> switch the VM to use latin-1 then characters are displayed correctly,
>> however
>> it is possible to create file names that contains characters that
>> UTF-8 allows but has not latin-1 mapping, thus the internal VM
>> mapping from that character to latin-1 fails.
>>
>> Keeping the VM in UTF-8 seems to work, of course FilePath mangles
>> that as latin-1 and the names are not displayed correctly.
>>
>>
>> On 29-Mar-06, at 6:42 PM, Yoshiki Ohshima wrote:
>>
>>>   Hello,
>>>
>>>> Well, my point is that if we expose that primitive to the image,
>>>> we will
>>>> never have the need to do it from a plugin because you can always
>>>> pass a
>>>> valid file name into it. It's really strange to me that the VM does
>>>> translation for stuff which is (supposedly) handled in the image
>>>> already. I could understand this if we would shoot for a file/
>>>> directory
>>>> abstraction inside the image but the way it's right now we're  
>>>> hiding
>>>> stuff from the image that the image supposedly is responsible for
>>>> (platform file name handling) and that's Just Wrong (tm).
>>>
>>>   Since Squeak3.8, FilePath class is supposed to do the  
>>> conversion.  I
>>> wonder if how ioFilename:fromString:ofLength:resolveAliases:  
>>> interacts
>>> with FilePath...
>>>
>>> -- Yoshiki
>>
>> --
>> =====================================================================
>> ===
>> ===
>> John M. McIntosh <[hidden email]> 1-800-477-2659
>> Corporate Smalltalk Consulting Ltd.  http://
>> www.smalltalkconsulting.com
>> =====================================================================
>> ===
>> ===
>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: InterpreterProxy>>ioFilename:fromString:ofLength:resolveAliases:?

Yoshiki Ohshima
In reply to this post by johnmci
  John,

> Well the VM is given a info.plist entry or cmd line that defines what  
> encoding to use. This information needs to be gotten from
> the VM in order to set the proper filenameConverter. My understanding  
> was that the default for the mac carbon vm was
> MacRoman, but the fileNameConverter used in 3.8? is latin-1. If I  
> choose latin-1 there are characters in the utf-8 character set  
> allowed in
> os-x that can't be translated to latin-1 so latin-1 technically is  
> not a viable choice. I understand UTF-8 is viable, but somehow the  
> image should configure that from the Vm without manual intervention.

  Note that the fileNameConverter to use is decided when the image is
started.  It can check the platform, vm version, etc., etc. to decide
which one to use.  An example is shown JapaneseEnvironment
class>>systemConverterClass.  In the other words, the image stuff can
be configured.

-- Yoshiki
12
Loading...