Smalltalk › Squeak › Squeak - Dev

CrLfFileStream on MacOSX

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

9 messages Options

Prof. Andrew P. Black

CrLfFileStream on MacOSX

I have noticed that CrLfFileStream class >>
guessDefaultLineEndConvention does not work on my MacOSX system

DirectoryClass (a class variable, defined in class FileDirectory) is
UnixFileDirectory on my system, which seems reasonable — but that
class hard-codes the path separator to be /, which
guessDefaultLineEndConvention uses to guess that LF is the right line
ending convention.

This explains, perhaps, why LF characters are increasingly polluting
the sources and changes files.

Andrew

Prof. Andrew P. Black

Re: CrLfFileStream on MacOSX

On 13 Feb 2007, at 17:17, Andrew P. Black wrote:

> I have noticed that CrLfFileStream class >>
> guessDefaultLineEndConvention does not work on my MacOSX system
>
> DirectoryClass (a class variable, defined in class FileDirectory)
> is UnixFileDirectory on my system, which seems reasonable — but
> that class hard-codes the path separator to be /, which
> guessDefaultLineEndConvention uses to guess that LF is the right
> line ending convention.
>
> This explains, perhaps, why LF characters are increasingly
> polluting the sources and changes files.

I deathly silence followed this post, which lead me to suspect that I
had said something very foolish. However, I haven't been able to
figure out what. Perhaps if I continue to prattle on, someone will
tell me.

The value of LineEndDefault seems to matter only when creating a new
file; when reading a file, or over-writing a file, the existing
content of the file governs whether the cr message writes LF or CR,
and what line endings cause next to answer CR.

However, when it matters, it matters. I can't be the only MacOSX
user to notice that writing a text file fills it with LFs!

The fix seems to be simple: check to see if we are running on Darwin.

CrLfFileStream class >> guessDefaultLineEndConvention
"Lets try to guess the line end convention from what we know about the
path name delimiter from FileDirectory and the identity of the OS."
FileDirectory pathNameDelimiter = $:
ifTrue: [^ self defaultToCR].
FileDirectory pathNameDelimiter = $/
ifTrue: [((SmalltalkImage current getSystemAttribute: 1002)
beginsWith: 'darwin')
ifTrue: [^ self defaultToCR]
ifFalse: [^ self defaultToLF]].
FileDirectory pathNameDelimiter = $\
ifTrue: [^ self defaultToCRLF].
"in case we don't know"
^ self defaultToCR

Shall I submit this as a bugfix?

Andrew

keith1y

Re: CrLfFileStream on MacOSX

In reply to this post by Prof. Andrew P. Black

I dont think that it is as simple as that, have a look at CrLfFileStream
new. In 3.9 this invokes MultiByteFileStream instead. Perhaps the
problem lies there instead.

I recently submitted a bug report to mantis 6086 and a (fix/workaround)
because I discovered that it was not possible to open a CrLfFileStream
(aka a MultiByteFileStream) on '/dev/stdout'. It is unable to
#detectLineEndConvention.

Keith

Andrew P. Black wrote:
>
> On 13 Feb 2007, at 17:17, Andrew P. Black wrote:
>
>> I have noticed that CrLfFileStream class >>
>> guessDefaultLineEndConvention does not work on my MacOSX system
>>

Inbox full of unwanted email? Get leading protection and 1GB storage with All New Yahoo! Mail.

Lex Spoon-3

Re: CrLfFileStream on MacOSX

Keith Hodges <[hidden email]> writes:
> I dont think that it is as simple as that, have a look at CrLfFileStream
> new. In 3.9 this invokes MultiByteFileStream instead. Perhaps the
> problem lies there instead.
>
> I recently submitted a bug report to mantis 6086 and a (fix/workaround)
> because I discovered that it was not possible to open a CrLfFileStream
> (aka a MultiByteFileStream) on '/dev/stdout'. It is unable to
> #detectLineEndConvention.

By the way, what ever happened to Andreas's TextFile class? I was
a long user of CrLfFileStream, until Andreas posted TextFile, and
I remember liking TextFile even better.

Have any of you CrLfFileStream fans tried TextFile? If so, can you
compare the two?

-Lex

Prof. Andrew P. Black

Re: CrLfFileStream on MacOSX

In reply to this post by keith1y

On 16 Feb 2007, at 1:06, Keith Hodges wrote:

I don't think that it is as simple as that, have a look at CrLfFileStream
new. In 3.9 this invokes MultiByteFileStream instead. Perhaps the
problem lies there instead.

I'm by no means an expert on MultiByteFileStreams, but I don't think that this makes any difference to the problem that I'm highlighting.

MultiByteFileStream has its own version of detectLineEndConvention, but it's clearly been copied from CrLfFileStream (and then some saves and restores of the converter have been added, in all (?) of the right places). Consequently the role played by LineEndDefault is the same, and the fact that OSX systems get this set wrong on every startup has the same impact. The fix that I posed should not adversely affect MultiByteFileStreams.

Related but tangential:

I believe that both of these detectLineEndConvention methods should be using ensure: to do the resetting of the stream state, and then they could be refactored into one. I also agree that TextFile makes more sense than CrLfFileStream. The latter is not part of the standard, and could be deprecated, and implemented as a TextFile. Allowing a CrLfFileStream to be a binary stream seems bogus, but maybe there is a reason.

Andrew P. Black

Department of Computer Science

Portland State University

+1 503 725 2411

Rik Smoody

Re: CrLfFileStream on MacOSX

In reply to this post by Prof. Andrew P. Black

I stumbled upon (probably) the same issue.
Evidence, from an old image, but running it on newer VM:

Performed by Squeak 3.8.15beta1U.app, on a Mac OS X 10.3.9:
FileDirectory allSubclasses collect: [:class | class->class
primPathNameDelimiter ]
a Set(MacHFSPlusFileDirectory->$/ AcornFileDirectory->$/
UnixFileDirectory->$/ MacFileDirectory->$/ DosFileDirectory->$/)

The answer from Squeak 3.8.9beta8.app, on the same Mac
a Set(MacHFSPlusFileDirectory->$: AcornFileDirectory->$:
UnixFileDirectory->$: MacFileDirectory->$: DosFileDirectory->$:)

The old-version answer is somewhat more compatible with the Mac.

For example, when running one old utility image, I got this message:
/Tomato:Users:Rik:Pictures does not exist
the problem apparently caused by that leading $/
as changing that manually back to a $: reveals the expected existence
of the file.

On 2007 Feb 15, at 3:41 PM, Andrew P. Black wrote:

>
> On 13 Feb 2007, at 17:17, Andrew P. Black wrote:
>
>> I have noticed that CrLfFileStream class >>
>> guessDefaultLineEndConvention does not work on my MacOSX system
>>
>> DirectoryClass (a class variable, defined in class FileDirectory) is
>> UnixFileDirectory on my system, which seems reasonable — but that
>> class hard-codes the path separator to be /, which
>> guessDefaultLineEndConvention uses to guess that LF is the right line
>> ending convention.
>>
>> This explains, perhaps, why LF characters are increasingly polluting
>> the sources and changes files.
>
> I deathly silence followed this post, which lead me to suspect that I
> had said something very foolish. However, I haven't been able to
> figure out what. Perhaps if I continue to prattle on, someone will
> tell me.
>
> The value of LineEndDefault seems to matter only when creating a new
> file; when reading a file, or over-writing a file, the existing
> content of the file governs whether the cr message writes LF or CR,
> and what line endings cause next to answer CR.
>
> However, when it matters, it matters. I can't be the only MacOSX user
> to notice that writing a text file fills it with LFs!
>
> The fix seems to be simple: check to see if we are running on Darwin.
>
> CrLfFileStream class >> guessDefaultLineEndConvention
> "Lets try to guess the line end convention from what we know about the
> path name delimiter from FileDirectory and the identity of the OS."
> FileDirectory pathNameDelimiter = $:
> ifTrue: [^ self defaultToCR].
> FileDirectory pathNameDelimiter = $/
> ifTrue: [((SmalltalkImage current getSystemAttribute: 1002)
> beginsWith: 'darwin')
> ifTrue: [^ self defaultToCR]
> ifFalse: [^ self defaultToLF]].
> FileDirectory pathNameDelimiter = $\
> ifTrue: [^ self defaultToCRLF].
> "in case we don't know"
> ^ self defaultToCR
>
>
> Shall I submit this as a bugfix?
>
> Andrew
>
>
>
>

Rik Smoody [hidden email]
smOOdynamics - Object-Orientation is the middle of our name
Portland, OR 503-249-8300

Prof. Andrew P. Black

Re: CrLfFileStream on MacOSX

Hi Rick,

If I interpret what you are saying correctly, the meaning of the primPathNameDelimiter method, which is given by

<primitive: 'primitiveDirectoryDelimitor' module: 'FilePlugin'>

has changed in more recent Mac VMs (I'm running 3.8.12Beta1U). It was $:, but is now $/

This may well be a bug! I wonder if someone could explain why there are platform-specific directory classes and what the "right" one should be under OSX?

The logic for choosing the appropriate directory class used primNameDelimiter, e.g.,

MacHFSPlusFileDirectory >> isActiveDirectoryClass

"Ok, lets see if we support HFS Plus file names, the long ones"

^ self pathNameDelimiter = self primPathNameDelimiter

and: [(SmalltalkImage current getSystemAttribute: 1201) notNil

and: [(SmalltalkImage current getSystemAttribute: 1201) asNumber > 31]]

MacFileDirectory>>pathNameDelimiter

^ $:

So, by changing the meaning of the primitive 'primitiveDirectoryDelimitor', the directory class that one gets on OSX has been changed too.

Again, what is the "correct" behavior? MacHFSPlusFileDirectory doesn't seem to work on my Mac, even after changing the default delimiter: maybe that is why the primitive was changed?

Andrew

P.S.

While poking around I found:

FileStream>>sourceFileSuffixes

^ {FileStream st. FileStream cs. FileStream multiSt. FileStream multiCs} asSet asArray

Is there a reason for the asSet asArray? To make a copy? To randomize the order? None of the senders seems to care ...

APB

On 16 Feb 2007, at 17:26, Rik Smoody wrote:

I stumbled upon (probably) the same issue.

Evidence, from an old image, but running it on newer VM:

Performed by Squeak 3.8.15beta1U.app, on a Mac OS X 10.3.9:

FileDirectory allSubclasses collect: [:class | class->class primPathNameDelimiter ]

a Set(MacHFSPlusFileDirectory->$/ AcornFileDirectory->$/ UnixFileDirectory->$/ MacFileDirectory->$/ DosFileDirectory->$/)

The answer from Squeak 3.8.9beta8.app, on the same Mac

a Set(MacHFSPlusFileDirectory->$: AcornFileDirectory->$: UnixFileDirectory->$: MacFileDirectory->$: DosFileDirectory->$:)

The old-version answer is somewhat more compatible with the Mac.

For example, when running one old utility image, I got this message:

/Tomato:Users:Rik:Pictures does not exist

the problem apparently caused by that leading $/

as changing that manually back to a $: reveals the expected existence of the file.

Andrew P. Black

Department of Computer Science

Portland State University

+1 503 725 2411

johnmci

Re: CrLfFileStream on MacOSX

On Feb 17, 2007, at 12:24 AM, Andrew P. Black wrote:

> Hi Rick,
>
> If I interpret what you are saying correctly, the meaning of the
> primPathNameDelimiter method, which is given by
>
> <primitive: 'primitiveDirectoryDelimitor' module: 'FilePlugin'>
>
> has changed in more recent Mac VMs (I'm running 3.8.12Beta1U). It
> was $:, but is now $/

Well I would have to dig about in a readme or three, however
historically the Mac VM started life
under os 7.x (or earlier) That means it lived with HFS 31? (was it)
file names those with ":"

Then HFS+ came along and introduced 255 byte filenames. so the VM
was over time migrated
to this which required the vm parm of 1201 to figure out if 31 or 255
was valid file name size.

And somewhere in there I drag the other vm developers into supporting
64bit file pointers.

Then OS-X came alone and we limped along with HFS+ for awhile until
people wanted to do
strange things like open /dev/foobar. Which wasn't possible.

The killer of course was lots of the API calls we were using for HFS+
were not ported to MacIntel
So I migrated to the unix file name code which I copied and altered
to provide proper HFS+ alias support.
The original unix code was quite weak in this area and assumed the
apple api's would handle multiple level aliases which it does not.

http://www.squeakvm.com/cgi-bin/viewcvs.cgi/trunk/platforms/Mac%20OS/
plugins/FilePlugin/sqMacUnixFileInterface.c?rev=1301&view=log

Ah interestingly enough the first check in was 1 year and 2 weeks ago.
Again thanks to Impara for providing a machine so I could do that
chore. (non-trivial btw).

So if you are running on a macintosh, well it's a unix machine, want
macintosh like cr/lf patterns well
you need to need deeper.

--
========================================================================
===
John M. McIntosh <[hidden email]>
Corporate Smalltalk Consulting Ltd. http://www.smalltalkconsulting.com
========================================================================
===

Prof. Andrew P. Black

Re: CrLfFileStream on MacOSX

Thanks, John, for this clarification.

On 17 Feb 2007, at 02:33, John M McIntosh wrote:

>
> So if you are running on a macintosh, well it's a unix machine,
> want macintosh like cr/lf patterns well
> you need to need deeper.
>

Given this and the previous discussion, I believe that my original
fix (looking to see if the OS is darwin) is the right one, but that
it should be applied to MultiByteFIleStream. CrLfFileStream is in
fact dead, and ought to be buried except for a compatibility stub,
but that is different topic.

I've created a Mantis report on this issue and the fix (number
6173) , and invite interested parties to kibitz further there.

http://bugs.impara.de/view.php?id=6173

Andrew