3.9 Oddities

I'm trying to move from 3.8 to 3.9, and I have a couple of odd problems.

In text areas, there are often odd boxes at the beginning of lines.  
These do not show up in 3.8. Any idea how I can get rid of them?

When I try to change the window colors, I get a ton of Error: key not  
found messages. I don't really need to change the window colors, but  
having errors on a built-in feature is a bit disturbing.

Thanks,

-Rich-

On 4 sept. 06, at 11:52, Rich Warren wrote:

> I'm trying to move from 3.8 to 3.9, and I have a couple of odd  
> problems.
>
> In text areas, there are often odd boxes at the beginning of lines.  
> These do not show up in 3.8. Any idea how I can get rid of them?

The "odd boxes" are bad characters that before were displayed as  
character space.
And it helps showing when you are copying and pasting code from web  
page and other foreign source.
Often I could not find the bugs and rewrite all the code that the  
students copied from pdf for example.

> When I try to change the window colors, I get a ton of Error: key  
> not found messages. I don't really need to change the window  
> colors, but having errors on a built-in feature is a bit disturbing.

Indeed.
Add a bug fix in the bug archive.

>
> Thanks,
>
> -Rich-
>

On Sep 4, 2006, at 4:03 AM, stéphane ducasse wrote:

They seem so prevalent. I though they might be a difference in EOL  
symbols between Win/Mac/Unix. Does squeak use a standard end of line  
character? Or does it vary based on OS. Is there a way you can adjust  
this setting, or automatically convert documents?

More to the point, why do they show up in 3.9 but not in 3.8?

-Rich-

On 5 sept. 06, at 07:19, Rich Warren wrote:

because before there was no glyph for them.

Stef
>
> -Rich-
>

On Sep 4, 2006, at 9:12 PM, stéphane ducasse wrote:

I'm sorry, but this really doesn't feel like a satisfactory answer.

I did some digging. It turns out the problem is line feeds. In the  
good sample I looked at, Squeak was using CR (hex 0D) to represent  
the end of a line. In the annoying box example, it used CR LF (hex 0D  
0A).

Now, if I remember correctly, the first is the standard ASCII newline  
for Unix. The second is the standard ASCII newline for DOS/Windows.  
They're both standard EOL markers on their respective platforms (both  
platforms Squeak supports).

Here's my point. As a cross-platform editor, Squeak must be able to  
handle these transparently. Either it needs to automatically  
normalize everything to a single Squeak-standard newline, or it needs  
to accept both of these (and others, old Mac os used a third newline  
variant, and there may be some I'm not familiar with) in a reasonable  
and transparent way.

I should be able to open any ascii text file (regardless of which OS  
it was written on) and it should appear as it was intended--
regardless of the particular newline encoding.

Displaying the box glyphs is possibly good in some cases (for  
example, non-ascii codes that may inadvertently get copied from  
pdfs). A better solution would be to strip out any invalid characters  
automatically. After all, if they're invalid, they can't be doing  
anything constructive.

But displaying the box glyphs for standard ASCII DOS/WIN newlines  
feels like a big step backwards.

-Rich-

On Sep 4, 2006, at 4:03 AM, stéphane ducasse wrote:
>
>> When I try to change the window colors, I get a ton of Error: key  
>> not found messages. I don't really need to change the window  
>> colors, but having errors on a built-in feature is a bit disturbing.
>
> Indeed.
> Add a bug fix in the bug archive.

While I would hope that someone working on 3.9 would notice that I  
brought up the issue here and take the initiative to check it out, I  
tried to go to the squeak site and clicked on the bug reporting link.  
Unfortunately, I did not find any obvious way to add a bug. I guess I  
have to register for an account (thought that's never explicitly  
stated anywhere). Regardless, I don't have the time or energy to deal  
with it tonight. Maybe I'll get around to it later...

-Rich-

On 9/5/06, Rich Warren <[hidden email]> wrote:
> Now, if I remember correctly, the first is the standard ASCII newline

Behold the allmighty wikipedia: http://en.wikipedia.org/wiki/Newline

> Here's my point. As a cross-platform editor, Squeak must be able to
> handle these transparently. Either it needs to automatically
> normalize everything to a single Squeak-standard newline, or it needs
> to accept both of these

Both approaches (accepting the 3 conventions *and* normalizing) would
be best IMHO. Maybe it should normalize only when the text gets
modified in another way, so as not to 'gratuitously' modify stuff.

--
 Damien Pollet
 type less, do more

Rich Warren schrieb:
> But displaying the box glyphs for standard ASCII DOS/WIN newlines feels
> like a big step backwards.

It might feel like this, but it is actually a big step forward.
Previously, such bad line endings went unnoticed, because you wouldn't
see the bad characters. Now, when you open a text file you immediately
notice it's wrong, and you can fix it.

IMHO the best thing to do would just to fix the methods that have line
feeds in them now.

- Bert -

On Tue, 05 Sep 2006 11:42:04 +0200, Rich Warren wrote:
...
> Here's my point.

I do understand your point. But please don't take the following as an  
offense.

> As a cross-platform editor, Squeak must be able to handle these  
> transparently.

Squeak is not the editor (Java is not the editor. Algol is (was :) not the  
editor). Mind to talk about the tools you critisize. Not every tool is  
newLine sensitive...

> Either it needs to automatically normalize everything to a single  
> Squeak-standard newline, or it needs to accept both of these (and  
> others, old Mac os used a third newline variant, and there may be some  
> I'm not familiar with) in a reasonable and transparent way.
>
> I should be able to open any ascii text file (regardless of which OS it  
> was written on) and it should appear as it was intended--regardless of  
> the particular newline encoding.

Yes, almost everybody complains, see for example

- http://www.google.com/search?q=newline+confusion

I do not believe that there is a simple solution (because then all these  
complaints would disappear immediately). I believe that a COMPUTER cannot  
"know" what I want do do (or not to do) with newLines. Perhaps I want to  
write newLine-compliant text on platform A for use on platform B, who  
knows.

> Displaying the box glyphs is possibly good in some cases (for example,  
> non-ascii codes that may inadvertently get copied from pdfs). A better  
> solution would be to strip out any invalid characters automatically.  
> After all, if they're invalid, they can't be doing anything constructive.

The parser could do that when accepting source code, for sure. But please  
do not do anything automagically when I just review source code. I would  
like to see *all* *bugs* *in* *their* *original* *form*.

> But displaying the box glyphs for standard ASCII DOS/WIN newlines feels  
> like a big step backwards.

AFAIK there are not so many in the 3.9 Squeak .sources and .changes files  
(examples are classes VersionNumber and VersionHistory).

/Klaus

> -Rich-
>

Yeap!
And for me this is really important to see these boxes!
in 3.10!

On Mon, Sep 04, 2006 at 07:19:16PM -1000, Rich Warren wrote:
>
> They seem so prevalent. I though they might be a difference in EOL  
> symbols between Win/Mac/Unix. Does squeak use a standard end of line  
> character? Or does it vary based on OS. Is there a way you can adjust  
> this setting, or automatically convert documents?

There is no such thing as a "standard end of line character". Certain
operating systems and file systems have adopted the convention of
treating <cr><lf>, <cr>, or <lf> as deliminators for file records,
while many other file systems have an actual concept of "record"
that does not require delimiters. Either way, there is no such thing
as a standard.

The convention (nothing more, nothing less) for Smalltalk systems
is to treat <cr> as a line terminator. This is "incompatible" with
Micosoft conventions in the same sense that (for example) Microsoft
line end conventions are "incompatible" with Unix systems and vice
versa.

All of these conventions date back to teletype hardware. The <cr>
means move the carriage to the full left stop position (hence position
it at a "new line"). The <lf> means move the platen one line position,
but don't move the carriage back to the full left position. This also
can reasonably be considered to be a "new line". And of course the
combination of <cr><lf> moves the tty carriage back to the full
left margin position, then advances the platen to the next line.

If you send <cr><lf> to a real teletype, it really does move to
the next "record" on the piece of paper. But if you were trying
to conserve bytes in a data file, you might quite reasonably make
the case that either <cr> or <lf> by themselves should be taken
as a "new line" indicator.

The good folks who invented Unix decided to use <lf>, and the
equally good folks who invented Smalltalk decided to use <cr>.
The folks who adopted the Microsoft <cr><lf> convention stole
the idea from CP/M or some such thing, which was probably stolen
from RMX-11. But regardless of who stole what from whom, none of
these conventions are in any way standard. They are conventions,
no more, no less.

Dave

p.s. The above pertains to ASCII based file conventions. EBCDIC
file systems have generally been record oriented from the get-go,
and do not seem to engender the sort of confusion that is typical
for ASCII based file systems.

On Sep 5, 2006, at 3:22 AM, Klaus D. Witzel wrote:

> On Tue, 05 Sep 2006 11:42:04 +0200, Rich Warren wrote:
> ...
>> Here's my point.
>
> I do understand your point. But please don't take the following as  
> an offense.

No offense taken.

>
>> As a cross-platform editor, Squeak must be able to handle these  
>> transparently.
>
> Squeak is not the editor (Java is not the editor. Algol is (was :)  
> not the editor). Mind to talk about the tools you critisize. Not  
> every tool is newLine sensitive...

In java I agree. In Squeak, I have to disagree. When I browse classes  
in the browser, I can also edit them. The browser is an editor. When  
I open text files from a file list, I can edit them. The file list is  
an editor. Squeak is a combination programming language/IDE/
multimedia playground. That's one of the things I love about it. One  
of its many roles is "text editor."

There are very few newline combinations. While you may not be able to  
"know" in the absolute sense, you can make a reasonable assumption  
that would work properly for 99.999% of the cases.

This is actually a solved problem. I have many text editors on my  
system (BBedit and TextMate to name two) that can open text files  
which use any of the common newline encodings. They display the text  
in a reasonable manner, regardless of the encoding. At least in the  
case of BBedit (though I believe others as well), they save the file  
in the original encoding, unless you actually ask it to change the  
line endings.

If other systems can handle this issue seamlessly, why can't we? 3.8  
handled it seamlessly? Why can't we distinguish between non-text  
ascii characters that cause problems for the systems (bugs) and  
display them as glyphs, and alternate whitespace encoding which has  
no effect on the system, which gets handled in the 3.8 manner?


Are alternate line endings bugs? If they are bugs, then the code  
shouldn't run and someone should fix them. Balloon3d has tonnes of  
line-ending boxes, but I can run 3d code fine. If they're not bugs,  
then the system should accept them and display them in a reasonable  
manner.

Maybe there should be a tool that goes through the codebase and  
strips all unknown glyphs from the text. I don't know. But there are  
a large number of classes in my system that are littered with  
annoying boxes (all in the original image or donwloaded from SqueakMap).

-Rich-

On Wed, Sep 06, 2006 at 09:09:27PM -1000, Rich Warren wrote:
Rich,

You may not be aware of CrLfFileStream, which has been part of Squeak
for many years. Setting CrLfFileStream as your default file stream
will do pretty much exactly what you are describing. If you want to
use it, change FileStream class>>concreteStream to answer CrLfFileStream.

Nowadays, Squeak uses MultiByteFileStream as its default, so changing back
to to CrLfFileStream might have some bad side effects (I don't know,
but I'm mentioning it as a caution). It certainly won't work for Squeak's
multilingual support, but this may not be important for you if you
mainly work in English.

Many people, myself included, prefer not to have Squeak do the automatic
line end conversion. But it's largely a matter of personal preference,
so give CrLfFileStream a try if you prefer that approach.

Dave
 

On Sep 6, 2006, at 11:26 PM, David T. Lewis wrote:

> On Wed, Sep 06, 2006 at 09:09:27PM -1000, Rich Warren wrote:
>
> Many people, myself included, prefer not to have Squeak do the  
> automatic
> line end conversion. But it's largely a matter of personal preference,
> so give CrLfFileStream a try if you prefer that approach.

This is what I'm struggling to understand. What possible benefit is  
there in not converting line ends? Can you give me an example where  
converting the line ends would cause a problem?

Here's my view. Back in the dark ages, moving text files from one  
system to another was a nightmare. There are a number of small unix  
apps to deal with this very issue. Now, however, transferring text  
files has become transparent. Modern text editors just deal with the  
details. If they can handle multibyte strings and various common  
newline characters (or character combinations) all at the same time,  
why can't we?

And one more question (ok, series of related questions), if everyone  
else is completely set against changing the way Squeak handles  
newlines, what are we going to do with all the existing code that is  
littered with these ugly glyphs? Surely we're not going to just leave  
them the way they are?

How are we going to handle viewing and editing text documents--
particularly text documents that come from Windows systems? Is there  
a benefit to displaying the text in a way other than what the author  
intended? Or are we going to force people to deal with these  
documents outside the Squeak environment?

What happens if I want to convert an existing web site to seaside? Am  
I going to be forced to convert all my html documents before I can  
copy the parts I want and paste them into my seaside application?

I agree that we should leave alone any strange characters that have  
potentially ambiguous meanings. Displaying them as a default glyph  
seems reasonable. But, as far as I can tell, the common end of line  
encodings do not fall into this category.

I'm sorry if I came across too strong earlier. However, this whole  
issue just seems ridiculous to me. If there's a good reason for it,  
fine. I'd love to hear it. But, if we're creating problems in a large  
number of cases just to solve a few odd corner cases (or worse yet,  
creating problems with no benefit whatsoever), then I think its a bad  
trade.

-Rich-

On Thu, Sep 07, 2006 at 12:11:34AM -1000, Rich Warren wrote:
I didn't say there was a problem. I said that this is my own
personal preference. Which it is.

As for an example, if I want to look at a file and see if it
should have <lf> removed, it's harder to do if the tools disguise
the line terminators. As a matter of fact, many of the junk characters
that you are seeing in Squeak crept in precisely because of people
using automatic line conversion, and not noticing that the real
contents of some file was incorrectly formatted. So I guess that
you could count that as a problem. But really, it's just a matter
of preference, so try using CrLfFileStream if you prefer that approach.

> Here's my view. Back in the dark ages, moving text files from one  
> system to another was a nightmare.

I guess if we were out of the dark ages, we would not still be
using files.

Dave

Exactly

> And one more question (ok, series of related questions), if  
> everyone else is completely set against changing the way Squeak  
> handles newlines, what are we going to do with all the existing  
> code that is littered with these ugly glyphs? Surely we're not  
> going to just leave them the way they are?

who is we?
We tried but we got some problems because some packages cannot be  
reloaded.

> How are we going to handle viewing and editing text documents--
> particularly text documents that come from Windows systems? Is  
> there a benefit to displaying the text in a way other than what the  
> author intended? Or are we going to force people to deal with these  
> documents outside the Squeak environment?

Better importer/exporters should do the work but someone has to code  
them.
>
> What happens if I want to convert an existing web site to seaside?  
> Am I going to be forced to convert all my html documents before I  
> can copy the parts I want and paste them into my seaside application?

Have you tried?
I guess that only css styles should be converted.

This is not that simple.
We are on mac, pc, linux, unix, risc os.......

Stef

On Sep 7, 2006, at 11:58 AM, stéphane ducasse wrote:

Yes, I really think it is that simple. No one has given me any  
example to the contrary.

The text editors I'm talking about may be single-platform, but they  
can edit documents produced on any environment (mac, pc, linux, unix,  
risc os, etc.). They open the documents without problem. They display  
the documents correctly (and here, I define correctly as "as the  
original author intended". If the original author put a newline (of  
whatever variety) the editor displays a newline).

I don't see why displaying newlines is a problem.

-Rich-

On Sep 7, 2006, at 11:18 AM, David T. Lewis wrote:


OK, here's a followup question. If there's no problem with <lf>, then  
why do we need to remove them? Does that justify uglifying the  
existing code base? It seems like way 3.9 handles newlines creates a  
lot more problems than it solves.

How often is removing <lf> an issue. Could displaying/hiding <lf> be  
a feature that is toggled on and off? Couldn't <lf> be automatically  
removed? This really doesn't seem like something humans should be  
wasting their time doing. This is something that should be automated.

How did the <lf>'s get into the code base to begin with? Can't they  
be blocked at the entry point?  (of course, that doesn't deal with  
the existing problems--but it seems odd that Squeak would allow me to  
paste in invalid characters--especially if those characters cause  
problems).

To me, displaying bad glyphs (at least with the source code in its  
current shape) is a much worse problem than the issue of finding  
rogue <lf>'s.

Unless there's a strong reason to the contrary, the editors should  
follow the old CS advice. Be generous in what you accept, be strict  
in what you transmit (in this case, in what you save).

You have yet to convince me that being able to see <lf>'s just so we  
can delete them is a strong reason. For it to be a strong reason, you  
need to show 1) that there is a concrete benefit to removing the  
<lf>'s and 2) that always displaying the <lf>'s so they can be  
deleted by hand is the best method for dealing with them.

> But really, it's just a matter
> of preference, so try using CrLfFileStream if you prefer that  
> approach.

Realistically speaking, I'll probably just go back to 3.8. However, I  
would like to do my part to make Squeak better. Part of that includes  
voicing my opinion when I think things are being changed for the worse.

To me, this seems like a poor design decision. I think there will be  
a lot of unforeseen consequences (for example, making Squeak  
incompatible with windows text files). Yes, there are ways to work  
around the issue, but that's not the point. Not all users will know  
how to work around the issue. Having a reasonable default behavior is  
vital.

>> Here's my view. Back in the dark ages, moving text files from one
>> system to another was a nightmare.
>
> I guess if we were out of the dark ages, we would not still be
> using files.

Sorry, I don't understand this comment at all.

My point was that now, in the 21st century, our systems should be  
smart enough to open text files properly, regardless of which system  
they originated on. For any modern text editor, newline  
incompatibilities are a think of the past. They should be a thing of  
the past in Squeak.

Remember, this is an issue that goes well beyond the Squeak source  
code. There are many reasons why someone might want to open and edit  
text files within squeak. This change will affect all of them, and  
will make it much harder to use any text files that came from other  
sources.

Compatibility is still a good thing. But that's just my 2 cents.

-Rich-

Rich everything is possible with time and money.

But this is not worse.
Because you have exactly the same problems in 3.8 but you do not see it.
So for our perspective this is really worse.

> To me, this seems like a poor design decision. I think there will  
> be a lot of unforeseen consequences (for example, making Squeak  
> incompatible with windows text files). Yes, there are ways to work  
> around the issue, but that's not the point. Not all users will know  
> how to work around the issue. Having a reasonable default behavior  
> is vital.

But this was ALWAYS like that. We just display a glyph that was empty  
before.




Sure but we do not have the time to build new editor for now.
So if you want to see that happening introduce this behavior and send  
it to us.