New Win32 VM [m17n testers needed]

On Jun 4, 2007, at 15:48 , subbukk wrote:

Right - see the thread the VM dev list. It would be great if we could  
get those queries into the 3.10 VMs.

> In Tim's words :
>
> On Tuesday 15 May 2007 12:46 am, tim Rowledge wrote:
>> .. Allowing #fileNamed: to
>> attempt to parse mangled platform related strings was a serious
>> error. Platform fiddle-faddle for filenames is just horrific.

Fully agreed. It's just so much easier to do everything in the image  
that many people prefer that way. Moving all the logic into the VM is  
a lot harder. For example, most, if not all of the m17n code in 3.8  
was done without VM modification. And it's still far from flawless -  
so once the VMs do have better support we should indeed refactor.

- Bert -

> The conditional is incorrect and unnecessary. filename encoding depends on
> filesystem, not the VM platform.

On a bad operating system, it does. On a good operating system, the
system provides abstraction from file system details as much as
feasible.

Win32 in particular provides a homogeneous encoding of file names;
using the "ansi code page" in the byte-oriented APIs, and using
Unicode UCS-2 in the wide-string APIs. The Smalltalk VM then
can provide UTF-8 file names across file systems on Windows.

It would actually be good if the VM would guarantee UTF-8 file
names on all systems.

> For instance, I could have a UTF-8 file on a
> USB flash and use it across different VMs.

I don't understand: you mean a file whose contents is UTF-8
encoded, or whose file name is? On an USB stick, you need some
file system; whether the file name will be UTF-8 encoded
on disk depends on the file system.

> AFAIK, FAT fs does not support UTF-8. NTFS, HPFS (Mac) and all current Linux
> filesystems support UTF-8 in filenames.

Wrong both ways. NTFS does not store file names in UTF-8, but in UTF-16;
whether Linux stores file names in UTF-8 depends on the locale. FAT (in
it's vFAT version which is now universally being used) also stores file
names in UTF-16.

Regards,
Martin

Ok. However, this still doesn't answer the questions:
a) why do I have to configure my image myself to set the language?
   (couldn't Squeak find out on its own what language was configured
   in the host system?), and
b) as I need to select the fonts to render the text *anyway* (through
   the appearance... menu), what is the purpose of the language
   environment then?

>   Although the current Squeak implementation is not there yet, you
> would like to do different sorting or uppercase/lowercase conversions
> based on the language (even within Latin-1 regions).  A segment of
> text generally should have more information other than the bare code
> point of Unicode.

That's an entirely different issue, and completely unrelated to the
changes discussed. I understand the concept of a locale. Whether
collation should take the locale into account is independent on
whether I can enter arbitrary characters through the keyboard,
or open files with arbitrary names.

Regards,
Martin

Hi Folks -

[cc: v3dot10 for consideration of related fixes]

After a few more rounds of fixes and debugging (incl. the
unicodification of the drag and drop and async file primitives) we have
a shiny new 3.10.3 VM which should be usable for a more general audience:

http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-bin.zip
http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-src.zip

I think this is a good candidate to get any pending fixes into 3.10 - I
have posted the list of required fixes to Mantis and would like to ask
the 3.10 team to consider those:

http://bugs.squeak.org/view.php?id=6523
http://bugs.squeak.org/view.php?id=6524
http://bugs.squeak.org/view.php?id=6525
http://bugs.squeak.org/view.php?id=6526

Those of you who have been using 3.10.1 or 3.10.2 please make sure you
upgrade to 3.10.3 - the prior versions had a fatal error in the
clipboard code which might cause VM crashes or other serious memory
corruptions (dumb mistake on my behalf). I have deleted these versions
from the servers and you should do the same.

Other than that: Enjoy!

Cheers,
   - Andreas

> Right, but take a look at the codepage parameter of the functions
>
>    http://msdn2.microsoft.com/en-us/library/ms776413.aspx
>    http://msdn2.microsoft.com/en-us/library/ms776420.aspx
>
> It only supports until UTF8...so, can it work for Japanese chars?

(Not sure what you mean by "supports until UTF8")

MultiByteToWideChar certainly works for Japanese code pages, such
as 932 (Japanese Shift-JIS). It works for all code pages for which
Microsoft has released a version of Windows, and that certainly
includes Japanese.

Regards,
Martin

Martin v. Löwis wrote:
> MultiByteToWideChar certainly works for Japanese code pages, such
> as 932 (Japanese Shift-JIS). It works for all code pages for which
> Microsoft has released a version of Windows, and that certainly
> includes Japanese.

Right, I think this was a misunderstanding. Yoshiki had already sent a
screenshot that shows that he can display file names properly so we know
that WideCharToMultiByte and back works (see
http://lists.squeakfoundation.org/pipermail/squeak-dev/2007-June/117487.html).
The problem appears to be on the IME input (which has no relation at all
to those functions).

Cheers,
   - Andreas

Yoshiki Ohshima wrote:
>   The file path interpretation works on Japanese Windows (see the file
> name pane in the attached picture) by changing the
> fileNameConverterClass of JapaneseEnvironment.  However, I can't quite
> figure out to make keyboard input work...  It doesn't seem that I get
> meaningful values when the input is done via an IME.

Give this another shot with the 3.10.3 VM - one of the bugs I fixed was
the fact that we were delegating from a Unicode WndProcW to an Ansi
DefWindowProcA which would *completely* and mysteriously upset some
functions and not others (like changing the window title would always
fail because it implicitly sends a WM_SETTEXT message which we delegated
with Unicode input to the Ansi version of the handler...) I strongly
suspect that the IME translation gets into similar trouble if you
delegate from WndProcW to DefWindowProcA.

Cheers,
   - Andreas

Hi Andreas,

I'm not part of the release team, but if they are consistent, and I
think they are, they will request unit tests from you before these
fixes get introduced. You have been warned :-)

Bye

2007/6/5, Andreas Raab <[hidden email]>:
> I think this is a good candidate to get any pending fixes into 3.10 - I
> have posted the list of required fixes to Mantis and would like to ask
> the 3.10 team to consider those:
>
> http://bugs.squeak.org/view.php?id=6523
> http://bugs.squeak.org/view.php?id=6524
> http://bugs.squeak.org/view.php?id=6525
> http://bugs.squeak.org/view.php?id=6526

--
Damien Cassou

Damien Cassou wrote:
> I'm not part of the release team, but if they are consistent, and I
> think they are, they will request unit tests from you before these
> fixes get introduced. You have been warned :-)

I'm looking forward to someone explaining to me how to write unit tests
that test the file system encodings and clipboard interface in a
meaningful way. If these tests are any good, they should fail on a 3.9.x
VM and pass on 3.10.3 but I have no idea how to write such a test.

Cheers,
   - Andreas

On Jun 5, 2007, at 7:02 , Martin v. Löwis wrote:

> a) why do I have to configure my image myself to set the language?
>    (couldn't Squeak find out on its own what language was configured
>    in the host system?), and

See below.

> b) as I need to select the fonts to render the text *anyway* (through
>    the appearance... menu), what is the purpose of the language
>    environment then?

See below.

>>    the current Squeak implementation is not there yet

There you have it.

The m17n framework was put in place in 3.8, and it works, in  
principle, but with lots of manual intervention. It has not seen much  
attention since, so now that people seem more interested in the  
remaining issues it's a nice time as ever to start fixing them.

- Bert -

Hi there, looks like this stuff is just in time for the project I'm working
on!

A couple of problems so far though. Seems that when the Ctrl key is down the
codes come back
As the ascii (Ctrl+a = 1) rather than the character "a" (along with the Ctrl
key flag). Additionally, mouse-wheel
activity comes back as ascii value 0. Here's a quick fix, though I expect
some of this should be handled in the
VM.

'From Squeak3.9 of 7 November 2006 [latest update: #7067] on 5 June 2007 at
10:26:08 am'!

!UnicodeInputInterpreter methodsFor:
'*Pinesoft-ReportBuilder-UI-Editing-override' stamp: 'gvc 6/5/2007 10:25'!
nextCharFrom: sensor firstEvt: evtBuf

        | keyValue |
" Unicode charCode expected as the sixth data member !! "
        keyValue := evtBuf sixth.
        keyValue < 32 ifTrue: [keyValue := evtBuf third].
        Transcript show: evtBuf asString; cr.
        ^ keyValue asCharacter.! !

Oh, and it would be nice if the VM version was reported correctly so that
checks can be made in-image as to
whether the Unicode interpreters/converters should be used.

Glad Unicode input is being addressed, however, keep it up!

Gary

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of Andreas
Raab
Sent: 03 June 2007 5:03 am
To: The general-purpose Squeak developers list; Squeak Virtual Machine
Development Discussion
Subject: New Win32 VM [m17n testers needed]


Hi Folks -

Thanks to some dedicated OLPC-related work done in Greece by Chris
Petsos[1], we now have a Windows VM with Unicode support enabled. This
VM will both generate UTF input from characters as well as support
clipboard, file and directory names in UTF-8. The VM is available here:

http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-bin.zip
http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-src.zip

You are invited to test the new work but be advised that this may
require some manual adjustments - for an understanding what needs to be
done, please see [2].

I'm interested in reports, both good and bad about whether the
clipboard, file, directory and input support behaves as expected.

[1]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-May/001194.html
[2]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-June/001306.html

Cheers,
   - Andreas

On Tuesday 05 June 2007 10:25 am, Martin v. Löwis wrote:
>It would actually be good if the VM would guarantee UTF-8 file
> names on all systems
Yes, indeed. The image could query the VM on startup to see if it supports
UTF-8 in filenames.
> > For instance, I could have a UTF-8 file on a
> > USB flash and use it across different VMs.
>
> I don't understand: you mean a file whose contents is UTF-8
> encoded, or whose file name is?
I meant the filename (the context of discussion was filenames). Sorry for the
confusion.
> > AFAIK, FAT fs does not support UTF-8. NTFS, HPFS (Mac) and all current
> > Linux filesystems support UTF-8 in filenames.
> Wrong both ways. NTFS does not store file names in UTF-8, but in UTF-16;
> whether Linux stores file names in UTF-8 depends on the locale. FAT (in
> it's vFAT version which is now universally being used) also stores file
> names in UTF-16.
I wasn't writing about the encoding used in physical store as much as support
for UTF-8 in filenames. On Linux, ntfs and vfat mounts convert to utf8 on the
fly with the right locale. For me, utf8 has been the default locale for so
long that I just plain forgot about this setting :-). Thanks for pointing it
out.

Regards .. Subbu

El 6/5/07 2:16 AM, "Andreas Raab" <[hidden email]> escribió:

I take note Andreas.
As soon I solve some weird things using Ralph advice, take this top on my
list.
I have a now small LAN with a PC clon for testing , besides my Mac.
A Linux machine could soon add and we have how test in the three most used
OS.

Edgar

> I think this is a good candidate to get any pending fixes into 3.10 - I
> have posted the list of required fixes to Mantis and would like to ask
> the 3.10 team to consider those:
>
> http://bugs.squeak.org/view.php?id=6523

This seems pretty Windows specific.  What is its effect on non-Windows
platforms?

> http://bugs.squeak.org/view.php?id=6524
> http://bugs.squeak.org/view.php?id=6525
> http://bugs.squeak.org/view.php?id=6526

How much have these been tested on non-Windows platforms?

-Ralph

  Martin,

> Ok. However, this still doesn't answer the questions:
> a) why do I have to configure my image myself to set the language?
>    (couldn't Squeak find out on its own what language was configured
>    in the host system?), and
>
> b) as I need to select the fonts to render the text *anyway* (through
>    the appearance... menu), what is the purpose of the language
>    environment then?

  The *default* one can be fetched from the host system, sure.  But
you still want to use, for example, Chinese on Japanese Windows, it is
necessary to be able to set it.

-- Yoshiki

Ralph Johnson wrote:
>> I think this is a good candidate to get any pending fixes into 3.10 - I
>> have posted the list of required fixes to Mantis and would like to ask
>> the 3.10 team to consider those:
>>
>> http://bugs.squeak.org/view.php?id=6523
>
> This seems pretty Windows specific.  What is its effect on non-Windows
> platforms?

They should have no effect at all.

>> http://bugs.squeak.org/view.php?id=6524
>> http://bugs.squeak.org/view.php?id=6525
>> http://bugs.squeak.org/view.php?id=6526
>
> How much have these been tested on non-Windows platforms?

Zero. I only have a windows box.

Cheers,
   - Andreas

Hi Andreas,

When I started Squeakland image with new vm on the desktop (I
extracted SqueakVM-Win32-3.10.3-bin.zip, and dragged
SqueakPlugin.image icon to Squeak.exe), I got this error.

Image file read problem (0 out of 4 bytes read)

Cheers,
- Takashi

Andreas Raab wrote:
>
> After a few more rounds of fixes and debugging (incl. the
> unicodification of the drag and drop and async file primitives) we have
> a shiny new 3.10.3 VM which should be usable for a more general audience:
>
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-bin.zip
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-src.zip

I just did the same without any problems. Can you check to see whether
that was a one-time problem or if a different image file works. And if
so, can you try to download the VM again (perhaps there was something
corrupted?). Oh, and finally, check your virus, spyware etc. checker -
they might think to take a closer look on an application that you just
put there and dropped a file on.

Cheers,
   - Andreas

Takashi Yamamiya wrote:

> I'm looking forward to someone explaining to me how to write unit tests
> that test the file system encodings and clipboard interface in a
> meaningful way. If these tests are any good, they should fail on a 3.9.x
> VM and pass on 3.10.3 but I have no idea how to write such a test.

It might be possible to escape to VBScript or JScript to achieve that;
I would hope (but am unsure) that the VBScript interpreter supports
non-CP_ACP filenames. So you would generate a VBScript program on the
fly, run it, and thus create a few interesting files.

To get you started: ChrW() is a function that creates a Unicode
character given its ordinal, and the sequence

dim filesys, filetxt, getname, path
Set filesys = CreateObject("Scripting.FileSystemObject")
Set filetxt = filesys.CreateTextFile("c:\somefile.txt", True)
filetxt.WriteLine("Your text goes here.")
filetxt.Close

should create a file named somefile.txt.

I read that neither wsh.exe nor csh.exe (the scripting hosts for
VB, in windowing or command-line mode) cannot do clipboard
IO, so you use MSIE for that:

Set objIE = CreateObject("InternetExplorer.Application")
objIE.Navigate("about:blank")
strURL = objIE.document.parentwindow.clipboardData.GetData("text")
objIE.Quit

To put data to the clipboard, do

objIE.document.parentwindow.clipboardData.SetData "text", "Hello This Is
A Test"

HTH,
Martin

  I think what Takashi meant was to put an image file on the "desktop",
which is translated to Katakana characters on Japanese Windows, and
then try to launch the image with the new VM.  I did get the same
error in this way.

-- Yoshiki

At Tue, 05 Jun 2007 09:58:20 -0700,
Andreas Raab wrote: