Smalltalk › Squeak › Squeak VM

New Win32 VM [m17n testers needed]

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

25 messages Options

Andreas.Raab

New Win32 VM [m17n testers needed]

Hi Folks -

Thanks to some dedicated OLPC-related work done in Greece by Chris
Petsos[1], we now have a Windows VM with Unicode support enabled. This
VM will both generate UTF input from characters as well as support
clipboard, file and directory names in UTF-8. The VM is available here:

http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-bin.zip
http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-src.zip

You are invited to test the new work but be advised that this may
require some manual adjustments - for an understanding what needs to be
done, please see [2].

I'm interested in reports, both good and bad about whether the
clipboard, file, directory and input support behaves as expected.

[1]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-May/001194.html
[2]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-June/001306.html

Cheers,
- Andreas

Chris Petsos

Re: New Win32 VM [m17n testers needed]

> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-bin.zip
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-src.zip
>
> You are invited to test the new work but be advised that this may require
> some manual adjustments - for an understanding what needs to be done

I am getting a primitive has failed error when i try to publish an EToy...
I think it's happening when it tries to create the resourcecache.map file in
the Squeaklets folder, or the folder itself.

To reproduce the problem:
1) Just run the exe inside the *bin.zip file above.
2) Try to publish the empty eToy.

Note that the VM version that i compiled does not create this error...

Christos

Andreas.Raab

Re: New Win32 VM [m17n testers needed]

Chris Petsos wrote:
> I am getting a primitive has failed error when i try to publish an EToy...
> I think it's happening when it tries to create the resourcecache.map
> file in the Squeaklets folder, or the folder itself.

Thanks for finding this. The problem was a left-over CreateDirectoryA()
call which of course should now be CreateDirectoryW(). It'll be fixed in
the next version.

Cheers,
- Andreas

> To reproduce the problem:
> 1) Just run the exe inside the *bin.zip file above.
> 2) Try to publish the empty eToy.
>
> Note that the VM version that i compiled does not create this error...
>
> Christos

Andreas.Raab

Re: New Win32 VM [m17n testers needed]

In reply to this post by Andreas.Raab

Yes, I just fixed that. A left-over call to CreateDirectoryA() would
make directory creation impossible and later file creation attempts as
well. Will be fixed in the next version.

Cheers,
- Andreas

Alexander Lazarevic' wrote:

> I just did a quick test out of curiosity. I did an "update code from
> server" on a rather old image (Squeak3.9gamma [latest update: #7066])
> and it seems there is an error while creating a cache directory/file.
> The primitive "primOpen: fileName writable: writableFlag" fails on the
> filename [1] and writableFlag true.
> This works with a 3.7.1 VM.
>
> Alex
>
> [1] 'C:\Dokumente und
> Einstellungen\laza\Desktop\SqueakVM-Win32-3.10.1-bin\package-cache\ScriptLoader-sd.324.mcz'
>
> Andreas Raab schrieb:
>> Hi Folks -
>>
>> Thanks to some dedicated OLPC-related work done in Greece by Chris
>> Petsos[1], we now have a Windows VM with Unicode support enabled. This
>> VM will both generate UTF input from characters as well as support
>> clipboard, file and directory names in UTF-8. The VM is available here:
>>
>> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-bin.zip
>> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-src.zip
>>
>> You are invited to test the new work but be advised that this may
>> require some manual adjustments - for an understanding what needs to be
>> done, please see [2].
>>
>> I'm interested in reports, both good and bad about whether the
>> clipboard, file, directory and input support behaves as expected.
>>
>> [1]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-May/001194.html
>> [2]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-June/001306.html
>>
>> Cheers,
>> - Andreas
>>
>
>

Andreas.Raab

Re: New Win32 VM [m17n testers needed]

In reply to this post by Andreas.Raab

Martin v. Löwis wrote:
> What I couldn't get working is the listing of file names in the
> file browser. To reproduce, create file names using Greek, Cyrillic,
> and Chinese letters, and then do "open/file list". With the wrong font,
> I get question marks. When I select a font that ought to be able to
> represent it correctly, I still get a mix of Latin letters and
> square boxes.

Hm ... there isn't any easy way to test this I guess? The code hasn't
changed that much so I would expect this to be working (in particular
considering that it seems to work fine for ascii file names).

> What I don't understand is: Why do I have to set the language
> environment (*) to make it work? It's Unicode, so Squeak
> shouldn't care what the language is. If it needs to know, it
> should get the language from the system.

I don't know. In particular considering that we have now the locale
plugin which can detect these settings easily.

Cheers,
- Andreas

Andreas.Raab

Re: New Win32 VM [m17n testers needed]

In reply to this post by Andreas.Raab

Martin v. Löwis wrote:
> What I couldn't get working is the listing of file names in the
> file browser. To reproduce, create file names using Greek, Cyrillic,
> and Chinese letters, and then do "open/file list". With the wrong font,
> I get question marks. When I select a font that ought to be able to
> represent it correctly, I still get a mix of Latin letters and
> square boxes.

Digging in the code it seems that the conversion of file names is broken
(or at least it seems that way). I can't seem to find the place where a
UTF8TextConverter would ever be used (which of course is a requirement
for this to work). It seems that the code still assumes that the VMs
present file names encoded in the corresponding code pages (which also
explains why you'd need to set the language environment etc). The thing
to try would be to go into LanguageEnvironment class and change
defaultFileNameConverter to include:

"Windows VMs always use UTF8-encoded file names now"
Smalltalk platformName = 'Win32'
ifTrue:[^UTF8TextConverter new].

Cheers,
- Andreas

Andreas.Raab

Re: New Win32 VM [m17n testers needed]

In reply to this post by Andreas.Raab

I have fixed the problem with directory creation and updated the VM to
3.10.2 which is up in the usual places:

http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.2-bin.zip
http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.2-src.zip

Cheers,
- Andreas

Andreas Raab wrote:

> Hi Folks -
>
> Thanks to some dedicated OLPC-related work done in Greece by Chris
> Petsos[1], we now have a Windows VM with Unicode support enabled. This
> VM will both generate UTF input from characters as well as support
> clipboard, file and directory names in UTF-8. The VM is available here:
>
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-bin.zip
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.1-src.zip
>
> You are invited to test the new work but be advised that this may
> require some manual adjustments - for an understanding what needs to be
> done, please see [2].
>
> I'm interested in reports, both good and bad about whether the
> clipboard, file, directory and input support behaves as expected.
>
> [1]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-May/001194.html
> [2]http://lists.squeakfoundation.org/pipermail/vm-dev/2007-June/001306.html
>
> Cheers,
> - Andreas
>
>

Yoshiki Ohshima

Re: New Win32 VM [m17n testers needed]

> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.2-bin.zip
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.2-src.zip

The file path interpretation works on Japanese Windows (see the file
name pane in the attached picture) by changing the
fileNameConverterClass of JapaneseEnvironment. However, I can't quite
figure out to make keyboard input work... It doesn't seem that I get
meaningful values when the input is done via an IME.

I'll take a look at it later...

-- Yoshiki

filelist.png (16K) Download Attachment

Yoshiki Ohshima

Re: New Win32 VM [m17n testers needed]

In reply to this post by Andreas.Raab

Martin,

> What I couldn't get working is the listing of file names in the
> file browser. To reproduce, create file names using Greek, Cyrillic,
> and Chinese letters, and then do "open/file list". With the wrong font,
> I get question marks. When I select a font that ought to be able to
> represent it correctly, I still get a mix of Latin letters and
> square boxes.

As Andreas wrote, defaultFileNameConverter has to be modified (and
the class var in the LanguageEnvironment has to be cleared.)

> What I don't understand is: Why do I have to set the language
> environment (*) to make it work? It's Unicode, so Squeak
> shouldn't care what the language is. If it needs to know, it
> should get the language from the system.

Read the Unicode standard.

Because it is Unicode, a mechanism out of scope of Unicode has to
supply language information to do sensible stuff.

-- Yoshiki

Chris Petsos

Re: Re: New Win32 VM [m17n testers needed]

In reply to this post by Yoshiki Ohshima

> The file path interpretation works on Japanese Windows (see the file
> name pane in the attached picture) by changing the
> fileNameConverterClass of JapaneseEnvironment. However, I can't quite
> figure out to make keyboard input work... It doesn't seem that I get
> meaningful values when the input is done via an IME.

Yoshiki,

Don't forget that the VM is sending unicode chars as the sixth data member
of the event buffer.
So advice your input interpreter as

evtbuf sixth.

instead of

evtbuf third.

I also tried to copy/paste same japanese text with the new VM and i couldn't
do it...
Japanese locale specific question... does a unicode Japanese character fit
inside a WCHAR struct?

Cause, there have been used Windows functions that convert WCHAR streams

MultiByteToWideChar
WideCharToMultiByte

Could the problem be there?

Christos.

Yoshiki Ohshima

Re: Re: New Win32 VM [m17n testers needed]

Chris,

> Don't forget that the VM is sending unicode chars as the sixth data member
> of the event buffer.

Sure. I looked at these values yet didn't see it.

(If I remember correctly, it got changed (from third to
sixth. take a look at MacUnicodeInputInterpreter>>initialize. It
tells you the history^^;))

> Japanese locale specific question... does a unicode Japanese character fit
> inside a WCHAR struct?

It does. UTF-16LE without surrogated pairs is almost ok for daily
use.

> Cause, there have been used Windows functions that convert WCHAR streams
>
> MultiByteToWideChar
> WideCharToMultiByte
>
> Could the problem be there?

I can't quite trace the detail (and my memory), but when the macro
UNICODE is defined, the latter should be just used?

-- Yoshiki

Andreas.Raab

Re: Re: New Win32 VM [m17n testers needed]

Yoshiki Ohshima wrote:
>> Cause, there have been used Windows functions that convert WCHAR streams
>>
>> MultiByteToWideChar
>> WideCharToMultiByte
>>
>> Could the problem be there?
>
> I can't quite trace the detail (and my memory), but when the macro
> UNICODE is defined, the latter should be just used?

The VM is not generally compiled with -DUNICODE; the places where we
utilize WCHAR are explicit and we use the explicit *W variants of the
Windows functions.

Cheers,
- Andreas

Chris Petsos

Re: Re: New Win32 VM [m17n testers needed]

In reply to this post by Yoshiki Ohshima

>> Japanese locale specific question... does a unicode Japanese character
>> fit
>> inside a WCHAR struct?
>
> It does. UTF-16LE without surrogated pairs is almost ok for daily
> use.
>

Right, but take a look at the codepage parameter of the functions

http://msdn2.microsoft.com/en-us/library/ms776413.aspx
http://msdn2.microsoft.com/en-us/library/ms776420.aspx

It only supports until UTF8...so, can it work for Japanese chars?

>> Cause, there have been used Windows functions that convert WCHAR streams
>>
>> MultiByteToWideChar
>> WideCharToMultiByte
>>

Christos.

Andreas.Raab

Re: New Win32 VM [m17n testers needed]

In reply to this post by Andreas.Raab

Martin v. Löwis wrote:

>> Hm ... there isn't any easy way to test this I guess?
>
> It's actually very easy.
>
> 1. Create a text file.
> 2. rename it to some Cyrillic (or Japanese, or whatever) name
> 3. Open it in the listing
>
> To rename it, the easiest way is to start up charmap.exe,
> select a few "funny" characters, copy them to the clipboard,
> and past them in explorer into the file name.

Ah great. Indeed, there is a series of steps you need to take to make it
work:
1) You need to fix LanguageEnvironment class defaultFileNameConverter
(make it return UTF8TextConverter new)
2) You need to load a TTF font with the glyphs. For this you need:
* Load the TTF loading fixes that Christos posted
* Drag and drop a TTF font with the right glyphs on Squeak (Arial
works fine)
3) Make this font the default font for text and lists.

Once you got all of this the file list shows the correct names.

Cheers,
- Andreas

Filelist.gif (39K) Download Attachment

Yoshiki Ohshima

Re: New Win32 VM [m17n testers needed]

In reply to this post by Yoshiki Ohshima

Martin,

> >> What I don't understand is: Why do I have to set the language
> >> environment (*) to make it work? It's Unicode, so Squeak
> >> shouldn't care what the language is. If it needs to know, it
> >> should get the language from the system.
> >
> > Read the Unicode standard.
>
> I did. What section are you specifically referring to?

For example, take a look at this FAQ entry:

http://www.unicode.org/faq/han_cjk.html#3

(and one before this and after).

> > Because it is Unicode, a mechanism out of scope of Unicode has to
> > supply language information to do sensible stuff.
>
> What is the sensible stuff it needs to do?

To display strings in an ok way.

http://www.unicode.org/faq/han_cjk.html#2

says that you should select a proper font based on the language you
would like to treat the character in.

Although the current Squeak implementation is not there yet, you
would like to do different sorting or uppercase/lowercase conversions
based on the language (even within Latin-1 regions). A segment of
text generally should have more information other than the bare code
point of Unicode.

-- Yoshiki

Andreas.Raab

New Win32 VM (3.10.3)

In reply to this post by Andreas.Raab

Hi Folks -

[cc: v3dot10 for consideration of related fixes]

After a few more rounds of fixes and debugging (incl. the
unicodification of the drag and drop and async file primitives) we have
a shiny new 3.10.3 VM which should be usable for a more general audience:

http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-bin.zip
http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-src.zip

I think this is a good candidate to get any pending fixes into 3.10 - I
have posted the list of required fixes to Mantis and would like to ask
the 3.10 team to consider those:

http://bugs.squeak.org/view.php?id=6523
http://bugs.squeak.org/view.php?id=6524
http://bugs.squeak.org/view.php?id=6525
http://bugs.squeak.org/view.php?id=6526

Those of you who have been using 3.10.1 or 3.10.2 please make sure you
upgrade to 3.10.3 - the prior versions had a fatal error in the
clipboard code which might cause VM crashes or other serious memory
corruptions (dumb mistake on my behalf). I have deleted these versions
from the servers and you should do the same.

Other than that: Enjoy!

Cheers,
- Andreas

Andreas.Raab

Re: Re: New Win32 VM [m17n testers needed]

In reply to this post by Chris Petsos

Martin v. Löwis wrote:
> MultiByteToWideChar certainly works for Japanese code pages, such
> as 932 (Japanese Shift-JIS). It works for all code pages for which
> Microsoft has released a version of Windows, and that certainly
> includes Japanese.

Right, I think this was a misunderstanding. Yoshiki had already sent a
screenshot that shows that he can display file names properly so we know
that WideCharToMultiByte and back works (see
http://lists.squeakfoundation.org/pipermail/squeak-dev/2007-June/117487.html).
The problem appears to be on the IME input (which has no relation at all
to those functions).

Cheers,
- Andreas

Andreas.Raab

Re: New Win32 VM [m17n testers needed]

In reply to this post by Yoshiki Ohshima

Yoshiki Ohshima wrote:
> The file path interpretation works on Japanese Windows (see the file
> name pane in the attached picture) by changing the
> fileNameConverterClass of JapaneseEnvironment. However, I can't quite
> figure out to make keyboard input work... It doesn't seem that I get
> meaningful values when the input is done via an IME.

Give this another shot with the 3.10.3 VM - one of the bugs I fixed was
the fact that we were delegating from a Unicode WndProcW to an Ansi
DefWindowProcA which would *completely* and mysteriously upset some
functions and not others (like changing the window title would always
fail because it implicitly sends a WM_SETTEXT message which we delegated
with Unicode input to the Ansi version of the handler...) I strongly
suspect that the IME translation gets into similar trouble if you
delegate from WndProcW to DefWindowProcA.

Cheers,
- Andreas

Yoshiki Ohshima

Re: New Win32 VM [m17n testers needed]

In reply to this post by Yoshiki Ohshima

Martin,

> Ok. However, this still doesn't answer the questions:
> a) why do I have to configure my image myself to set the language?
> (couldn't Squeak find out on its own what language was configured
> in the host system?), and
>
> b) as I need to select the fonts to render the text *anyway* (through
> the appearance... menu), what is the purpose of the language
> environment then?

The *default* one can be fetched from the host system, sure. But
you still want to use, for example, Chinese on Japanese Windows, it is
necessary to be able to set it.

-- Yoshiki

Takashi Yamamiya

Re: New Win32 VM (3.10.3)

In reply to this post by Andreas.Raab

Hi Andreas,

When I started Squeakland image with new vm on the desktop (I
extracted SqueakVM-Win32-3.10.3-bin.zip, and dragged
SqueakPlugin.image icon to Squeak.exe), I got this error.

Image file read problem (0 out of 4 bytes read)

Cheers,
- Takashi

Andreas Raab wrote:
>
> After a few more rounds of fixes and debugging (incl. the
> unicodification of the drag and drop and async file primitives) we have
> a shiny new 3.10.3 VM which should be usable for a more general audience:
>
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-bin.zip
> http://www.squeakvm.org/win32/release/SqueakVM-Win32-3.10.3-src.zip

vm-win32-desktop.png (8K) Download Attachment