Croquet VM issues

Hi Folks -

Since we're nearing release status I was going over the VM requirements
for Croquet to see if we could just go with a bunch of standard VMs or
whether we'd need some customized ones. Looking over some recent VMMaker
versions, it turns out that there are actually a few important bits and
pieces missing from the current VMMaker release which we will want to
use in Croquet.

Most importantly, we need the following issues resolved:

1) GC improvements: As discussed in April last year
(see
http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?list=vm-dev&cmd=showmsg&msgnum=184)
we need the weak array fixes and the GC measurements/tweaks that John
did. Those actually turn out to be important for us, if only for
analysis of current behavior.

Questions:
* What's the general status of those changes? The way I understand the
discussion we had back in april, is that John did the adjustments to
64bit and then ... what?
* @Tim: Does a VMMaker repository exist by now? Like I was saying back
then, I am not alltogether happy with the current situation of new
releases just popping up on SqueakMap and with no way of seeing what's
in there and how things have developed.

2) Floating point accuracy: As discussed in november last year
(see
http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?list=vm-dev&cmd=showmsg&msgnum=365)
we need consistent floating point behavior across platforms. I'm
actually in the process or rolling out a bunch of tests for it which
will be based on fdlibm as described in the above message. One issue
that is critical here is that I am planning to checksum (SHA1) the VMs
while we are running Croquet to see if the VM is "compliant" (if not
we'd run a whole suite of tests to verify the behavior).

Questions:
* On Mac and Unix, do we implement getSystemAttribute: -1 to return the
full path to the executable?
* On Mac and Unix, can one read the executable while running Squeak? I
need this to checksum the VM proper.

3) Some Socket fixes: David Reed made some changes for Sockets which
turn out to be critical in particular on Macs and Unix systems. I'm
still in the process of bundling up these changes but they are fairly
straightforward so I don't see many issues with them.

Besides the above three, I have one additional question: What is the
general status of the 64bit work right now? Do we consider this complete
or work in progress or experimental? Can we rule out any ill effects for
the 32bit VMs?

Thanks and Cheers,
   - Andreas

On 16-Mar-06, at 2:10 PM, Andreas Raab wrote:

> Questions:
> * What's the general status of those changes? The way I understand  
> the discussion we had back in april, is that John did the  
> adjustments to 64bit and then ... what?

These were folded into the VMMaker in mid may 2005, waiting feedback  
from others.


I did load that floating point lib and it works on the mac vm,  
waiting a decision from others...

>
> Questions:
> * On Mac and Unix, do we implement getSystemAttribute: -1 to return  
> the full path to the executable?
> * On Mac and Unix, can one read the executable while running  
> Squeak? I need this to checksum the VM proper.

Mmmm let me see, nope (-1) doesn't work, do you know if that works  
with unix?

You should be able to read the vm file, although you should consider  
at some point a vm could be marked as executable and not readable.

>
> 3) Some Socket fixes: David Reed made some changes for Sockets  
> which turn out to be critical in particular on Macs and Unix  
> systems. I'm still in the process of bundling up these changes but  
> they are fairly straightforward so I don't see many issues with them.

Well then share.

>
> Besides the above three, I have one additional question: What is  
> the general status of the 64bit work right now? Do we consider this  
> complete or work in progress or experimental? Can we rule out any  
> ill effects for the 32bit VMs?

? Ask Ian or Dan

>
> Thanks and Cheers,
>   - Andreas

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Hi John -

Thanks for the information.

>> 1) GC improvements:
>> Questions:
>> * What's the general status of those changes? The way I understand the
>> discussion we had back in april, is that John did the adjustments to
>> 64bit and then ... what?
>
> These were folded into the VMMaker in mid may 2005, waiting feedback
> from others.

So is there a "current" VMMaker version that includes these? (I'm asking
since when Josh sent the message lately you replied by pointing to a
webpage with a few bits of loose code on there ;-)

>> Questions:
>> * On Mac and Unix, do we implement getSystemAttribute: -1 to return
>> the full path to the executable?
>> * On Mac and Unix, can one read the executable while running Squeak? I
>> need this to checksum the VM proper.
>
> Mmmm let me see, nope (-1) doesn't work, do you know if that works with
> unix?

No, I don't know. The main thing is that I need a way of computing the
checksum for the actual binary (not just a shell script which invoked
the the binary somehow) and I would expect that getSystemAttribute: 0
gives us the shell script, no?

> You should be able to read the vm file, although you should consider at
> some point a vm could be marked as executable and not readable.

An alternative would be to require a primitive that gives us a build
UUID but I'm trying to avoid that.

Cheers,
   - Andreas

>
> 1) GC improvements: As discussed in April last year
> (see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?
> list=vm-dev&cmd=showmsg&msgnum=184)
> we need the weak array fixes and the GC measurements/tweaks that  
> John did. Those actually turn out to be important for us, if only  
> for analysis of current behavior.

Those changes were incorporated LAST APRIL. If you had bothered to  
actually build a VM with any vmmaker since then you'd have found them  
in as agreed. The latest vmmaker can be found on SM, along with  
several earlier versions.

> * @Tim: Does a VMMaker repository exist by now? Like I was saying  
> back then, I am not alltogether happy with the current situation of  
> new releases just popping up on SqueakMap and with no way of seeing  
> what's in there and how things have developed.
No, there is no vmmaker repository.

>
> 2) Floating point accuracy: As discussed in november last year
> (see http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?
> list=vm-dev&cmd=showmsg&msgnum=365)
Yes, and .... what? I have no record of anything going beyond a vague  
suggestion. How would this have anything to affect vmmaker? You want  
to use it, you change your makefile settings.

>
> Questions:
> * On Mac and Unix, do we implement getSystemAttribute: -1 to return  
> the full path to the executable?
See http://minnow.cc.gatech.edu/squeak/314 This is a spec that was  
agreed to about 8 years ago. If any platform is non-compliant it  
should be brought into compliance asap.
>
> 3) Some Socket fixes: David Reed made some changes for Sockets  
> which turn out to be critical in particular on Macs and Unix  
> systems. I'm still in the process of bundling up these changes but  
> they are fairly straightforward so I don't see many issues with them.
Submit them in the usual manner and I'll look at them sometime.

>
> Besides the above three, I have one additional question: What is  
> the general status of the 64bit work right now? Do we consider this  
> complete or work in progress or experimental? Can we rule out any  
> ill effects for the 32bit VMs?
As far as I'm concerned the 64 bit stuff is pathetically sulking in a  
corner due to almost nobody having paid it any attention.  I have  
done everything reasoanble for vmmaker but there are a largish number  
of places where there are 'flag: #dan' type comments and there are  
places where there are comments about some change needing work  
someday. The only VMs I know of that have been built and released  
with the 64bit changes incorporated have been on RISC OS and OSX so  
to the best of my knowledge there is no information on whether they  
are any use on windoze or *nix. I was considering backing out all the  
changes since it seems that nobody cares even fractionally enough to  
finish what they started.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: ED: Eject Disk

On 16-Mar-06, at 5:13 PM, tim Rowledge wrote:

>>
>> Questions:
>> * On Mac and Unix, do we implement getSystemAttribute: -1 to  
>> return the full path to the executable?
> See http://minnow.cc.gatech.edu/squeak/314 This is a spec that was  
> agreed to about 8 years ago. If any platform is non-compliant it  
> should be brought into compliance asap.

Sigh
The mac vm has always returned the path to the vm, and not the name  
of the executable, note the comment in getSystemAttribute:
However I'll consider changing that.

Smalltalk getSystemAttribute: -1

does nothing on the mac or unix unless you've passed in a parameter  
on the cmd line.


--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

Hi Tim -

No reason to get upset. I was just asking for a status update. I checked
with 3.8b4 as reference version since all the others have big fat
disclaimers on them about "use at your own peril etc." and 3.8b4 (btw,
SqueakMap says it's from september '05) does not contain these changes.
They first show up in 3.8b5-64 (from december '05). And talking about
versions what *is* the latest stable version? SqueakMap lists 3.8b5-64B
as the latest; is there any version that is either newer or considered
"non-beta" or both?

>> * @Tim: Does a VMMaker repository exist by now? Like I was saying back
>> then, I am not alltogether happy with the current situation of new
>> releases just popping up on SqueakMap and with no way of seeing what's
>> in there and how things have developed.
> No, there is no vmmaker repository.

Okay, can we agree on one? Say, source.squeakfoundation.org?

>> 2) Floating point accuracy: As discussed in november last year
>> (see
>> http://discuss.squeakfoundation.org/cgi-bin/ezmlm-browse?list=vm-dev&cmd=showmsg&msgnum=365)
>>
> Yes, and .... what? I have no record of anything going beyond a vague
> suggestion. How would this have anything to affect vmmaker? You want to
> use it, you change your makefile settings.

I didn't say it would affect VMMaker. I am trying to find out whether it
is possible on various platforms to checksum the VM since we have very
specific requirements (like float behavior) for Croquet. Also, in the
post back then I was asking the question about what people would think
about building entire VMs using fdlibm. That's still on the table (but
the lack of comments back then makes me think that I'll probably opt for
a MathPlugin instead of fixing the VMs internally).

>> Questions:
>> * On Mac and Unix, do we implement getSystemAttribute: -1 to return
>> the full path to the executable?
> See http://minnow.cc.gatech.edu/squeak/314 This is a spec that was
> agreed to about 8 years ago. If any platform is non-compliant it should
> be brought into compliance asap.

It's actually not clear to me what is "compliant" in this case. For one
thing, is reporting the name of a shell script that starts Squeak
compliant for system attribute 0? (I would think so) For another, is the
name of the VM part of the "negative set" of attributes (yes for Windows
but mostly for historical reasons since system attribute 0 used to be
reported in 8.3 naming style)?

>> 3) Some Socket fixes: David Reed made some changes for Sockets which
>> turn out to be critical in particular on Macs and Unix systems. I'm
>> still in the process of bundling up these changes but they are fairly
>> straightforward so I don't see many issues with them.
> Submit them in the usual manner and I'll look at them sometime.

Will do, thanks.

Well, as always, I guess the lack of an actual user base is problematic
here. And my main problem is that I don't know much about the actual
changes that happened (I know them on the large scale, yes, but not in
detail) and since these things went back and forth I am just not certain
what the current status of them is.

On the other hand, if John has been using those changes for OSX VMs and
not gotten a whole lot of horrible complaints I think I'm willing to
take that risk.

Cheers,
   - Andreas

On 16-Mar-06, at 5:59 PM, Andreas Raab wrote:

I'm changing the mac vm to return the full vm executable path name  
for system attribute 0 as of 3.8.11b3


No complaints.

Oh and btw if you use sqGnu.h and think that the register overrides  
for the interp  method actually improves things in gcc 4.0, then you  
should
think again.... Also do some benchmarking and cross checking, perhaps  
GCC 4.x is smarter now.


>
> Cheers,
>   - Andreas

--
========================================================================
===
John M. McIntosh <[hidden email]> 1-800-477-2659
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
========================================================================
===

On Thu, Mar 16, 2006 at 02:10:08PM -0800, Andreas Raab wrote:
>
> Questions:
> * On Mac and Unix, do we implement getSystemAttribute: -1 to return the
> full path to the executable?

Did you mean system attribute 0, which corresponds to argv[0]? But no,
this would not necessarily return "the" full path to the executable,
nor does it necessarily return *any* full path to the executable.
It's just the name that some arbitrary shell program used to
locate an executable file after having consulted with the $PATH and
following various symbolic links and so forth. Once the executable
file was located, the shell did a fork() and exec(), after which
all was forgotten. As far as I know there is no reliable way to
determine the actual identity of the file system artifact from
which the VM was loaded.

> * On Mac and Unix, can one read the executable while running Squeak? I
> need this to checksum the VM proper.

Yes, you can do what the shell would have done with argv[0], and you
can find the executable file that most probably might have been used
to provide the bytes that got loaded to produce the VM that is currently
running, and you can read that file once you have located it. But this
sounds like a rather shakey foundation for a security system.

By way of illustrating the general awfulness of this approach, the
following works on my system to get a file stream on the image file:

  (FileStream readOnlyFileNamed:
    (OSProcess accessor realpath:
      (ShellSyntax new
          findExecutablePathFor: (Smalltalk getSystemAttribute: 0)
          inDirectoryPath: nil)))

Seriously, this seems likely to attract crackers who might be
entertained by the challenge of breaking the "security", and it is
almost certain to be a source of annoyance to ordinary folks who will
not understand why their system does not work after it was installed
on whatever whacky network setup has been provided in their school
system.

> Besides the above three, I have one additional question: What is the
> general status of the 64bit work right now? Do we consider this complete
> or work in progress or experimental? Can we rule out any ill effects for
> the 32bit VMs?

In my experience, the 64 bit changes have no negative impact for a 32
bit VM. In general, they represent a tidying up of the system, and are
all for the good. There is a a small amount of added complexity for
address decoding, but it is quite localized in the source code. The
compiler warnings resulting from the 64 bit changes tend to be either
harmless, or to identify things that really do need to be cleaned up.

Clearly the 64 bit changes are a work in progress, and I am not in
a position to say if they will be "finished". But the work to date
is fundamentally sound, and I cannot think of any reason that it
should not move forward. FWIW, I'll help where I can.

Dave

Hi Dave -

That's too bad, but I guess it cannot be helped. Windows has the ability
to tell you the name of the binary that it loaded (be that an .exe or a
.dll) and that's what I was really hoping for.

>> * On Mac and Unix, can one read the executable while running Squeak? I
>> need this to checksum the VM proper.
>
> Yes, you can do what the shell would have done with argv[0], and you
> can find the executable file that most probably might have been used
> to provide the bytes that got loaded to produce the VM that is currently
> running, and you can read that file once you have located it. But this
> sounds like a rather shakey foundation for a security system.

It's not for security. It's really so that we know whether a VM has ever
been put to the test or not. In other words when the image starts, it
would check whether it has been started with an "unknown" VM and if so,
runs its VM tests and reports to the user if there are any issues. The
thing is that some of these tests take some time and rather than running
them anew upon each start of the image I'd like to be able to checksum
the VM and run them only when we detect that the VM has changed.

Cheers,
   - Andreas

On 3/17/06, Andreas Raab <[hidden email]> wrote:
> The thing is that some of these tests take some time and rather than running
> them anew upon each start of the image I'd like to be able to checksum
> the VM and run them only when we detect that the VM has changed.
>
Apart from the problems with reproducing the shell path lookup (doable
and there's bound to float code for that on the Net), the real killer
here, as was remarked, is that admins may mark the executable as
execute only. Without the read bit set on the file, you can't do the
checksumming. It smells like a hack, too - it'd be much cleaner just
to have a UUID generated on each build, that would *exactly* represent
what you are looking for and would be relatively simple to implement
on all platforms (except for wxSqueak, probably, it using MSVC to be
built but that's something that needs to be solved as well :-))

Am 17.03.2006 um 05:35 schrieb Andreas Raab:

Well, under most circumstances it *will* be the path to the  
executable. Like, if you run Croquet using the shell script that is  
in the Croquet distribution, which we could adapt to provide the  
whole path (or maybe it already does, haven't actually looked into it  
for a while).

Right. So if a user chooses some non-canonical way to run the VM that  
screws up system attribute 0, he just will have to wait longer.

OTOH, if you're building a new VM anyways using the new math lib,  
what's the problem with adding a primitive for checking? There are  
several ways in which the checksumming by the image could fail, or be  
slow. Like, the VM might be executable but not readable. Or it might  
sit on a slow network drive. And, are you sure check-summing the VM  
is sufficient, and you do not need to verify some external modules,  
too? There are people having all modules external.

- Bert -

> Besides the above three, I have one additional question: What is the
> general status of the 64bit work right now? Do we consider this complete
> or work in progress or experimental? Can we rule out any ill effects for
> the 32bit VMs?

I extent the question:

- Do the 64bit changes solve the problems with more than 2G of RAM?

Cheers,

-- Diego

On Fri, Mar 17, 2006 at 11:07:31AM +0100, Bert Freudenberg wrote:
Yes, having the VM identify itself through a primitive sounds like a
simpler and more reliable approach.

Dave