Unix 3.7 gamma VM

Folks,

I thought I'd put out a 'gamma' before releasing the final 3.7.  
Archives of binaries and sources are available from the (new) usual
place:

     http://squeak.hpl.hp.com/unix/

The same sources can be checked out from our Subversion repository, if
you prefer:

     svn co http://squeak.hpl.hp.com/svn/squeak/tags/unix-3.7g-6

If there are no show-stoppers in the next 24 hours or so I'll promote
it to 3.7-7 'final' and begin pulling in lots of new stuff for 3.8...

Thanks,
Ian

[I don't think this'll get to the list; it doesn't want to let me post most of the time]

On Thursday 17 March 2005 2:08 pm, Ian Piumarta wrote: A question on module loading:

1. How come it insists on loading external modules when it
has internal ones already loaded (see LargeIntegers, etc.) below:?


Smalltalk listLoadedModules #('FT2Plugin 17 March 2005 (e)' 'CairoPlugin 17 March 2005 (e)'
 'LargeIntegers v1.3 5 July 2004 (e)' 'Matrix2x3Plugin 5 July 2004 (e)' 'FloatArrayPlugin 5 July 2004 (e)'
 'B2DPlugin 5 July 2004 (e)' 'BitBltPlugin 5 July 2004 (e)' 'SecurityPlugin 5 July 2004 (e)' 'FilePlugin 5 July 2004 (e)'
 'MiscPrimitivePlugin 5 July 2004 (e)')

Smalltalk listBuiltinModules  #('B2DPlugin 16 March 2005 (i)' 'BitBltPlugin 16 March 2005 (i)'
 'ZipPlugin 16 March 2005 (i)' 'FileCopyPlugin 16 March 2005 (i)' 'FilePlugin 16 March 2005 (i)'
 'FloatArrayPlugin 16 March 2005 (i)' 'LargeIntegers v1.5 16 March 2005 (i)' 'Matrix2x3Plugin 16 March 2005 (i)'
 'MiscPrimitivePlugin 16 March 2005 (i)' 'RePlugin 16 March 2005 (i)' 'SecurityPlugin 16 March 2005 (i)'
 'SocketPlugin 16 March 2005 (i)')


I'm trying to run and debug from the directory just above the build directory:

#!/bin/sh
here=$PWD
build=bld
export SQUEAK_PLUGINS="${here}/${build}/%n/.libs/%n"
${here}/${build}/squeak  -plugins $SQUEAK_PLUGINS -vm-display-X11 -swapbtn -vm-sound-null "$@" squeak.image


I've attached the log from the sqUnixExternalPrims.c , if it'll help.

Thanks,
--
Ned Konz
http://bike-nomad.com/squeak/

> 1. How come it insists on loading external modules when it
> has internal ones already loaded (see LargeIntegers, etc.) below:?

Simple: Externals *always* override internals. If there is an external
module the system should use the external module.

Cheers,
   - Andreas

Ned Konz <[hidden email]> wrote:
> [I don't think this'll get to the list; it doesn't want to let me post most of the time]

:)

I am moderator of vm-dev and all posts that get stuck will end up in my
inbox and I let them through when I see them (and I see that they are
indeed "valid" posts). Somehow your subscription address was foobared
(which I emailed you about) but I fixed it so you should be fine now.

regards, Göran

Am 17.03.2005 um 23:08 schrieb Ian Piumarta:
The image segment loading bug is a show-stopper - changeset attached.

"Change Set:        imageSegmentFix-jl
Date:            18 March 2005
Author:            Jens Lincke

Fix loading of image segments as proposed by Dan I."

- Bert -

On Friday 18 March 2005 1:48 am, Bert Freudenberg wrote:
> The image segment loading bug is a show-stopper - changeset attached.

But I'm not sure that's all the problems we're seeing. For one thing, the
comparison

 oop < endOfMemory

when oop is < 2Gb and endOfMemory is >2Gb will fail. There are similar
problems throughout the code.

I have some changes that make endOfMemory, youngSpaceStart, etc. into
unsigned. I think they're more comprehensive, and still seem to work. You
might want to take a look at them.

For instance, here's the memory load information on the image I'm running
right now.

With the default arguments, it spans the 2Gb memory boundary, and works OK:

PID 16361:   /home/ned/Squeak/cairo/bld-i386/squeak
-plugins /home/ned/Squeak/cairo/bld-i386/%n/.libs/%n -vm-display-X11 -swapbtn
-vm-sound-null squeak.image
77656000-b7656000  1048576       -       -       - rw---    [ anon ]

With an explicit '-mmap 100M', all the segments are above 2Gb. Still works OK:

PID 16428:   /home/ned/Squeak/cairo/bld-i386/squeak
-plugins /home/ned/Squeak/cairo/bld-i386/%n/.libs/%n -vm-display-X11 -swapbtn
-vm-sound-null -mmap 100M squeak.image
b0875000-b1253000    10104       -       -       - rw---    [ anon ]
b1256000-b7656000   102400       -       -       - rw---    [ anon ]

You can get my VMMaker changes from SVN using Monticello:

MCHttpRepository
 location: 'http://squeak.hpl.hp.com/svn/squeak/branches/ned-branch/platforms'
 user: ''
 password: ''

--
Ned Konz
http://bike-nomad.com

On Mar 18, 2005, at 09:07, Ned Konz wrote:

> On Friday 18 March 2005 1:48 am, Bert Freudenberg wrote:
>> The image segment loading bug is a show-stopper - changeset attached.
>
> But I'm not sure that's all the problems we're seeing.

Fixing it a little might be better than not fixing it at all.

Bert: does applying this changeset turn a 3.7-5898 VM that you cannot
use into a VM that you can use?

Cheers,
Ian

In message <[hidden email]>
          Ned Konz <[hidden email]> wrote:

Exactly the point I was trying to make a week ago; did anyone compile
and try the simple testcase I included?
>
> I have some changes that make endOfMemory, youngSpaceStart, etc. into
> unsigned. I think they're more comprehensive, and still seem to work. You
> might want to take a look at them.
Will do. I'm trying to make some progress on the 64bit stuff anyway.

tim
--
Tim Rowledge, [hidden email], http://sumeru.stanford.edu/tim
Useful random insult:- Sat under the ozone hole too long.

Am 18.03.2005 um 18:13 schrieb Ian Piumarta:

Yes. Project loading will fail on two of our Linux machines without
this, now it works. The -mmap workaround only worked on one machine,
with the CS applied it isn't even necessary. So, if all agree that it
cannot hurt, it definitely helps in our case (which also was reported a
few times by other Linux users).

- Bert -

On Thursday 17 March 2005 2:08 pm, Ian Piumarta wrote:
What VMMaker version does that correspond to?

I see significant differences (for instance, in interp.c and in the
LargeIntegersPlugin) between the sources generated from VMMaker-tpr.5.mcz (30
Mar 2004) on SqueakSource (repository at http://kilana.unibe.ch:8888/VMMaker)
and the generated sources in your src directory.

Where is the Squeak source code available if someone wants to work on the VM?
I can't find it on SVN or in your tarball.

In my opinion, this is why the VMMaker sources (the Squeak code, possibly in
MCZ format) should be versioned along with the C sources in SVN.

BTW, it works fine to read MCZ files directly from the SVN repository; commits
would be done in the usual way (save the mcz to the working directory and do
a commit).

Attached are the diffs I found.

--
Ned Konz
http://bike-nomad.com

> Yes. Project loading will fail on two of our Linux machines without
> this, now it works. The -mmap workaround only worked on one machine,
> with the CS applied it isn't even necessary. So, if all agree that it
> cannot hurt, it definitely helps in our case (which also was reported a
> few times by other Linux users).

Let me second this. This is a *major* issue for Squeakland-people
running on Linux.

Cheers,
   - Andreas

Hi Bert,

On Mar 18, 2005, at 10:32, Bert Freudenberg wrote:
>> Bert: does applying this changeset turn a 3.7-5898 VM that you cannot
>> use into a VM that you can use?
>
> Yes.

Good enough for me.

Ian

In message <[hidden email]>
          Ned Konz <[hidden email]> wrote:

>
> What VMMaker version does that correspond to?
>
> I see significant differences (for instance, in interp.c and in the
> LargeIntegersPlugin) between the sources generated from VMMaker-tpr.5.mcz (30
> Mar 2004) on SqueakSource (repository at http://kilana.unibe.ch:8888/VMMaker)
> and the generated sources in your src directory.
Whoah, that code is massively out of date; I was never able to get a
decent working setup that would save there, then they had problems and
I had network problems and then I guess I simply forgot about it. The SM
version is the latest for public consumption.  Hmm, I'm up to version
13 in my local repository which is (I think) equivalent to
VMMaker3-8b2.1.cs from SM.

Right now I'm spending a little time trying to integrate the 64bit
stuff and then I'll look at your changes to properly type the main oop
values which should be a good step to fixing the >2Gb problems. It
might save me some time if you could cast those changes in terms of the
sqLong/sqInt etc #defines from the 64bit changes.


tim
--
Tim Rowledge, [hidden email], http://sumeru.stanford.edu/tim
Performance is easier to add than clarity.

On Mar 18, 2005, at 10:40, Ned Konz wrote:

> On Thursday 17 March 2005 2:08 pm, Ian Piumarta wrote:
>
>>      svn co http://squeak.hpl.hp.com/svn/squeak/tags/unix-3.7g-6
>
> What VMMaker version does that correspond to?

The one in 3.7-5898-full.

1.  It's a 3.7 VM, so the rule says it has to be built from a 3.7
image.  The final 3.7 VM is built from the final 3.7-full image.
2.  Look in configure.ac (or watch the first few lines printed by
configure as it runs) and it tells you exactly which image version
generated the (bundled) interpreter sources.

> I see significant differences (for instance, in interp.c and in the
> LargeIntegersPlugin) between the sources generated from
> VMMaker-tpr.5.mcz (30
> Mar 2004) on SqueakSource (repository at
> http://kilana.unibe.ch:8888/VMMaker)
> and the generated sources in your src directory.

I regenerated everything from scratch using the final 3.7 image.  
Here's what I do:

     run the latest X.Y-Z image
     click on 'update'
     save the image iff anything VM-related was updated
     'clear out' the 'src' directory
     'generate all' into the empty 'src' directory
     quit
     type 'make clean; make' to build a VM from scratch
     type 'make dist-src dist-image dist-bin' to archive and release the
new VM sources and binary (for the local platform) along with the image
that generated it

IOW, whatever is in the latest 3.7 image is definitive for building a
3.7 VM.  This is not negotiable.  (Other than fixing show-stoppers like
the image segment thing, given the fix is so trivial, but then I would
also file that fix into my 3.7-5898-full.image before distributing it
in a tarball, as above.)

If the released 3.7 full image is generating broken Interpreter or
plugins, then either the released image should be fixed (or changes
posted to its update stream), or they should be in the 3.8 stream and
the 3.8 VM declared the minimum version suitable for anyone
encountering problems with 3.7.

> Where is the Squeak source code available if someone wants to work on
> the VM?

3.7-5898-full.image on the FTP site of your choice (given that nothing
at all happened when I pressed 'update' in the process described
earlier).

> I can't find it on SVN or in your tarball.

It's on my download page (3.7-5898 image and changes tarball, which
contains the Interpreter sources used to generate the core VM
corresponding to the released sources, always).

I'm not about to check images into the repostory.  OTOH, my generated
'src' tree is now included in the unix area within the repository
(platforms/unix/src), to make life easier for anyone building with
those sources.  (configure now understands to look in two places for
'src', and to use the one nearest the top of the hierarchy --
corresponding to sources generated outside of the 'platforms/unix'
tree, which is where most people will be generating them.)

> In my opinion, this is why the VMMaker sources (the Squeak code,
> possibly in
> MCZ format) should be versioned along with the C sources in SVN.

The VM sources are stamped with the image version to which they apply
(see above re: configure).  You can ignore this if you really know what
you're doing, but you get to keep all the pieces afterwards.

> Attached are the diffs I found.

Thanks -- but I don't think I can do much with them.  If they aren't in
the 3.7 full image (or updates) then they aren't relevant to building a
3.7 VM.  When 3.8 becomes current later today (hopefully), the
Interpreter therein will be generated from the most recent 3.8 image
and its update stream.

Hope this helps to clear up the process for anyone who wasn't already
familiar with it.

Ian

On Mar 18, 2005, at 11:13, Tim Rowledge wrote:

> might save me some time if you could cast those changes in terms of the
> sqLong/sqInt etc #defines from the 64bit changes.

Wherever you see 'unsigned int' in Ned's changes attached to something
that can be interpreted as an object pointer, use 'usqInt' instead in
64-bit compatible code.

[u]sqInt always describes an integer of the same width as an oop in the
generated sources.  This is described in detail in section 4 of
http://squeak.hpl.hp.com/squeak64/.  If you aren't intimately familiar
with the contents of section 4 of that document, you are in very
dangerous territory trying to pull the 64-bit changes I made to VMM and
Interpreter forward into more recent 32-bit versions of the same.

Ian

On Friday 18 March 2005 11:16 am, Ian Piumarta wrote:
> > What VMMaker version does that correspond to?
>
> The one in 3.7-5898-full.
>
> 1.  It's a 3.7 VM, so the rule says it has to be built from a 3.7
> image.  The final 3.7 VM is built from the final 3.7-full image.
> 2.  Look in configure.ac (or watch the first few lines printed by
> configure as it runs) and it tells you exactly which image version
> generated the (bundled) interpreter sources.

Great. I'll apply my changes to it instead.

Thanks,
--
Ned Konz
http://bike-nomad.com

On Mar 18, 2005, at 11:39, Ned Konz wrote:

> On Friday 18 March 2005 11:16 am, Ian Piumarta wrote:
>>> What VMMaker version does that correspond to?
>>
>> The one in 3.7-5898-full.
>
> Great. I'll apply my changes to it instead.

Double great!  If you (or anyone) then confirm that they fix the image
segment loading problem seen by Bert and the Squeaklanders, they'll be
in my final released source/binary VM and image archives.  (Provided
Bert sees no problem with using them instead of Jens' changeset.)

Is it too late to post these changes to the 3.7 update stream?

Cheers,
Ian

In message <[hidden email]>
          Ian Piumarta <[hidden email]> wrote:
Not so much too late as irrelevant. They are VMMaker changes and would
go into a re-released VMMaker package on SM.


tim
--
Tim Rowledge, [hidden email], http://sumeru.stanford.edu/tim
Do you like me for my brain or my baud?

On Mar 18, 2005, at 11:51, Tim Rowledge wrote:

>> Is it too late to post these changes to the 3.7 update stream?
> Not so much too late as irrelevant. They are VMMaker changes and would
> go into a re-released VMMaker package on SM.

They would be very, very relevant to anyone trying to make a working VM
from a 3.7 image and the 3.7 final VM sources.  (Including me.  I
normally would refuse to include anything not in the image/update
stream when building a release VM.)

As I understand it, this is just the point Ned was trying to make and
address.  The more places you have to look for particular versions of
stuff (SVN source with tag X.Y + SqF image version X.Y-full + SM or
other source for VMM package version P.Q + updates through but not
passing ABCD + changeset posted to vm-dev on 2005-03-18 by Anonymouse
Squeaklander + etc...) to make a compatible whole, the harder it is for
anyone and everyone to recreate a VM from scratch.

Just saying "get all the latest stuff" doesn't really work if you're
trying to recreate a particular version of the VM.

The simplest useful and scalable thing I can imagine is to say: VM X.Y
"final" can be recreated verbatim from scratch by using X.Y final
platform sources + X.Y final image + any big fixes subsequently
retrofitted to its Interpreter through the X.Y update stream.

Ian

Hang on mo'; it's just finally sunk in that you're talking about
version 3-point-frigging-seven here. That is _so_ last year. June last
year in fact.

tim
--
Tim Rowledge, [hidden email], http://sumeru.stanford.edu/tim
Oxymorons: Religious tolerance