Is anyone else running a 64-bit image on a regular basis?

I tend to switch back and forth between a variety of VMs and images (Spur, Cog,
interpreter VM etc). Lately I have been doing my updates from a format 68002
64-bit image. Just curious, is anybody else out there using a 64-bit image on
a regular basis and keeping it updated from the trunk development stream?

Dave

On 16 July 2014 04:27, David T. Lewis <[hidden email]> wrote:
> I tend to switch back and forth between a variety of VMs and images (Spur, Cog,
> interpreter VM etc). Lately I have been doing my updates from a format 68002
> 64-bit image. Just curious, is anybody else out there using a 64-bit image on
> a regular basis and keeping it updated from the trunk development stream?

I'm not, and I really should, in CI. What would I need to do to set up
such a thing?

frank

> Dave

On Wed, Jul 16, 2014 at 09:14:20AM +0100, Frank Shearar wrote:
> On 16 July 2014 04:27, David T. Lewis <[hidden email]> wrote:
> > I tend to switch back and forth between a variety of VMs and images (Spur, Cog,
> > interpreter VM etc). Lately I have been doing my updates from a format 68002
> > 64-bit image. Just curious, is anybody else out there using a 64-bit image on
> > a regular basis and keeping it updated from the trunk development stream?
>
> I'm not, and I really should, in CI. What would I need to do to set up
> such a thing?
>

I was just mostly curious if anyone else is just using a 64 bit image on a day
to day basis.

As far as CI jobs on build.squeak.org, we would need to install the squeakvm64
VM on box3. I put the deb in the expected place but it is not installed.

  /root/localdebs/squeakvm64_20131020-1_i386.deb

Note that this deb is *only* for use on box3. If you want to run the 64-bit
image yourself, please use the official distribution on squeakvm.org/unix.

The trunk image is regularly being traced to 64 bits in the CI job at
http://build.squeak.org/job/Squeak%2064-bit%20image/ so the output of that
job should be usable.

Dave

On 16 July 2014 12:48, David T. Lewis <[hidden email]> wrote:
I'd probably build the 64-bit image from source: it makes things
easier for the other build agents, which (a) aren't on box3 and (b)
aren't always Linux. He says, knowing that all non-Linux build agents
are currently off-line...

The CI build scripts behind SqueakTrunk, ExternalPackages and so on
already build 32-bit interpreter VMs from source, so extending the
process to 64 should be easy.

> The trunk image is regularly being traced to 64 bits in the CI job at
> http://build.squeak.org/job/Squeak%2064-bit%20image/ so the output of that
> job should be usable.

Ah, handy. So I could build a 64 bit Interpreter from source, and just
pull in the output of the above job as the image to use?

frank

> Dave

On Wed, Jul 16, 2014 at 01:02:55PM +0100, Frank Shearar wrote:
There's nothing really to build, just use a SystemTracer to convert
whichever image you want to start with into 68002 64-bit format. You
can use the existing CI job for reference, but that's really all there
is to it. This is similar to what you're doing when you convert an
image to the new Spur format (image format 6521 for 32-bit Spur, and
image format 68019 for 64-bit Spur).

>
> > The trunk image is regularly being traced to 64 bits in the CI job at
> > http://build.squeak.org/job/Squeak%2064-bit%20image/ so the output of that
> > job should be usable.
>
> Ah, handy. So I could build a 64 bit Interpreter from source, and just
> pull in the output of the above job as the image to use?
>

Yes, just pass the --image64 parameter to the unix/cmake/configure script,
and you will have VM for 64-bit images. If you install it along with the
normal squeakvm VM, the /usr/local/bin/squeak script figures out the
right VM executable to use based on image format.

Dave

I tend to switch back and forth between a variety of VMs and images (Spur, Cog,
interpreter VM etc). Lately I have been doing my updates from a format 68002
64-bit image. Just curious, is anybody else out there using a 64-bit image on
a regular basis and keeping it updated from the trunk development stream?

Dave

On Fri, Jul 18, 2014 at 04:01:45PM -0300, Casimiro de Almeida Barreto wrote:
It is definitely slower than Cog. I recall from a few years ago (using whatever
PC I had at the time) that the 64-bit image ran slower than the 32-bit image.
However, on the PC that I am using now, the 64-bit image is somewhat faster than
the 32-bit image. I cannot explain the difference. I expect the 64-bit image
to be slower due to various inefficiencies in the use of sqInt as a default
data type throughout the VM (which is a 64 bit integer when the VM is compiled
for the 64-bit image). Maybe some CPUs handle this difference more efficiently
than others, I really don't know.


> b) Somethings just itch my ears like:
> b.1) Compiler recompileAll fails (and shouldn't)
> b.2) Smalltalk condenseChanges that works for 32bit images work, fails
> in 64bit image.

Running the full unit test suite will also lead to some unpleasantness.

>
> b.2 is not that important but b.1 shows that there is code that cannot
> be re-compiled and it's not a good sign.
>

That is not good, and it might be worthwhile to find the cause.

Dave

On Fri, Jul 18, 2014 at 04:01:45PM -0300, Casimiro de Almeida Barreto wrote:
I fixed a small but nasty bug in one of the primitives that appears to have
been the cause of quite a few problems:

 http://lists.squeakfoundation.org/pipermail/vm-dev/2014-September/016443.html

With freshly compiled VM, Compiler recompileAll and Smalltalk condenseChanges
are working again in a 64-bit image.

Dave

On 08.09.2014, at 08:35, Nicolas Cellier <[hidden email]> wrote:
>
> Ah, the beauty of C signed arithmetic model...
> We should try to avoid it as frequently as we can.

Indeed. Not just C though. In JavaScript I discovered that (42 << 32) == 42. Seriously.

- Bert -

On 9/8/14, Bert Freudenberg <[hidden email]> wrote:
> On 08.09.2014, at 08:35, Nicolas Cellier
> <[hidden email]> wrote:
>>
>> Ah, the beauty of C signed arithmetic model...
>> We should try to avoid it as frequently as we can.
>
> Indeed. Not just C though.
> In JavaScript I discovered that (42 << 32) == 42.

Firefox version 27
Menu Tools / Web Developer / WebConsole

paste
    (42 << 32) == 42.
into command line
CONFIRMED.

Thanks, Bert, for this info.

> Seriously.
>
> - Bert -
>
>
>
>

Thanks David for the fix. Very nice that the Smalltalk primitive can
just fail when the shift amount is too large.

Bert Freudenberg wrote:

>> Indeed. Not just C though.
>> In JavaScript I discovered that (42 << 32) == 42.

Thanks Bert.

Also confirmed on Google Chrome 36.0 (V8 3.26) and Node v0.10.

Right shift is the same (42 >> 32) == 42.
And (42 >> 35) == 5

Did you have to work around it when designing SqueakJS ? My guess is
not: I guess that the SqueakJS VM is a purely 32 bit machine, and that
this behaviour won't affect Smalltalk programs.

Have fun!
David

p.s.

I wonder if this is a restriction imposed by the language standard, or
is merely a side effect of a common implementation choice. The MDN
reference page says "Bitwise operators treat their operands as a
sequence of 32 bits".  Which, with hindsight, leaves shifts of more
than 31 bits undefined.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Bitwise_Operators

However, my naive reading of this paragraph of the ECMAScript 5
standard suggests that wrapping around a bit shift is just plain
wrong, and that it should yield 0 or fail (until a 64 bit Javascript
standard is written)
http://www.ecma-international.org/ecma-262/5.1/#sec-11.7

On Mon, Sep 08, 2014 at 08:35:55AM +0200, Nicolas Cellier wrote: format than have
Actually, in this case it was not related to signed arithmetic. It was a
word size issue involving a variable that took the default sqInt
declaration (which may be 64 bits) when it should have been explicitly
declared as a 32 bit variable in order to match the logic elsewhere in the
primitive. The shift operation itself was working fine, but the primitive
was trimming the result to 32 bits without failing the primitive.

Arguably the primitive should be updated to use a 64 bit variable so that
larger shift amounts will succeed. But I don't know if this might have
performance implications on various platforms, so I did not make that
change.

Dave

On 08.09.2014, at 16:45, David Corking <[hidden email]> wrote: It's in the ECMAScript standard. At least it's the same everywhere.

> Did you have to work around it when designing SqueakJS ?

Yes: https://github.com/bertfreudenberg/SqueakJS/commit/6d694d0733

> My guess is
> not: I guess that the SqueakJS VM is a purely 32 bit machine, and that
> this behaviour won't affect Smalltalk programs.

The VM needs to provide the bitShift primitive for SmallIntegers. That is, it needs to return a correct result if that fits into a SmallInt, and fail otherwise (it could also generate a LargeInt, but doesn't have to).

In Squeak, the single "bitShift:" methods shifts both left and right depending on the argument's sign, whereas in most other languages, you cannot shift by a negative amount. So there is one thing to watch out when implementing a VM.

I'm not really sure what happens in C, but in JS you get this nicety when shifting by a negative number:

        16 << -8 == 268435456

So the VM needs to handle negative shifts separately anyway, and in JS it needs to special-case shifts larger than 31, too.
They're well defined, they just operate on 32 bit signed values:

        1<<31 == -2147483648

That is, the result of shifts and bitwise operators is always done in 32 bits. But there's no direct way to detect overflow (even though other JS integer operations are accurate to 53 bits [because there are no actual integers]).

> However, my naive reading of this paragraph of the ECMAScript 5
> standard suggests that wrapping around a bit shift is just plain
> wrong, and that it should yield 0 or fail (until a 64 bit Javascript
> standard is written)
> http://www.ecma-international.org/ecma-262/5.1/#sec-11.7

Marvel at 11.7.1 step 7:

        • Let shiftCount be the result of masking out all but the least significant 5 bits of rnum, that is, compute rnum & 0x1F.

- Bert -

On 08.09.2014, at 18:28, Bert Freudenberg <[hidden email]> wrote:

> On 08.09.2014, at 16:45, David Corking <[hidden email]> wrote:
>
>> Did you have to work around it when designing SqueakJS ?
>
> Yes: https://github.com/bertfreudenberg/SqueakJS/commit/6d694d0733

Yikes. I just spotted another bug in that function. Can you see it, too?

(SmallIntegerTests did not uncover that bug, they all pass. Most tests assume a correct VM)

(IntegerTest and IntegerDigitLogicTest contain too many brute-force tests, which time out on a slow VM. SqueakJS passes 44 of 60)

(SqueakJS could use testers. But it's not much fun, opening a TestRunner in 4.5 takes like 30 seconds)

- Bert -

> On 08.09.2014, at 18:28, Bert Freudenberg <[hidden email]> wrote:
>
>> On 08.09.2014, at 16:45, David Corking <[hidden email]> wrote:
>>
>>> Did you have to work around it when designing SqueakJS ?
>>
>> Yes: https://github.com/bertfreudenberg/SqueakJS/commit/6d694d0733
>
> Yikes. I just spotted another bug in that function. Can you see it, too?

I think it should be using an unsigned right shift like this:

   if ((shifted>>>shiftCount) === smallInt) return shifted;

Dave

On 08.09.2014, at 19:44, David T. Lewis <[hidden email]> wrote:
No, "bitShift:" is supposed to be signed. This is my test if the result still fits in 32 bits, by reversing the shift. I *think* that part is correct ;)

E.g.
        -55 << 5 ==>  -1760
        -1760 >> 5 ==> -55

meaning we can use the result but

        -555555555 << 5 ==> -597908576
        -597908576 >> 5 ==> -18684643

so we cannot use the result because bits got dropped.

- Bert -