VM crash using #markReadOnly (9.1)

Hello folks,

I found a simple way to create a GPF which should not happen:

(1->2) markReadOnly: true; value: 3

Happens in 9.1, we did not upgrade our application to 9.2 yet.

Cheers,
Hans-Martin

I'm not able to repeat that in our latest 9.2.1....I get an attemptedROMStore prim error.
I don't see a specific case for that fix but I suspect it was corrected during the just-in-time

compiler work and bugfixing that went into the 9.2 vm.

- Seth

On Tuesday, March 31, 2020 at 10:03:35 AM UTC-4, Hans-Martin Mosner wrote:

Hello folks,

I found a simple way to create a GPF which should not happen:

(1->2) markReadOnly: true; value: 3

Happens in 9.1, we did not upgrade our application to 9.2 yet.

Cheers,
Hans-Martin

Thanks a lot for your report. I tried this on all below combinations:

8.6.3 (x86 only)    OK
9.0 x86   OK
9.0 x64   crash
9.1 fixpack x86 OK
9.1 fixpack x64 crash
9.2.1 x86 OK
9.2.1 x64 OK 

My conclusion is that this was a bug with the new VM (started with 9.0), only in 64 bit version, and was fixed in 9.2. With that in mind, I see two workaround for you. The first one, is to just use 32 bits (please confirm if looks fixed for you too). The second one, below. 

We have the instructions on hand as most folks are moving up quickly, especially since the performance is 2-3x what 9.1 due to the Just-In-Time Compiler that is now included. 

-------------------------------------------------------------------------------------------
INSTUCTIONS TO RUN 9.2.1 VM ON A 9.1 INSTALLATION/IMAGE
---------------------------------------------------------------------------------------------

The steps I am explaining below are for a "development" image. But if you are packaging a runtime, you will probably apply similar steps to it.

0) Be sure to use same bitness of both VASTs (would be easier), that is, either x86 or x64. I assume you are in 64 bits.

1) I would make a backup of your current 9.1 installation just in case, before start trying to update it.

2) Install 9.2.1 with the installer. That will install it in is own path and will perfectly co-exist with your previous 9.1. You can even use this installation if you want to play/experiment/test with 9.2.1 VM AND IMAGE.

3) From 9.2.1 VAST ROOT (say c:\Program Files\Instantiations\VA Smalltalk\9.2.1x64\) copy and replace all these files to 9.1 VAST root:

*.exe , *.dll, *.bin

4) You will also need to download an abtpay92.dll from our website. You should copy this one too over VAST 9.1 ROOT.

5) Edit your VAST 9.1 .ini and change "Abt_Primitives" variable to point to 9.2 dll. Example:

Abt_Primitives=abtprc92

6) Note that esvm40.bin is a new file and that is what makes the Just in Time Compiler to work. If you don't have this file, you won't have a JIT. Remember to copy this over into a packaged runtime image if you do have that. 

---------------------------------------------------------------------------------------------

Hope this helps,

I was also checking our > 16k Unit Test product suite to see why that wasn't covered. It seems the problem is not only x64 bit only but also only with "instVar" storage... which is the case you shared:

(1->2) markReadOnly: true; value: 3

But other examples, such as indexed, do work correctly:

 (Array new: 10)
markReadOnly: true;
at: 1 put: 2

I am adding now your example and a few more tests...

Thanks for reporting! 

Mariano, thank you very much! Sory I didn't mention 64 bit...

Since this problem only manifests in our development environment where we use immutability to catch modifications outside of database transactions, but not in production, we can wait for 9.2 which we will probably adopt during this year. Better than going back to 32 bit in our case for various reasons.

Cheers,
Hans-Martin

Weird, I just downloaded 9.2.1 64 bit (version from 2020-03-09) and it still has the problem.

VMTRAP.LOG claims it's using the right VM DLL, I have also tried to run it under dependency walker to see which DLLs are actually used, and it does not seem to be contaminated by 9.1 DLLs.

This GPF happens sometime after the code has been executed (instantaneous as seen from a user perspective, but the stack is showing debugger activity already, this is somewhat variable between test runs).

Cheers,
Hans-Martin

Can you tell me exactly what you are executing and it what context.
For example, debugging an expression vs executing the expression uses different vm code, so this small distinction is important.
And it's still the expression: (1->2) markReadOnly: true; value: 3

I know its delayed a bit which most often means the memory is corrupted but you won't know until the next GC when memory is walked.
But is it repeatable for you after a single evaluation of the expression...(or you could try single evaluation and manual System globalGarbageCollect to force object walking) or is it more random?

Thanks for any info you can provide.

- Seth

On Thursday, April 2, 2020 at 4:32:27 AM UTC-4, Hans-Martin Mosner wrote:

Am Dienstag, 31. März 2020 21:46:32 UTC+2 schrieb Mariano Martinez Peck:

BTW.... I can confirm this was fixed on Jan 07 2019 as part of the JIT work of 9.2.

Weird, I just downloaded 9.2.1 64 bit (version from 2020-03-09) and it still has the problem.

VMTRAP.LOG claims it's using the right VM DLL, I have also tried to run it under dependency walker to see which DLLs are actually used, and it does not seem to be contaminated by 9.1 DLLs.

-Platform Information-------------------
VM Timestamp: 4.0, 03/03/20 (102)
CPU Architecture: amd64
OS Type: WIN32_NT
OS Version: 6.1 build 7601 Service Pack 1
-State Information----------------------
Handler=0xcb809200
vmState=0x00000000
vmError=0x00000035
ExceptionCode=0xc0000005
ExceptionAddress=0xcb7e8420
ContextFlags=0x0010005f
Module=C:\Program Files\Instantiations\VA Smalltalk\9.2.1x64\esvm40.dll
Module_base_address=0xcb7b0000
Offset_in_DLL=0x00038420
RDI=0x0014ffc0
RSI=0x00000000
RAX=0x2af48030
RBX=0x07d8ef10
RCX=0x00000000
RDX=0x05fd0e60
RBP=0x07d8ef50
RSP=0x002cf3f0
RIP=0xcb7e8420

This GPF happens sometime after the code has been executed (instantaneous as seen from a user perspective, but the stack is showing debugger activity already, this is somewhat variable between test runs).

Cheers,
Hans-Martin

Hi Seth, this happens reproducibly when I directly execute that statement in the Transcript of a newly opened 9.2.1 abt.icx image (after confirming that I'm the Library Supervisor and letting the image recache its method pointers of course).

The image is still started as a trial version because the colleague who should have received the abtpay.dll probably still has it in her mailbox, but that should not make a difference for this behavior I suppose :-)

Cheers,

Hans-Martin

I am in the same boat as Seth, I cannot reproduce it even on a fresh 9.2.1 install, evaluating that snippet from a Transcript. 

Is that a fresh 9.2.1 installation that you are trying?  The abt.ini being used is the shipped/unmodified one?

Thanks!

I have 2 additional questions you could help me with (much appreciated btw).

1. If you pass the -mcd command line parameter (no-jit), can you still reproduce this?

2. Can you give me basic details for the machine in question:
- Operating system version and bitness
- CPU model

These would be very helpful.

Thanks

- Seth

On Thursday, April 2, 2020 at 9:09:33 AM UTC-4, Hans-Martin Mosner wrote:

Am Donnerstag, 2. April 2020 13:08:52 UTC+2 schrieb Seth Berman:

Greetings,

Can you tell me exactly what you are executing and it what context.
For example, debugging an expression vs executing the expression uses different vm code, so this small distinction is important.
And it's still the expression: (1->2) markReadOnly: true; value: 3

I know its delayed a bit which most often means the memory is corrupted but you won't know until the next GC when memory is walked.
But is it repeatable for you after a single evaluation of the expression...(or you could try single evaluation and manual System globalGarbageCollect to force object walking) or is it more random?

Thanks for any info you can provide.

Hi Seth, this happens reproducibly when I directly execute that statement in the Transcript of a newly opened 9.2.1 abt.icx image (after confirming that I'm the Library Supervisor and letting the image recache its method pointers of course).

The image is still started as a trial version because the colleague who should have received the abtpay.dll probably still has it in her mailbox, but that should not make a difference for this behavior I suppose :-)

Cheers,

Hans-Martin

Yes, just unpacked the 64 bit install and manager so the image has a repository to use.

You're also using the version from 9th of  March (the one on the website) or a newer one?

I can't currently run the example as my manager.dat is locked by an importer process running over WAN due to COVID-19, so this will take some more hours...

Cheers,

Hans-Martin

Hello

it's reproducible with VAST 9.2.1 64 bit (Windows 8.1 PRO and Windows 10), vmtrap files uploaded to VAST support

Norbert

Thanks for the inputs.
Ok, so I can not reproduce with JIT enabled.
I can always reproduce if I disable the JIT with -mcd switch.
So, we'll see what the interpreter is doing wrong.
Getting closer:)

- Seth

On Thursday, April 2, 2020 at 11:21:56 AM UTC-4, Norbert Schlemmer wrote:

Hello

it's reproducible with VAST 9.2.1 64 bit (Windows 8.1 PRO and Windows 10), vmtrap files uploaded to VAST support

Norbert

We know why now, but we don't have a good sense of overall impact.

There is actually a bit of magic at play here, so it's not obvious.
I'll just try to explain as briefly as I can

1. Association>>value: uses a special primitive call (marked in the 3rd byte of the compiled method, not a prim call like you would see it in the source code)

    for setting inst vars efficiently.  (VMprSetInstVarWP or VMprMBASetInstVarWP)
2. The smalltalk compiler does this in CodeStream>>inlineSetter and its able to to effectively do all those push/pops you see in inlineSetter in one prim call without interpreter overhead.

3. The code you see in Association>>value: code browser  ... under most circumstances that is never run during normal execution.

4. Unless, VMprSetInstVarWP triggers a primitive failure ... in which case the code you see in the code browser is the fallback code that will be executed.
5. Why would VMprSetInstVarWP trigger a primitive failure ... how about a failed read-only object check (how this whole discussion got started).

6. When the prim failure occurs, it returns and starts executing the code from the bytecodes of the Association>>value: compiled method.

7. Unfortunately, when this got run through the 64-bit autoconverter, it didn't fix it up correctly.

8. The byte at position 6 in the bytecode array should be an 8 on 64-bit...not a 4.

9. The smalltalk compiler in the image can do it right...touch the method and save....now its correct.

We are reviewing the portion of the vm that must fixup 32-bit bytecodes into their 64-bit equivalents to see how far reaching.

I suspect its pretty isolated to this, and possibly a few others, since this has been true since 9.0 back in 2017 and this is the first we are hearing about it.

Naturally, we will get to work fixing it up.

Thanks to all for bringing this forward.

- Seth

On Thursday, April 2, 2020 at 11:30:55 AM UTC-4, Seth Berman wrote:

Hello,

Thanks for the inputs.
Ok, so I can not reproduce with JIT enabled.
I can always reproduce if I disable the JIT with -mcd switch.
So, we'll see what the interpreter is doing wrong.
Getting closer:)

- Seth

On Thursday, April 2, 2020 at 11:21:56 AM UTC-4, Norbert Schlemmer wrote:

Hello

it's reproducible with VAST 9.2.1 64 bit (Windows 8.1 PRO and Windows 10), vmtrap files uploaded to VAST support

Norbert

And, btw, the Just-In-Time compiled form of this will hide the problem indeed.

The reason is even more complicated to explain, but I'll just say that most all of the time

it will put something that looks like a valid SmallInteger in the low 4 bytes of a certain location that would be walked by the memory manager.
While the interpreter will put all 0s (due to bad offset)..which does not look like a valid object and why this breaks later during a GC.

In either case, these values are not important due to the read-only prim fail.

But, important from the standpoint that it will break your program if it doesn't look like an object.

- Seth

On Friday, April 3, 2020 at 12:05:25 PM UTC-4, Seth Berman wrote:

Hi all,

We know why now, but we don't have a good sense of overall impact.

There is actually a bit of magic at play here, so it's not obvious.
I'll just try to explain as briefly as I can

1. Association>>value: uses a special primitive call (marked in the 3rd byte of the compiled method, not a prim call like you would see it in the source code)

    for setting inst vars efficiently.  (VMprSetInstVarWP or VMprMBASetInstVarWP)
2. The smalltalk compiler does this in CodeStream>>inlineSetter and its able to to effectively do all those push/pops you see in inlineSetter in one prim call without interpreter overhead.

3. The code you see in Association>>value: code browser  ... under most circumstances that is never run during normal execution.

4. Unless, VMprSetInstVarWP triggers a primitive failure ... in which case the code you see in the code browser is the fallback code that will be executed.
5. Why would VMprSetInstVarWP trigger a primitive failure ... how about a failed read-only object check (how this whole discussion got started).

6. When the prim failure occurs, it returns and starts executing the code from the bytecodes of the Association>>value: compiled method.

7. Unfortunately, when this got run through the 64-bit autoconverter, it didn't fix it up correctly.

8. The byte at position 6 in the bytecode array should be an 8 on 64-bit...not a 4.

9. The smalltalk compiler in the image can do it right...touch the method and save....now its correct.

We are reviewing the portion of the vm that must fixup 32-bit bytecodes into their 64-bit equivalents to see how far reaching.

I suspect its pretty isolated to this, and possibly a few others, since this has been true since 9.0 back in 2017 and this is the first we are hearing about it.

Naturally, we will get to work fixing it up.

Thanks to all for bringing this forward.

- Seth

On Thursday, April 2, 2020 at 11:30:55 AM UTC-4, Seth Berman wrote:

Hello,

Thanks for the inputs.
Ok, so I can not reproduce with JIT enabled.
I can always reproduce if I disable the JIT with -mcd switch.
So, we'll see what the interpreter is doing wrong.
Getting closer:)

- Seth

On Thursday, April 2, 2020 at 11:21:56 AM UTC-4, Norbert Schlemmer wrote:

Hello

it's reproducible with VAST 9.2.1 64 bit (Windows 8.1 PRO and Windows 10), vmtrap files uploaded to VAST support

Norbert

Seth,

Thanks for being so open and explaining this! It's really appreciated and rarely done by commercial operations. (Never done???)

Thanks Richard, 

My pleasure and I hope the information is helpful.

- Seth

On Friday, April 3, 2020 at 3:22:39 PM UTC-4, Richard Sargent wrote:

Seth,

Thanks for being so open and explaining this! It's really appreciated and rarely done by commercial operations. (Never done???)

On Friday, April 3, 2020 at 10:44:50 AM UTC-7, Seth Berman wrote:

Hi All,

And, btw, the Just-In-Time compiled form of this will hide the problem indeed.

The reason is even more complicated to explain, but I'll just say that most all of the time

it will put something that looks like a valid SmallInteger in the low 4 bytes of a certain location that would be walked by the memory manager.
While the interpreter will put all 0s (due to bad offset)..which does not look like a valid object and why this breaks later during a GC.

In either case, these values are not important due to the read-only prim fail.

But, important from the standpoint that it will break your program if it doesn't look like an object.

- Seth

On Friday, April 3, 2020 at 12:05:25 PM UTC-4, Seth Berman wrote:

Hi all,

We know why now, but we don't have a good sense of overall impact.

There is actually a bit of magic at play here, so it's not obvious.
I'll just try to explain as briefly as I can

1. Association>>value: uses a special primitive call (marked in the 3rd byte of the compiled method, not a prim call like you would see it in the source code)

    for setting inst vars efficiently.  (VMprSetInstVarWP or VMprMBASetInstVarWP)
2. The smalltalk compiler does this in CodeStream>>inlineSetter and its able to to effectively do all those push/pops you see in inlineSetter in one prim call without interpreter overhead.

3. The code you see in Association>>value: code browser  ... under most circumstances that is never run during normal execution.

4. Unless, VMprSetInstVarWP triggers a primitive failure ... in which case the code you see in the code browser is the fallback code that will be executed.
5. Why would VMprSetInstVarWP trigger a primitive failure ... how about a failed read-only object check (how this whole discussion got started).

6. When the prim failure occurs, it returns and starts executing the code from the bytecodes of the Association>>value: compiled method.

7. Unfortunately, when this got run through the 64-bit autoconverter, it didn't fix it up correctly.

8. The byte at position 6 in the bytecode array should be an 8 on 64-bit...not a 4.

9. The smalltalk compiler in the image can do it right...touch the method and save....now its correct.

We are reviewing the portion of the vm that must fixup 32-bit bytecodes into their 64-bit equivalents to see how far reaching.

I suspect its pretty isolated to this, and possibly a few others, since this has been true since 9.0 back in 2017 and this is the first we are hearing about it.

Naturally, we will get to work fixing it up.

Thanks to all for bringing this forward.

- Seth

On Thursday, April 2, 2020 at 11:30:55 AM UTC-4, Seth Berman wrote:

Hello,

Thanks for the inputs.
Ok, so I can not reproduce with JIT enabled.
I can always reproduce if I disable the JIT with -mcd switch.
So, we'll see what the interpreter is doing wrong.
Getting closer:)

- Seth

On Thursday, April 2, 2020 at 11:21:56 AM UTC-4, Norbert Schlemmer wrote:

Hello

it's reproducible with VAST 9.2.1 64 bit (Windows 8.1 PRO and Windows 10), vmtrap files uploaded to VAST support

Norbert

Ah, that's really tricky, thanks for the detailed analysis! So just recompiling the affected methods works, that will be easy.

I'll write a short snipped to identify and recompile those affected methods.

Cheers,

Hans-Martin

Here's a short snippet that should do it.

It looks like normal methods aren't effected, only compact methods, so this only checks for primitive 194.

System allClasses do: [:cls |