[VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Paulo-27
Dear all,

after updating a running system to the version 11.10 of Ubuntu,
VisualWorks (64 bit) runs for a certain amount of time until it dies
after a memory access violation. Here are the last messages printed
by the "debug aware"-virtual machine:

lost time signal in waitForIOWithTimeout, milliseconds 2
lost time signal in waitForIO, milliseconds 0
lost time signal in waitForIO, milliseconds 0
Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
Speicherzugriffsfehler

The version of VisualWorks in question is:

dellani@KITT:~/$ visual -v
VisualWorks(R) 7.7.1 Jul  7 2010
Copyright (C) 1999-2009 Cincom Systems, Inc.
Release identification ...
       OE   version: 68 platform: 64 release: 68.15 flags: 0
       VI   version: 0 minor: 0 release: 0 variant: 0
       From version: 0 platform: 0 release: 0.1 flags: 0

and here are some details about the system itself:

dellani@KITT:~/$ uname -a
Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
x86_64 x86_64 x86_64 GNU/Linux

I tried the 32-bits virtual machine on the same system and it just runs fine!
Anyone experiencing these same problems?

Cheers, Paulo
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Andres Valloud-6
We are aware of these problems in Fedora, but this is the first time I
hear about this under Ubuntu.  We would like to take care of this for
the next release.

On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:

> Dear all,
>
> after updating a running system to the version 11.10 of Ubuntu,
> VisualWorks (64 bit) runs for a certain amount of time until it dies
> after a memory access violation. Here are the last messages printed
> by the "debug aware"-virtual machine:
>
> lost time signal in waitForIOWithTimeout, milliseconds 2
> lost time signal in waitForIO, milliseconds 0
> lost time signal in waitForIO, milliseconds 0
> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
> Speicherzugriffsfehler
>
> The version of VisualWorks in question is:
>
> dellani@KITT:~/$ visual -v
> VisualWorks(R) 7.7.1 Jul  7 2010
> Copyright (C) 1999-2009 Cincom Systems, Inc.
> Release identification ...
>         OE   version: 68 platform: 64 release: 68.15 flags: 0
>         VI   version: 0 minor: 0 release: 0 variant: 0
>         From version: 0 platform: 0 release: 0.1 flags: 0
>
> and here are some details about the system itself:
>
> dellani@KITT:~/$ uname -a
> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
> x86_64 x86_64 x86_64 GNU/Linux
>
> I tried the 32-bits virtual machine on the same system and it just runs fine!
> Anyone experiencing these same problems?
>
> Cheers, Paulo
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Paulo-27
Any hint on what could be the cause of it? Just for the record, I tried
different versions of the Linux Kernel available in Ubuntu 11.10:

2.6.38-12-generic
3.0.0-12-generic
3.0.8-030008-generic
3.1.0-030100rc10-generic
3.1.0-999-generic

VisualWorks fails with all of them as well. May be I should try
another version of the GlibC,
before changing the complete system ... What do you mean with next
release, Andres? 7.8?

Cheers,

Paulo


2011/11/2 Andres Valloud <[hidden email]>:

> We are aware of these problems in Fedora, but this is the first time I
> hear about this under Ubuntu.  We would like to take care of this for
> the next release.
>
> On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:
>> Dear all,
>>
>> after updating a running system to the version 11.10 of Ubuntu,
>> VisualWorks (64 bit) runs for a certain amount of time until it dies
>> after a memory access violation. Here are the last messages printed
>> by the "debug aware"-virtual machine:
>>
>> lost time signal in waitForIOWithTimeout, milliseconds 2
>> lost time signal in waitForIO, milliseconds 0
>> lost time signal in waitForIO, milliseconds 0
>> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
>> Speicherzugriffsfehler
>>
>> The version of VisualWorks in question is:
>>
>> dellani@KITT:~/$ visual -v
>> VisualWorks(R) 7.7.1 Jul  7 2010
>> Copyright (C) 1999-2009 Cincom Systems, Inc.
>> Release identification ...
>>         OE   version: 68 platform: 64 release: 68.15 flags: 0
>>         VI   version: 0 minor: 0 release: 0 variant: 0
>>         From version: 0 platform: 0 release: 0.1 flags: 0
>>
>> and here are some details about the system itself:
>>
>> dellani@KITT:~/$ uname -a
>> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>> I tried the 32-bits virtual machine on the same system and it just runs fine!
>> Anyone experiencing these same problems?
>>
>> Cheers, Paulo
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Andres Valloud-6
So far I know somehow an nmethod is thrown away even though another
nmethod references it, so eventually the JIT jumps into code that has
been overwritten by something else.

It would be interesting to see if you can correlate versions of glibc
with the problem.

We'd like to see this solved for 7.9.

On 11/3/2011 7:52 AM, Paulo Roberto Dellani wrote:

> Any hint on what could be the cause of it? Just for the record, I tried
> different versions of the Linux Kernel available in Ubuntu 11.10:
>
> 2.6.38-12-generic
> 3.0.0-12-generic
> 3.0.8-030008-generic
> 3.1.0-030100rc10-generic
> 3.1.0-999-generic
>
> VisualWorks fails with all of them as well. May be I should try
> another version of the GlibC,
> before changing the complete system ... What do you mean with next
> release, Andres? 7.8?
>
> Cheers,
>
> Paulo
>
>
> 2011/11/2 Andres Valloud<[hidden email]>:
>> We are aware of these problems in Fedora, but this is the first time I
>> hear about this under Ubuntu.  We would like to take care of this for
>> the next release.
>>
>> On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:
>>> Dear all,
>>>
>>> after updating a running system to the version 11.10 of Ubuntu,
>>> VisualWorks (64 bit) runs for a certain amount of time until it dies
>>> after a memory access violation. Here are the last messages printed
>>> by the "debug aware"-virtual machine:
>>>
>>> lost time signal in waitForIOWithTimeout, milliseconds 2
>>> lost time signal in waitForIO, milliseconds 0
>>> lost time signal in waitForIO, milliseconds 0
>>> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
>>> Speicherzugriffsfehler
>>>
>>> The version of VisualWorks in question is:
>>>
>>> dellani@KITT:~/$ visual -v
>>> VisualWorks(R) 7.7.1 Jul  7 2010
>>> Copyright (C) 1999-2009 Cincom Systems, Inc.
>>> Release identification ...
>>>          OE   version: 68 platform: 64 release: 68.15 flags: 0
>>>          VI   version: 0 minor: 0 release: 0 variant: 0
>>>          From version: 0 platform: 0 release: 0.1 flags: 0
>>>
>>> and here are some details about the system itself:
>>>
>>> dellani@KITT:~/$ uname -a
>>> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> I tried the 32-bits virtual machine on the same system and it just runs fine!
>>> Anyone experiencing these same problems?
>>>
>>> Cheers, Paulo
>>> _______________________________________________
>>> vwnc mailing list
>>> [hidden email]
>>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>>
>> _______________________________________________
>> vwnc mailing list
>> [hidden email]
>> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>>
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Paulo-27
Dear all,

I was not only able to isolate the version of glibc with which
VisualWorks exhibits this behavior, but found its possible cause
and one workaround, so that the 64-bits version of the virtual
machine can run on the affected systems (at least at the Debian-
based ones).

Beginning with the version 2.13 of the GlibC, an optimized version
of memcpy() is provided on the amd64 architecture. According to
Aurelien Jarno, one of the maintainers of the GlibC Debian package,

"this version uses SSSE3 optimizations and might copy memory
backwards in some cases, which causes issues if the source and
destination overlap. memmove() should be used in such cases, but
some programs still wrongly use memcpy()."

This is the case, because by following the instructions gave by him to
debug this issue, I was able to track it back in Visualworks.

Here are the general instructions by Aurelien Jarno:

"For this reason, on the amd64 architecture the Debian package provides
  two wrappers which can be use to workaround and/or debug the issue:
  - /usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so simply replace all
    calls to memcpy() by a call to memmove()
  - /usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so does the same,
    but in addition logs (with rate limit) the issue to syslog, so that it
    can be detected and fixed.

  To use these wrapper on a single binary, the easiest way is to use the
  LD_PRELOAD environment variable:
  - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so /path/to/binary
  - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so
/path/to/binary"

So entering the following prior to starting VisualWorks "solved" the problem:

export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so
/usr/local/vw7.7.1nc/bin/linuxx86_64/visual"

If you want to see debug messages in the system syslog while the workaround does
its job, replace "memcpy-preload.so" by "memcpy-syslog-preload.so" in the above
command.

Cheers,

Paulo

2011/11/4 Andres Valloud <[hidden email]>:

> So far I know somehow an nmethod is thrown away even though another
> nmethod references it, so eventually the JIT jumps into code that has
> been overwritten by something else.
>
> It would be interesting to see if you can correlate versions of glibc
> with the problem.
>
> We'd like to see this solved for 7.9.
>
> On 11/3/2011 7:52 AM, Paulo Roberto Dellani wrote:
>> Any hint on what could be the cause of it? Just for the record, I tried
>> different versions of the Linux Kernel available in Ubuntu 11.10:
>>
>> 2.6.38-12-generic
>> 3.0.0-12-generic
>> 3.0.8-030008-generic
>> 3.1.0-030100rc10-generic
>> 3.1.0-999-generic
>>
>> VisualWorks fails with all of them as well. May be I should try
>> another version of the GlibC,
>> before changing the complete system ... What do you mean with next
>> release, Andres? 7.8?
>>
>> Cheers,
>>
>> Paulo
>>
>>
>> 2011/11/2 Andres Valloud<[hidden email]>:
>>> We are aware of these problems in Fedora, but this is the first time I
>>> hear about this under Ubuntu.  We would like to take care of this for
>>> the next release.
>>>
>>> On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:
>>>> Dear all,
>>>>
>>>> after updating a running system to the version 11.10 of Ubuntu,
>>>> VisualWorks (64 bit) runs for a certain amount of time until it dies
>>>> after a memory access violation. Here are the last messages printed
>>>> by the "debug aware"-virtual machine:
>>>>
>>>> lost time signal in waitForIOWithTimeout, milliseconds 2
>>>> lost time signal in waitForIO, milliseconds 0
>>>> lost time signal in waitForIO, milliseconds 0
>>>> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
>>>> Speicherzugriffsfehler
>>>>
>>>> The version of VisualWorks in question is:
>>>>
>>>> dellani@KITT:~/$ visual -v
>>>> VisualWorks(R) 7.7.1 Jul  7 2010
>>>> Copyright (C) 1999-2009 Cincom Systems, Inc.
>>>> Release identification ...
>>>>          OE   version: 68 platform: 64 release: 68.15 flags: 0
>>>>          VI   version: 0 minor: 0 release: 0 variant: 0
>>>>          From version: 0 platform: 0 release: 0.1 flags: 0
>>>>
>>>> and here are some details about the system itself:
>>>>
>>>> dellani@KITT:~/$ uname -a
>>>> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> I tried the 32-bits virtual machine on the same system and it just runs fine!
>>>> Anyone experiencing these same problems?
>>>>
>>>> Cheers, Paulo

_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

mkobetic
In reply to this post by Paulo-27
This sounds good. Does anyone have any ideas how can I apply the same fix on Fedora and see if it fixes my crashes too ?

"Paulo Roberto Dellani"<[hidden email]> wrote:

> I was not only able to isolate the version of glibc with which
> VisualWorks exhibits this behavior, but found its possible cause
> and one workaround, so that the 64-bits version of the virtual
> machine can run on the affected systems (at least at the Debian-
> based ones).
>
> Beginning with the version 2.13 of the GlibC, an optimized version
> of memcpy() is provided on the amd64 architecture. According to
> Aurelien Jarno, one of the maintainers of the GlibC Debian package,
>
> "this version uses SSSE3 optimizations and might copy memory
> backwards in some cases, which causes issues if the source and
> destination overlap. memmove() should be used in such cases, but
> some programs still wrongly use memcpy()."
>
> This is the case, because by following the instructions gave by him to
> debug this issue, I was able to track it back in Visualworks.
>
> Here are the general instructions by Aurelien Jarno:
>
> "For this reason, on the amd64 architecture the Debian package provides
>   two wrappers which can be use to workaround and/or debug the issue:
>   - /usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so simply replace all
>     calls to memcpy() by a call to memmove()
>   - /usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so does the same,
>     but in addition logs (with rate limit) the issue to syslog, so that it
>     can be detected and fixed.
>
>   To use these wrapper on a single binary, the easiest way is to use the
>   LD_PRELOAD environment variable:
>   - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so /path/to/binary
>   - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so
> /path/to/binary"
>
> So entering the following prior to starting VisualWorks "solved" the problem:
>
> export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so
> /usr/local/vw7.7.1nc/bin/linuxx86_64/visual"
>
> If you want to see debug messages in the system syslog while the workaround does
> its job, replace "memcpy-preload.so" by "memcpy-syslog-preload.so" in the above
> command.
>
> Cheers,
>
> Paulo
>
> 2011/11/4 Andres Valloud <[hidden email]>:
> > So far I know somehow an nmethod is thrown away even though another
> > nmethod references it, so eventually the JIT jumps into code that has
> > been overwritten by something else.
> >
> > It would be interesting to see if you can correlate versions of glibc
> > with the problem.
> >
> > We'd like to see this solved for 7.9.
> >
> > On 11/3/2011 7:52 AM, Paulo Roberto Dellani wrote:
> >> Any hint on what could be the cause of it? Just for the record, I tried
> >> different versions of the Linux Kernel available in Ubuntu 11.10:
> >>
> >> 2.6.38-12-generic
> >> 3.0.0-12-generic
> >> 3.0.8-030008-generic
> >> 3.1.0-030100rc10-generic
> >> 3.1.0-999-generic
> >>
> >> VisualWorks fails with all of them as well. May be I should try
> >> another version of the GlibC,
> >> before changing the complete system ... What do you mean with next
> >> release, Andres? 7.8?
> >>
> >> Cheers,
> >>
> >> Paulo
> >>
> >>
> >> 2011/11/2 Andres Valloud<[hidden email]>:
> >>> We are aware of these problems in Fedora, but this is the first time I
> >>> hear about this under Ubuntu.  We would like to take care of this for
> >>> the next release.
> >>>
> >>> On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:
> >>>> Dear all,
> >>>>
> >>>> after updating a running system to the version 11.10 of Ubuntu,
> >>>> VisualWorks (64 bit) runs for a certain amount of time until it dies
> >>>> after a memory access violation. Here are the last messages printed
> >>>> by the "debug aware"-virtual machine:
> >>>>
> >>>> lost time signal in waitForIOWithTimeout, milliseconds 2
> >>>> lost time signal in waitForIO, milliseconds 0
> >>>> lost time signal in waitForIO, milliseconds 0
> >>>> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
> >>>> Speicherzugriffsfehler
> >>>>
> >>>> The version of VisualWorks in question is:
> >>>>
> >>>> dellani@KITT:~/$ visual -v
> >>>> VisualWorks(R) 7.7.1 Jul  7 2010
> >>>> Copyright (C) 1999-2009 Cincom Systems, Inc.
> >>>> Release identification ...
> >>>>          OE   version: 68 platform: 64 release: 68.15 flags: 0
> >>>>          VI   version: 0 minor: 0 release: 0 variant: 0
> >>>>          From version: 0 platform: 0 release: 0.1 flags: 0
> >>>>
> >>>> and here are some details about the system itself:
> >>>>
> >>>> dellani@KITT:~/$ uname -a
> >>>> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
> >>>> x86_64 x86_64 x86_64 GNU/Linux
> >>>>
> >>>> I tried the 32-bits virtual machine on the same system and it just runs fine!
> >>>> Anyone experiencing these same problems?
> >>>>
> >>>> Cheers, Paulo
>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc


_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Andres Valloud-6
In reply to this post by Paulo-27
Wow, that is an awesome.  It looks like we need to change some memcpy()
to memmove().  Thank you so much!  I'm at Smalltalks 2011 right now, but
I'll take a look at this as soon as I can.

On 11/4/2011 7:54 AM, Paulo Roberto Dellani wrote:

> Dear all,
>
> I was not only able to isolate the version of glibc with which
> VisualWorks exhibits this behavior, but found its possible cause
> and one workaround, so that the 64-bits version of the virtual
> machine can run on the affected systems (at least at the Debian-
> based ones).
>
> Beginning with the version 2.13 of the GlibC, an optimized version
> of memcpy() is provided on the amd64 architecture. According to
> Aurelien Jarno, one of the maintainers of the GlibC Debian package,
>
> "this version uses SSSE3 optimizations and might copy memory
> backwards in some cases, which causes issues if the source and
> destination overlap. memmove() should be used in such cases, but
> some programs still wrongly use memcpy()."
>
> This is the case, because by following the instructions gave by him to
> debug this issue, I was able to track it back in Visualworks.
>
> Here are the general instructions by Aurelien Jarno:
>
> "For this reason, on the amd64 architecture the Debian package provides
>    two wrappers which can be use to workaround and/or debug the issue:
>    - /usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so simply replace all
>      calls to memcpy() by a call to memmove()
>    - /usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so does the same,
>      but in addition logs (with rate limit) the issue to syslog, so that it
>      can be detected and fixed.
>
>    To use these wrapper on a single binary, the easiest way is to use the
>    LD_PRELOAD environment variable:
>    - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so /path/to/binary
>    - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so
> /path/to/binary"
>
> So entering the following prior to starting VisualWorks "solved" the problem:
>
> export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so
> /usr/local/vw7.7.1nc/bin/linuxx86_64/visual"
>
> If you want to see debug messages in the system syslog while the workaround does
> its job, replace "memcpy-preload.so" by "memcpy-syslog-preload.so" in the above
> command.
>
> Cheers,
>
> Paulo
>
> 2011/11/4 Andres Valloud<[hidden email]>:
>> So far I know somehow an nmethod is thrown away even though another
>> nmethod references it, so eventually the JIT jumps into code that has
>> been overwritten by something else.
>>
>> It would be interesting to see if you can correlate versions of glibc
>> with the problem.
>>
>> We'd like to see this solved for 7.9.
>>
>> On 11/3/2011 7:52 AM, Paulo Roberto Dellani wrote:
>>> Any hint on what could be the cause of it? Just for the record, I tried
>>> different versions of the Linux Kernel available in Ubuntu 11.10:
>>>
>>> 2.6.38-12-generic
>>> 3.0.0-12-generic
>>> 3.0.8-030008-generic
>>> 3.1.0-030100rc10-generic
>>> 3.1.0-999-generic
>>>
>>> VisualWorks fails with all of them as well. May be I should try
>>> another version of the GlibC,
>>> before changing the complete system ... What do you mean with next
>>> release, Andres? 7.8?
>>>
>>> Cheers,
>>>
>>> Paulo
>>>
>>>
>>> 2011/11/2 Andres Valloud<[hidden email]>:
>>>> We are aware of these problems in Fedora, but this is the first time I
>>>> hear about this under Ubuntu.  We would like to take care of this for
>>>> the next release.
>>>>
>>>> On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:
>>>>> Dear all,
>>>>>
>>>>> after updating a running system to the version 11.10 of Ubuntu,
>>>>> VisualWorks (64 bit) runs for a certain amount of time until it dies
>>>>> after a memory access violation. Here are the last messages printed
>>>>> by the "debug aware"-virtual machine:
>>>>>
>>>>> lost time signal in waitForIOWithTimeout, milliseconds 2
>>>>> lost time signal in waitForIO, milliseconds 0
>>>>> lost time signal in waitForIO, milliseconds 0
>>>>> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
>>>>> Speicherzugriffsfehler
>>>>>
>>>>> The version of VisualWorks in question is:
>>>>>
>>>>> dellani@KITT:~/$ visual -v
>>>>> VisualWorks(R) 7.7.1 Jul  7 2010
>>>>> Copyright (C) 1999-2009 Cincom Systems, Inc.
>>>>> Release identification ...
>>>>>           OE   version: 68 platform: 64 release: 68.15 flags: 0
>>>>>           VI   version: 0 minor: 0 release: 0 variant: 0
>>>>>           From version: 0 platform: 0 release: 0.1 flags: 0
>>>>>
>>>>> and here are some details about the system itself:
>>>>>
>>>>> dellani@KITT:~/$ uname -a
>>>>> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>>
>>>>> I tried the 32-bits virtual machine on the same system and it just runs fine!
>>>>> Anyone experiencing these same problems?
>>>>>
>>>>> Cheers, Paulo
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Andres Valloud-6
It looks like we're done fixing this problem.  In addition to the
immediate crash, we found a number of opportunities for cleanup and
optimization.  There are changes to about 250 LOC, and there is a net
code loss of about 30 LOC.  We also have new implementations for some
specialized memcpy()-like functions the VM uses for a variety of
purposes.  C compilers (not just x86) produce half the assembler
instructions required by the old functions.  The corresponding AR is

62841: "64 bit VMs crash due to improper use of memcpy()"

Assuming all goes well with the code review, this AR should hit the 7.9
builds any time now.

Thanks again for the very nice detective work.  Keep it up!

On 11/5/2011 4:29 AM, Andres Valloud wrote:

> Wow, that is an awesome.  It looks like we need to change some memcpy()
> to memmove().  Thank you so much!  I'm at Smalltalks 2011 right now, but
> I'll take a look at this as soon as I can.
>
> On 11/4/2011 7:54 AM, Paulo Roberto Dellani wrote:
>> Dear all,
>>
>> I was not only able to isolate the version of glibc with which
>> VisualWorks exhibits this behavior, but found its possible cause
>> and one workaround, so that the 64-bits version of the virtual
>> machine can run on the affected systems (at least at the Debian-
>> based ones).
>>
>> Beginning with the version 2.13 of the GlibC, an optimized version
>> of memcpy() is provided on the amd64 architecture. According to
>> Aurelien Jarno, one of the maintainers of the GlibC Debian package,
>>
>> "this version uses SSSE3 optimizations and might copy memory
>> backwards in some cases, which causes issues if the source and
>> destination overlap. memmove() should be used in such cases, but
>> some programs still wrongly use memcpy()."
>>
>> This is the case, because by following the instructions gave by him to
>> debug this issue, I was able to track it back in Visualworks.
>>
>> Here are the general instructions by Aurelien Jarno:
>>
>> "For this reason, on the amd64 architecture the Debian package provides
>>     two wrappers which can be use to workaround and/or debug the issue:
>>     - /usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so simply replace all
>>       calls to memcpy() by a call to memmove()
>>     - /usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so does the same,
>>       but in addition logs (with rate limit) the issue to syslog, so that it
>>       can be detected and fixed.
>>
>>     To use these wrapper on a single binary, the easiest way is to use the
>>     LD_PRELOAD environment variable:
>>     - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so /path/to/binary
>>     - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so
>> /path/to/binary"
>>
>> So entering the following prior to starting VisualWorks "solved" the problem:
>>
>> export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so
>> /usr/local/vw7.7.1nc/bin/linuxx86_64/visual"
>>
>> If you want to see debug messages in the system syslog while the workaround does
>> its job, replace "memcpy-preload.so" by "memcpy-syslog-preload.so" in the above
>> command.
>>
>> Cheers,
>>
>> Paulo
>>
>> 2011/11/4 Andres Valloud<[hidden email]>:
>>> So far I know somehow an nmethod is thrown away even though another
>>> nmethod references it, so eventually the JIT jumps into code that has
>>> been overwritten by something else.
>>>
>>> It would be interesting to see if you can correlate versions of glibc
>>> with the problem.
>>>
>>> We'd like to see this solved for 7.9.
>>>
>>> On 11/3/2011 7:52 AM, Paulo Roberto Dellani wrote:
>>>> Any hint on what could be the cause of it? Just for the record, I tried
>>>> different versions of the Linux Kernel available in Ubuntu 11.10:
>>>>
>>>> 2.6.38-12-generic
>>>> 3.0.0-12-generic
>>>> 3.0.8-030008-generic
>>>> 3.1.0-030100rc10-generic
>>>> 3.1.0-999-generic
>>>>
>>>> VisualWorks fails with all of them as well. May be I should try
>>>> another version of the GlibC,
>>>> before changing the complete system ... What do you mean with next
>>>> release, Andres? 7.8?
>>>>
>>>> Cheers,
>>>>
>>>> Paulo
>>>>
>>>>
>>>> 2011/11/2 Andres Valloud<[hidden email]>:
>>>>> We are aware of these problems in Fedora, but this is the first time I
>>>>> hear about this under Ubuntu.  We would like to take care of this for
>>>>> the next release.
>>>>>
>>>>> On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:
>>>>>> Dear all,
>>>>>>
>>>>>> after updating a running system to the version 11.10 of Ubuntu,
>>>>>> VisualWorks (64 bit) runs for a certain amount of time until it dies
>>>>>> after a memory access violation. Here are the last messages printed
>>>>>> by the "debug aware"-virtual machine:
>>>>>>
>>>>>> lost time signal in waitForIOWithTimeout, milliseconds 2
>>>>>> lost time signal in waitForIO, milliseconds 0
>>>>>> lost time signal in waitForIO, milliseconds 0
>>>>>> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
>>>>>> Speicherzugriffsfehler
>>>>>>
>>>>>> The version of VisualWorks in question is:
>>>>>>
>>>>>> dellani@KITT:~/$ visual -v
>>>>>> VisualWorks(R) 7.7.1 Jul  7 2010
>>>>>> Copyright (C) 1999-2009 Cincom Systems, Inc.
>>>>>> Release identification ...
>>>>>>            OE   version: 68 platform: 64 release: 68.15 flags: 0
>>>>>>            VI   version: 0 minor: 0 release: 0 variant: 0
>>>>>>            From version: 0 platform: 0 release: 0.1 flags: 0
>>>>>>
>>>>>> and here are some details about the system itself:
>>>>>>
>>>>>> dellani@KITT:~/$ uname -a
>>>>>> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
>>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>>>
>>>>>> I tried the 32-bits virtual machine on the same system and it just runs fine!
>>>>>> Anyone experiencing these same problems?
>>>>>>
>>>>>> Cheers, Paulo
>>
> _______________________________________________
> vwnc mailing list
> [hidden email]
> http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc
Reply | Threaded
Open this post in threaded view
|

Re: [VW 7.7.1 64 bits] Memory access violations on Ubuntu Linux 11.10

Andres Valloud-6
In reply to this post by Paulo-27
FYI, we just integrated a comprehensive fix to this problem with AR
62841.  We changed about 270 LOC, and we got a net code deletion of
about 30 LOC.  In addition, we deleted a bunch of unnecessary code and
cleaned up a significant amount of cruft.  In particular, we use Duff
devices for a few specialized cases of memmove() and memset().  Our new
versions result in half the assembler instructions required for the old
ones on x86, SPARC and PPC.

Thanks again for letting us know about this problem, and for taking the
time to investigate where the problem was.  Very nice, more of the same
please!

On 11/4/2011 7:54 AM, Paulo Roberto Dellani wrote:

> Dear all,
>
> I was not only able to isolate the version of glibc with which
> VisualWorks exhibits this behavior, but found its possible cause
> and one workaround, so that the 64-bits version of the virtual
> machine can run on the affected systems (at least at the Debian-
> based ones).
>
> Beginning with the version 2.13 of the GlibC, an optimized version
> of memcpy() is provided on the amd64 architecture. According to
> Aurelien Jarno, one of the maintainers of the GlibC Debian package,
>
> "this version uses SSSE3 optimizations and might copy memory
> backwards in some cases, which causes issues if the source and
> destination overlap. memmove() should be used in such cases, but
> some programs still wrongly use memcpy()."
>
> This is the case, because by following the instructions gave by him to
> debug this issue, I was able to track it back in Visualworks.
>
> Here are the general instructions by Aurelien Jarno:
>
> "For this reason, on the amd64 architecture the Debian package provides
>    two wrappers which can be use to workaround and/or debug the issue:
>    - /usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so simply replace all
>      calls to memcpy() by a call to memmove()
>    - /usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so does the same,
>      but in addition logs (with rate limit) the issue to syslog, so that it
>      can be detected and fixed.
>
>    To use these wrapper on a single binary, the easiest way is to use the
>    LD_PRELOAD environment variable:
>    - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so /path/to/binary
>    - LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libc/memcpy-syslog-preload.so
> /path/to/binary"
>
> So entering the following prior to starting VisualWorks "solved" the problem:
>
> export LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libc/memcpy-preload.so
> /usr/local/vw7.7.1nc/bin/linuxx86_64/visual"
>
> If you want to see debug messages in the system syslog while the workaround does
> its job, replace "memcpy-preload.so" by "memcpy-syslog-preload.so" in the above
> command.
>
> Cheers,
>
> Paulo
>
> 2011/11/4 Andres Valloud<[hidden email]>:
>> So far I know somehow an nmethod is thrown away even though another
>> nmethod references it, so eventually the JIT jumps into code that has
>> been overwritten by something else.
>>
>> It would be interesting to see if you can correlate versions of glibc
>> with the problem.
>>
>> We'd like to see this solved for 7.9.
>>
>> On 11/3/2011 7:52 AM, Paulo Roberto Dellani wrote:
>>> Any hint on what could be the cause of it? Just for the record, I tried
>>> different versions of the Linux Kernel available in Ubuntu 11.10:
>>>
>>> 2.6.38-12-generic
>>> 3.0.0-12-generic
>>> 3.0.8-030008-generic
>>> 3.1.0-030100rc10-generic
>>> 3.1.0-999-generic
>>>
>>> VisualWorks fails with all of them as well. May be I should try
>>> another version of the GlibC,
>>> before changing the complete system ... What do you mean with next
>>> release, Andres? 7.8?
>>>
>>> Cheers,
>>>
>>> Paulo
>>>
>>>
>>> 2011/11/2 Andres Valloud<[hidden email]>:
>>>> We are aware of these problems in Fedora, but this is the first time I
>>>> hear about this under Ubuntu.  We would like to take care of this for
>>>> the next release.
>>>>
>>>> On 11/2/2011 4:43 AM, Paulo Roberto Dellani wrote:
>>>>> Dear all,
>>>>>
>>>>> after updating a running system to the version 11.10 of Ubuntu,
>>>>> VisualWorks (64 bit) runs for a certain amount of time until it dies
>>>>> after a memory access violation. Here are the last messages printed
>>>>> by the "debug aware"-virtual machine:
>>>>>
>>>>> lost time signal in waitForIOWithTimeout, milliseconds 2
>>>>> lost time signal in waitForIO, milliseconds 0
>>>>> lost time signal in waitForIO, milliseconds 0
>>>>> Assert fail @ 310 in tran/nmcompact.c   nMethodOK(curr)
>>>>> Speicherzugriffsfehler
>>>>>
>>>>> The version of VisualWorks in question is:
>>>>>
>>>>> dellani@KITT:~/$ visual -v
>>>>> VisualWorks(R) 7.7.1 Jul  7 2010
>>>>> Copyright (C) 1999-2009 Cincom Systems, Inc.
>>>>> Release identification ...
>>>>>           OE   version: 68 platform: 64 release: 68.15 flags: 0
>>>>>           VI   version: 0 minor: 0 release: 0 variant: 0
>>>>>           From version: 0 platform: 0 release: 0.1 flags: 0
>>>>>
>>>>> and here are some details about the system itself:
>>>>>
>>>>> dellani@KITT:~/$ uname -a
>>>>> Linux KITT 3.0.0-12-generic #20-Ubuntu SMP Fri Oct 7 14:56:25 UTC 2011
>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>>
>>>>> I tried the 32-bits virtual machine on the same system and it just runs fine!
>>>>> Anyone experiencing these same problems?
>>>>>
>>>>> Cheers, Paulo
>
_______________________________________________
vwnc mailing list
[hidden email]
http://lists.cs.uiuc.edu/mailman/listinfo/vwnc