Cog on Mac. Problems with mprotect()?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Cog on Mac. Problems with mprotect()?

Igor Stasenko
 
I have no idea why mprotect() returns -1 (failure),
and then perror prints following:

mprotect(x,y,PROT_READ | PROT_WRITE | PROT_EXEC): Cannot allocate memory
mprotect(x,y,PROT_READ | PROT_WRITE): Invalid argument

While it looks like Cog VM runs stable at the same time. But it
crashing at crucial point of NB tests,
needless to say, that same tests are running on other platforms and
everything ok.

So, there is something wrong with it. The only question is what :)

While in general, i can run native code, generated by NB, which means
that memory execution is enabled,
something leads to memory access error during GC, in test which
checking if relocated native code will continue to run from
expected point as if it was not relocated.
What is strange with this thing, that i expected to see errors in my
code. But why it crashes in the middle of pow() function,
which is called from initializeMemoryFirstFree()??

Thread 0 Crashed:  Dispatch queue: com.apple.main-thread
0   libSystem.B.dylib             0x98273176 __kill + 10
1   libSystem.B.dylib             0x98273168 kill$UNIX2003 + 32
2   libSystem.B.dylib             0x9830589d raise + 26
3   libSystem.B.dylib             0x9831b9bc abort + 93
4   com.teleplace.Teleplace       0x00090f86 error + 86
5   libSystem.B.dylib             0x9827846b _sigtramp + 43
6   ???                           0xffffffff 0 + 4294967295
7   com.teleplace.Teleplace       0x0010d468 initializeMemoryFirstFree + 184
8   com.teleplace.Teleplace       0x00113381 incCompBody + 1057
9   com.teleplace.Teleplace       0x001229ec fullGC + 1356
10  ???                           0x17203026 0 + 387985446
11  com.teleplace.Teleplace       0x0014fe98 primitiveNativeCall + 216
12  com.teleplace.Teleplace       0x00132422 executeNewMethod + 130
13  com.teleplace.Teleplace       0x00133111
primitiveExecuteMethodArgsArray + 497
14  com.teleplace.Teleplace       0x001365f6 interpret + 1078
15  com.teleplace.Teleplace       0x0013d94e
enterSmalltalkExecutiveImplementation + 110
16  com.teleplace.Teleplace       0x0013db81 initStackPagesAndInterpret + 545
17  com.teleplace.Teleplace       0x00079494 EventLoopEventHandler + 132


The stack frame #11 is a primitive which calls my native code (frame #10)
then the native code deliberately calls fullGC() (frame #9)
and it seems like works, till something bad happen in pow() function,
called by initializeMemoryFirstFree()

Here the code near call to pow() function:

0x0010d43d  <+0141>  shr    %ebx
0x0010d43f  <+0143>  add    $0x1,%eax
0x0010d442  <+0146>  mov    0x182be8,%edi
0x0010d448  <+0152>  add    %ebx,%eax
0x0010d44a  <+0154>  cvtsi2sd %eax,%xmm0
0x0010d44e  <+0158>  movsd  %xmm0,0x8(%esp)
0x0010d454  <+0164>  movl   $0x0,(%esp)
0x0010d45b  <+0171>  movl   $0x40000000,0x4(%esp)
0x0010d463  <+0179>  call   0x163a84 <dyld_stub_pow>
0x0010d468  <+0184>  fstpl  -0x30(%ebp)
0x0010d46b  <+0187>  cvttsd2si -0x30(%ebp),%eax
0x0010d470  <+0192>  add    -0x1c(%ebp),%eax

and here the code where it crashing:

0x98262f56  <+1078>  jmp    0x98262b66 <pow$fenv_access_off+70>
0x98262f5b  <+1083>  movapd %xmm2,%xmm1
0x98262f5f  <+1087>  andpd  0x13e049(%ebx),%xmm1
0x98262f67  <+1095>  movapd %xmm2,%xmm0
0x98262f6b  <+1099>  andpd  0x13e079(%ebx),%xmm0
*** 0x98262f73  <+1107>  movapd %xmm0,-0x78(%ebp)
0x98262f78  <+1112>  movapd %xmm1,%xmm0
0x98262f7c  <+1116>  lea    0x13e069(%ebx),%esi
0x98262f82  <+1122>  addpd  (%esi),%xmm0
0x98262f86  <+1126>  andpd  -0x78(%ebp),%xmm0
0x98262f8b  <+1131>  movapd %xmm0,-0x78(%ebp)
0x98262f90  <+1136>  paddq  0x13cfb9(%ebx),%xmm1


i can only guess, that it either a problem with mprotect(),
or that due to heavy use of MMX/SSE instructions, i miss some
precautions (not saving some registers/processor state) between the
calls in my code.
Any ideas?

--
Best regards,
Igor Stasenko AKA sig.
Reply | Threaded
Open this post in threaded view
|

Re: Cog on Mac. Problems with mprotect()?

johnmci

mprotect  likely returns Invalid Argument because the range is wrong.

I traded some notes with Eliot on this in August, you'll need to chase him to see what we decided to change in order to avoid the problem.
Yes this is the for mprotect(x,y,PROT_READ | PROT_WRITE): Invalid argument  I guess...

"This is the mprotect for the squeak heap
start 525336576 end 1043333096 firstPage 525336576 roundUpLength 517996544
that works with my change to subtract out the cogCodeSize

        sqMakeMemoryNotExecutableFromTo(((usqInt)heapBase), ((usqInt)GIV(memoryLimit))-GIV(cogCodeSize));
But the
        sqMakeMemoryNotExecutableFromTo(((usqInt)theStackMemory), (((usqInt)theStackMemory)) + stackPagesBytes);

fails

start 2953754048 end 2954415812 firstPage 2953752576 roundUpLength 663552
mprotect(x,y,PROT_READ | PROT_WRITE): Cannot allocate memory"


On 2010-12-05, at 3:48 PM, Igor Stasenko wrote:

>
> I have no idea why mprotect() returns -1 (failure),
> and then perror prints following:
>
> mprotect(x,y,PROT_READ | PROT_WRITE | PROT_EXEC): Cannot allocate memory
> mprotect(x,y,PROT_READ | PROT_WRITE): Invalid argument
>
> While it looks like Cog VM runs stable at the same time. But it
> crashing at crucial point of NB tests,
> needless to say, that same tests are running on other platforms and
> everything ok.
>
> So, there is something wrong with it. The only question is what :)
>
> While in general, i can run native code, generated by NB, which means
> that memory execution is enabled,
> something leads to memory access error during GC, in test which
> checking if relocated native code will continue to run from
> expected point as if it was not relocated.
> What is strange with this thing, that i expected to see errors in my
> code. But why it crashes in the middle of pow() function,
> which is called from initializeMemoryFirstFree()??
>
> Thread 0 Crashed:  Dispatch queue: com.apple.main-thread
> 0   libSystem.B.dylib             0x98273176 __kill + 10
> 1   libSystem.B.dylib             0x98273168 kill$UNIX2003 + 32
> 2   libSystem.B.dylib             0x9830589d raise + 26
> 3   libSystem.B.dylib             0x9831b9bc abort + 93
> 4   com.teleplace.Teleplace       0x00090f86 error + 86
> 5   libSystem.B.dylib             0x9827846b _sigtramp + 43
> 6   ???                           0xffffffff 0 + 4294967295
> 7   com.teleplace.Teleplace       0x0010d468 initializeMemoryFirstFree + 184
> 8   com.teleplace.Teleplace       0x00113381 incCompBody + 1057
> 9   com.teleplace.Teleplace       0x001229ec fullGC + 1356
> 10  ???                           0x17203026 0 + 387985446
> 11  com.teleplace.Teleplace       0x0014fe98 primitiveNativeCall + 216
> 12  com.teleplace.Teleplace       0x00132422 executeNewMethod + 130
> 13  com.teleplace.Teleplace       0x00133111
> primitiveExecuteMethodArgsArray + 497
> 14  com.teleplace.Teleplace       0x001365f6 interpret + 1078
> 15  com.teleplace.Teleplace       0x0013d94e
> enterSmalltalkExecutiveImplementation + 110
> 16  com.teleplace.Teleplace       0x0013db81 initStackPagesAndInterpret + 545
> 17  com.teleplace.Teleplace       0x00079494 EventLoopEventHandler + 132
>
>
> The stack frame #11 is a primitive which calls my native code (frame #10)
> then the native code deliberately calls fullGC() (frame #9)
> and it seems like works, till something bad happen in pow() function,
> called by initializeMemoryFirstFree()
>
> Here the code near call to pow() function:
>
> 0x0010d43d  <+0141>  shr    %ebx
> 0x0010d43f  <+0143>  add    $0x1,%eax
> 0x0010d442  <+0146>  mov    0x182be8,%edi
> 0x0010d448  <+0152>  add    %ebx,%eax
> 0x0010d44a  <+0154>  cvtsi2sd %eax,%xmm0
> 0x0010d44e  <+0158>  movsd  %xmm0,0x8(%esp)
> 0x0010d454  <+0164>  movl   $0x0,(%esp)
> 0x0010d45b  <+0171>  movl   $0x40000000,0x4(%esp)
> 0x0010d463  <+0179>  call   0x163a84 <dyld_stub_pow>
> 0x0010d468  <+0184>  fstpl  -0x30(%ebp)
> 0x0010d46b  <+0187>  cvttsd2si -0x30(%ebp),%eax
> 0x0010d470  <+0192>  add    -0x1c(%ebp),%eax
>
> and here the code where it crashing:
>
> 0x98262f56  <+1078>  jmp    0x98262b66 <pow$fenv_access_off+70>
> 0x98262f5b  <+1083>  movapd %xmm2,%xmm1
> 0x98262f5f  <+1087>  andpd  0x13e049(%ebx),%xmm1
> 0x98262f67  <+1095>  movapd %xmm2,%xmm0
> 0x98262f6b  <+1099>  andpd  0x13e079(%ebx),%xmm0
> *** 0x98262f73  <+1107>  movapd %xmm0,-0x78(%ebp)
> 0x98262f78  <+1112>  movapd %xmm1,%xmm0
> 0x98262f7c  <+1116>  lea    0x13e069(%ebx),%esi
> 0x98262f82  <+1122>  addpd  (%esi),%xmm0
> 0x98262f86  <+1126>  andpd  -0x78(%ebp),%xmm0
> 0x98262f8b  <+1131>  movapd %xmm0,-0x78(%ebp)
> 0x98262f90  <+1136>  paddq  0x13cfb9(%ebx),%xmm1
>
>
> i can only guess, that it either a problem with mprotect(),
> or that due to heavy use of MMX/SSE instructions, i miss some
> precautions (not saving some registers/processor state) between the
> calls in my code.
> Any ideas?
>
> --
> Best regards,
> Igor Stasenko AKA sig.

--
===========================================================================
John M. McIntosh <[hidden email]>   Twitter:  squeaker68882
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
===========================================================================




Reply | Threaded
Open this post in threaded view
|

Re: Cog on Mac. Problems with mprotect()?

Igor Stasenko

On 6 December 2010 00:59, John M McIntosh
<[hidden email]> wrote:
>
> mprotect  likely returns Invalid Argument because the range is wrong.
>
> I traded some notes with Eliot on this in August, you'll need to chase him to see what we decided to change in order to avoid the problem.
> Yes this is the for mprotect(x,y,PROT_READ | PROT_WRITE): Invalid argument  I guess...
>

In NativeBoost i overriding the #initStackPagesAndInterpret,
to replace call:
        self sqMakeMemoryNotExecutableFrom: objectMemory startOfMemory
asUnsignedInteger
                To: objectMemory memoryLimit asUnsignedInteger.

with:
        self sqMakeMemoryExecutableFrom: objectMemory startOfMemory asUnsignedInteger
                To: objectMemory memoryLimit asUnsignedInteger.

so, really. This should not be a problem with my little patch, because
i using same arguments as in original version.


mmap and mprotect explicitly says that start address should be always
a multiple of page size (which is usually 4k on x86, and there are
usually some special functions to get the page size somewhere).


> "This is the mprotect for the squeak heap
> start 525336576 end 1043333096 firstPage 525336576 roundUpLength 517996544

What is strange, that docs says nothing about the length parameter
constraints. Should it also be a page size aligned or not?

because:

start: 525336576 \\ 4096 0
end: 1043333096 \\ 4096  4072

that could be the cause of problem, if length not rounded to page size.


> that works with my change to subtract out the cogCodeSize
>
>        sqMakeMemoryNotExecutableFromTo(((usqInt)heapBase), ((usqInt)GIV(memoryLimit))-GIV(cogCodeSize));
> But the
>        sqMakeMemoryNotExecutableFromTo(((usqInt)theStackMemory), (((usqInt)theStackMemory)) + stackPagesBytes);
>
> fails
>

Since the sqMakeMemory[Not]ExecutableFromTo() in both variants using:

mprotect(startAddr, endAddr - startAddr + 1, ...

so i think that
     sqMakeMemoryNotExecutableFromTo(((usqInt)theStackMemory),
(((usqInt)theStackMemory)) + stackPagesBytes);

should be:

     sqMakeMemoryNotExecutableFromTo(((usqInt)theStackMemory),
(((usqInt)theStackMemory)) + stackPagesBytes - 1);

and then we should ensure that stackPagesBytes is page-size aligned.
Otherwise, if even that won't work, i can't imagine what else we can do :)


Btw, Eliot why you made these functions
(sqMakeMemory[Not]Executable...) accept an address range instead of
address + length ?

mmap(), VirtualAlloc(),  mprotect() and VirtualProtect all using addr
+ size arguments, not the address range.


--
Best regards,
Igor Stasenko AKA sig.
Reply | Threaded
Open this post in threaded view
|

Re: Cog on Mac. Problems with mprotect()?

Igor Stasenko
 
OK, i changed

interpreterAllocationReserveBytes
        | availableBytesPerPage maxFramesPerPage |
        availableBytesPerPage := self stackPageByteSize - self
stackLimitOffset - self stackPageHeadroom.
        maxFramesPerPage := availableBytesPerPage / BytesPerWord // MFrameSlots.
        ^2 raisedTo: (maxFramesPerPage * LargeContextSize * numStackPages) highBit

to:

interpreterAllocationReserveBytes
        | availableBytesPerPage maxFramesPerPage |
        availableBytesPerPage := self stackPageByteSize - self
stackLimitOffset - self stackPageHeadroom.
        maxFramesPerPage := availableBytesPerPage / BytesPerWord // MFrameSlots.
        ^2 << ((maxFramesPerPage * LargeContextSize * numStackPages) highBit -1)


which is equal, except that latter its not using pow() function and
somehow this no longer crashing the VM.
The only conclusion i can make that this is really have nothing to do
with mprotect() and other mmap() stuff.

I can only guess, that xcode doing some aggressive optimization,
violating calling conventions, or in some other way the fullGC is
optimized in a way
that it expected to be called from certain places (but obviously not
from native code generated on the fly).

That's really weird.

--
Best regards,
Igor Stasenko AKA sig.
Reply | Threaded
Open this post in threaded view
|

Re: Cog on Mac. Problems with mprotect()?

Mariano Martinez Peck
 
Hi Igor. I think I have a similar/same problem. I tried what you suggested about changing interpreterAllocationReserveBytes

I think it now doesn't crash because of that (I think), but the message

mprotect(x,y,PROT_READ | PROT_WRITE | PROT_EXEC): Cannot allocate memory
mprotect(x,y,PROT_READ | PROT_WRITE): Invalid argument

is still present. As far as I understood you said your crash was not due to that...but anyway, is that really a problem? is it fixed somewhere?
because I am having weird crashes and I cannot discover why.

thanks

mariano

2010/12/6 Igor Stasenko <[hidden email]>

OK, i changed

interpreterAllocationReserveBytes
       | availableBytesPerPage maxFramesPerPage |
       availableBytesPerPage := self stackPageByteSize - self
stackLimitOffset - self stackPageHeadroom.
       maxFramesPerPage := availableBytesPerPage / BytesPerWord // MFrameSlots.
       ^2 raisedTo: (maxFramesPerPage * LargeContextSize * numStackPages) highBit

to:

interpreterAllocationReserveBytes
       | availableBytesPerPage maxFramesPerPage |
       availableBytesPerPage := self stackPageByteSize - self
stackLimitOffset - self stackPageHeadroom.
       maxFramesPerPage := availableBytesPerPage / BytesPerWord // MFrameSlots.
       ^2 << ((maxFramesPerPage * LargeContextSize * numStackPages) highBit -1)


which is equal, except that latter its not using pow() function and
somehow this no longer crashing the VM.
The only conclusion i can make that this is really have nothing to do
with mprotect() and other mmap() stuff.

I can only guess, that xcode doing some aggressive optimization,
violating calling conventions, or in some other way the fullGC is
optimized in a way
that it expected to be called from certain places (but obviously not
from native code generated on the fly).

That's really weird.

--
Best regards,
Igor Stasenko AKA sig.

Reply | Threaded
Open this post in threaded view
|

Re: Cog on Mac. Problems with mprotect()?

Igor Stasenko

On 14 December 2010 11:58, Mariano Martinez Peck <[hidden email]> wrote:
>
> Hi Igor. I think I have a similar/same problem. I tried what you suggested about changing interpreterAllocationReserveBytes
>
This fixes nothing , actually.
It was crashed in pow() function , because i called thing without
properly aligning the stack.
So, forget about this change :)

> I think it now doesn't crash because of that (I think), but the message
>
> mprotect(x,y,PROT_READ | PROT_WRITE | PROT_EXEC): Cannot allocate memory
> mprotect(x,y,PROT_READ | PROT_WRITE): Invalid argument
>
> is still present. As far as I understood you said your crash was not due to that...but anyway, is that really a problem? is it fixed somewhere?
> because I am having weird crashes and I cannot discover why.
>

In NB-enabled VM it works as expected (memory code execution is
enabled) , which means that mprotect() works,
despite it barfs with errors.
I tried to play with arguments to get rid of this barfing, but without success.


> thanks
>
> mariano
>
> 2010/12/6 Igor Stasenko <[hidden email]>
>>
>> OK, i changed
>>
>> interpreterAllocationReserveBytes
>>        | availableBytesPerPage maxFramesPerPage |
>>        availableBytesPerPage := self stackPageByteSize - self
>> stackLimitOffset - self stackPageHeadroom.
>>        maxFramesPerPage := availableBytesPerPage / BytesPerWord // MFrameSlots.
>>        ^2 raisedTo: (maxFramesPerPage * LargeContextSize * numStackPages) highBit
>>
>> to:
>>
>> interpreterAllocationReserveBytes
>>        | availableBytesPerPage maxFramesPerPage |
>>        availableBytesPerPage := self stackPageByteSize - self
>> stackLimitOffset - self stackPageHeadroom.
>>        maxFramesPerPage := availableBytesPerPage / BytesPerWord // MFrameSlots.
>>        ^2 << ((maxFramesPerPage * LargeContextSize * numStackPages) highBit -1)
>>
>>
>> which is equal, except that latter its not using pow() function and
>> somehow this no longer crashing the VM.
>> The only conclusion i can make that this is really have nothing to do
>> with mprotect() and other mmap() stuff.
>>
>> I can only guess, that xcode doing some aggressive optimization,
>> violating calling conventions, or in some other way the fullGC is
>> optimized in a way
>> that it expected to be called from certain places (but obviously not
>> from native code generated on the fly).
>>
>> That's really weird.
>>
>> --
>> Best regards,
>> Igor Stasenko AKA sig.
>
>
>



--
Best regards,
Igor Stasenko AKA sig.