MS Data Execution Prevention (DEP) ?

Hello everybody,

That was reported on the Gemstone related group:

"Microsoft Data Execution Prevention (DEP) is standard on Win2003 SP1
and it is known to kill GS/S 6.1.3 processes without giving
notification.  You may have to test both with software-enforced DEP and
hardware-enforced DEP.".

"I can see why DEP might kill gem processes (gem and topaz).  The native
code generator can cause memory pages allocated for use as data (the
default) to be executed as text, which is what DEP appears to prevent".


Does anybody know if DEP may affect Smalltalk applications in the
similar manner?


The quote form Microsoft:

http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/sp2mempr.mspx

...Application Compatibility

Some application behaviors are expected to be incompatible with data
execution prevention. Applications which perform dynamic code generation
(such as Just-In-Time code generation) and that do not explicitly mark
generated code with Execute permission might have compatibility issues
with data execution prevention. Applications which are not built with
SafeSEH must have their exception handlers located in executable memory
regions.

Applications that attempt to violate DEP will receive an exception with
status code STATUS_ACCESS_VIOLATION (0xC0000005). If an application
requires executable memory, it must explicitly set this attribute on the
appropriate memory by specifying PAGE_EXECUTE, PAGE_EXECUTE_READ,
PAGE_EXECUTE_READWRITE or PAGE_EXECUTE_WRITECOPY in the memory
protection argument of the Virtual* memory allocation functions. Heap
allocations using the malloc() and HeapAlloc() functions are
non-executable.
....

Of course one could always turn off DEP for VM executable if there were any
issues. For instance turning off DEP for gem.exe will make GemStone behave
normally again.

Cheers!

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message
header. Unless otherwise indicated, it contains information that is
private and confidential. If you have received it in error, please
notify the sender and delete the entire message including any
attachments.

Thank you.
-----Original Message-----
From: Mark Pirogovsky [mailto:[hidden email]]
Sent: Thursday, February 02, 2006 7:50 AM
To: [hidden email]
Subject: MS Data Execution Prevention (DEP) ?

Hello everybody,

That was reported on the Gemstone related group:

"Microsoft Data Execution Prevention (DEP) is standard on Win2003 SP1 and it
is known to kill GS/S 6.1.3 processes without giving notification.  You may
have to test both with software-enforced DEP and hardware-enforced DEP.".

"I can see why DEP might kill gem processes (gem and topaz).  The native
code generator can cause memory pages allocated for use as data (the
default) to be executed as text, which is what DEP appears to prevent".


Does anybody know if DEP may affect Smalltalk applications in the similar
manner?


The quote form Microsoft:

http://www.microsoft.com/technet/prodtechnol/winxppro/maintain/sp2mempr.mspx

...Application Compatibility

Some application behaviors are expected to be incompatible with data
execution prevention. Applications which perform dynamic code generation
(such as Just-In-Time code generation) and that do not explicitly mark
generated code with Execute permission might have compatibility issues
with data execution prevention. Applications which are not built with
SafeSEH must have their exception handlers located in executable memory
regions.

Applications that attempt to violate DEP will receive an exception with
status code STATUS_ACCESS_VIOLATION (0xC0000005). If an application
requires executable memory, it must explicitly set this attribute on the
appropriate memory by specifying PAGE_EXECUTE, PAGE_EXECUTE_READ,
PAGE_EXECUTE_READWRITE or PAGE_EXECUTE_WRITECOPY in the memory
protection argument of the Virtual* memory allocation functions. Heap
allocations using the malloc() and HeapAlloc() functions are
non-executable.
....

Boris,

Thanks for the answer,  however my question really is:

Has anybody actually experienced the DEP not letting VW to run ?

My problem is that we distribute our application do diverse user
community and they can run it on any possible OS/hardware combination.
So should I start doing something now, or I can do it later?

just to give you an example...We already came across the problem on Win
XP 64, where one of the third party drivers, we use, doesn't work. The
good news for us -- it was not only our application which had a problem,
so the user had to switch back to the normal WinXP to be able to do work...


Truly,

--Mark

Boris Popov wrote:

> Of course one could always turn off DEP for VM executable if there were any
> issues. For instance turning off DEP for gem.exe will make GemStone behave
> normally again.
>
> Cheers!
>
> -Boris
>

I guess if you aren't able to test your application on a DEP-enabled machine
or don't feel like taking chances, you could use preemptive strike and
disable DEP for your application during the installation process ;)

Cheers!

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message
header. Unless otherwise indicated, it contains information that is
private and confidential. If you have received it in error, please
notify the sender and delete the entire message including any
attachments.

Thank you.

-----Original Message-----
From: Mark Pirogovsky [mailto:[hidden email]]
Sent: Thursday, February 02, 2006 8:44 AM
To: Boris Popov
Cc: [hidden email]
Subject: Re: MS Data Execution Prevention (DEP) ?

Boris,

Thanks for the answer,  however my question really is:

Has anybody actually experienced the DEP not letting VW to run ?

My problem is that we distribute our application do diverse user community
and they can run it on any possible OS/hardware combination.
So should I start doing something now, or I can do it later?

just to give you an example...We already came across the problem on Win XP
64, where one of the third party drivers, we use, doesn't work. The good
news for us -- it was not only our application which had a problem, so the
user had to switch back to the normal WinXP to be able to do work...


Truly,

--Mark

Boris Popov wrote:

> Of course one could always turn off DEP for VM executable if there
> were any issues. For instance turning off DEP for gem.exe will make
> GemStone behave normally again.
>
> Cheers!
>
> -Boris
>

Mark

I think users are going to have to know how to address this
problem. I ran into it with a game I downloaded, I think it
was a Java based game. Anyway, for the software based DEP
you can add the program to an exclusion list and then it
will run.

Terry
 
===========================================================
Terry Raymond       Smalltalk Professional Debug Package
Crafted Smalltalk
80 Lazywood Ln.
Tiverton, RI  02878
(401) 624-4517      [hidden email]
<http://www.craftedsmalltalk.com>
===========================================================

Two questions
1. What about the hardware enabled DEP ?

2.
What is the best way to do the recommended:

Boris Popov wrote: ...
 > disable DEP for your application during the installation process

Terry Raymond wrote:

Anyway, for the software based DEP
> you can add the program to an exclusion list and then it
> will run.
>
> Terry
>  


Thanks,
--Mark

1. It appears that VisualWorks runs okay under DEP, as I've gotten some
private responses from people who happen to run it on 2k3 with SP2; would be
good to try to get some assurances from VisualWorks engineering though.

2. See the following,

<quote>
Per-program DEP configuration
For the purposes of program compatibility, you can selectively disable DEP
for individual 32-bit programs when DEP is set to the OptOut policy level.
To do this, use the Data Execution Prevention tab in System Properties to
selectively disable DEP for a program.

For IT professionals, a new program compatibility fix that is named
DisableNX is included with Windows XP SP2. The DisableNX compatibility fix
disables Data Execution Prevention for the program that the fix is applied
to.

The DisableNX compatibility fix can be applied to a program by using the
Application Compatibility Toolkit. For more information about Windows
application compatibility, see Windows Application Compatibility on the
following Microsoft Web site:
http://www.microsoft.com/technet/prodtechnol/windows/appcompatibility/defaul
t.mspx
</quote>

Cheers!

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message
header. Unless otherwise indicated, it contains information that is
private and confidential. If you have received it in error, please
notify the sender and delete the entire message including any
attachments.

Thank you.

-----Original Message-----
From: Mark Pirogovsky [mailto:[hidden email]]
Sent: Thursday, February 02, 2006 9:39 AM
To: [hidden email]
Subject: Re: MS Data Execution Prevention (DEP) ?

Two questions
1. What about the hardware enabled DEP ?

2.
What is the best way to do the recommended:

Boris Popov wrote: ...
 > disable DEP for your application during the installation process

Terry Raymond wrote:

Anyway, for the software based DEP
> you can add the program to an exclusion list and then it will run.
>
> Terry
>  


Thanks,
--Mark

James Robertson said the change would happen in VW 7.2.1:
(beware line breaks in URL, 2nd msg in thread):
<http://groups.google.fi/group/comp.lang.smalltalk/browse_frm/thread/78d
71cf321904355/236ea71b6dcd58d6?lnk=st&q=group%3Acomp.lang.smalltalk+prot
ection&rnum=10&hl=fi#236ea71b6dcd58d6>

I couldn't spot it in the release notes or fixed ars, so a confirmation
would be good still...

Cheers,
Steve
ibility/defaul
t.mspx
</quote>

Cheers!

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message header.
Unless otherwise indicated, it contains information that is private and
confidential. If you have received it in error, please notify the sender
and delete the entire message including any attachments.

Thank you.

-----Original Message-----
From: Mark Pirogovsky [mailto:[hidden email]]
Sent: Thursday, February 02, 2006 9:39 AM
To: [hidden email]
Subject: Re: MS Data Execution Prevention (DEP) ?

Two questions
1. What about the hardware enabled DEP ?

2.
What is the best way to do the recommended:

Boris Popov wrote: ...
 > disable DEP for your application during the installation process

Terry Raymond wrote:

Anyway, for the software based DEP
> you can add the program to an exclusion list and then it will run.
>
> Terry
>  


Thanks,
--Mark

There should not be a problem with this on VW I run win2k3 with this
turned on as well as xp. I received confirmation from eliot that we
should not have a problem with DEP.

Sean

-----Original Message-----
From: Boris Popov [mailto:[hidden email]]
Sent: Thursday, February 02, 2006 12:44 PM
To: Mark Pirogovsky; [hidden email]
Subject: RE: MS Data Execution Prevention (DEP) ?

1. It appears that VisualWorks runs okay under DEP, as I've gotten some
private responses from people who happen to run it on 2k3 with SP2;
would be good to try to get some assurances from VisualWorks engineering
though.

2. See the following,

<quote>
Per-program DEP configuration
For the purposes of program compatibility, you can selectively disable
DEP for individual 32-bit programs when DEP is set to the OptOut policy
level.
To do this, use the Data Execution Prevention tab in System Properties
to selectively disable DEP for a program.

For IT professionals, a new program compatibility fix that is named
DisableNX is included with Windows XP SP2. The DisableNX compatibility
fix disables Data Execution Prevention for the program that the fix is
applied to.

The DisableNX compatibility fix can be applied to a program by using the
Application Compatibility Toolkit. For more information about Windows
application compatibility, see Windows Application Compatibility on the
following Microsoft Web site:
http://www.microsoft.com/technet/prodtechnol/windows/appcompatibility/de
faul
t.mspx
</quote>

Cheers!

-Boris

--
+1.604.689.0322
DeepCove Labs Ltd.
4th floor 595 Howe Street
Vancouver, Canada V6C 2T5

[hidden email]

CONFIDENTIALITY NOTICE

This email is intended only for the persons named in the message header.
Unless otherwise indicated, it contains information that is private and
confidential. If you have received it in error, please notify the sender
and delete the entire message including any attachments.

Thank you.

-----Original Message-----
From: Mark Pirogovsky [mailto:[hidden email]]
Sent: Thursday, February 02, 2006 9:39 AM
To: [hidden email]
Subject: Re: MS Data Execution Prevention (DEP) ?

Two questions
1. What about the hardware enabled DEP ?

2.
What is the best way to do the recommended:

Boris Popov wrote: ...
 > disable DEP for your application during the installation process

Terry Raymond wrote:

Anyway, for the software based DEP
> you can add the program to an exclusion list and then it will run.
>
> Terry
>  


Thanks,
--Mark

On Thu, 2 Feb 2006, Mark Pirogovsky wrote:

> Applications that attempt to violate DEP will receive an exception with
> status code STATUS_ACCESS_VIOLATION (0xC0000005). If an application
> requires executable memory, it must explicitly set this attribute on the
> appropriate memory by specifying PAGE_EXECUTE, PAGE_EXECUTE_READ,
> PAGE_EXECUTE_READWRITE or PAGE_EXECUTE_WRITECOPY in the memory
> protection argument of the Virtual* memory allocation functions. Heap
> allocations using the malloc() and HeapAlloc() functions are
> non-executable.

The memory where VisualWorks translates/compiles the Smalltalk bytecodes
into native code does have the appropriate flags set on Windows.
Therefore, this shouldn't be issue for the VM.   If it was, you would
quite likely see access violations as soon as tried to execute the first
translated compiled method.  As far as I can tell, we've been doing this
for quite some time.  I've even got off my chair to go confirm this with
Eliot.

However, we do *not* set execute permissions on thunks for C Callbacks.
So it is possible that if you have code in VW that is calling out to C
which callsback into VW, the memory violation occurs.  So, if someone runs
the dllcc test suite (which includes callbacks) on one of these systems
with "execute protection" turned on and the callback tests fail, we'd like
to hear about it.

Good grief, I need to proofread my mail first.  See below.

On Thu, 2 Feb 2006, Robert Westergaard wrote:

Above, I ment:

    If it was *not*, you would

Sorry for the confusion.