Wrongly-applied DEP

We have a rather strange problem. It’s well-established that VisualSmalltalk has issues with Data Execution Prevention. Up until now we’ve always simply asked the customer to list the application as a DEP exception, and this solves the problem. But what appears to be happening here is that, even though the application is listed as an exception, DEP is *still* being applied (and killing the application at launch); it is only if we set DEP to Opt-In rather than Opt-Out that the box allows the application to run.

It seems to be affecting only this one customer, but two different computers on that site (running different OS versions).

Has anyone else ever seen anything like this?

Hardware DEP or software DEP?  The past conversations in this VSE group, from my recollection, were all about hardware DEP.  Software DEP is quite different under the covers (check Wikipedia) and is only on for Windows components by default, so the admin would have to go out of their way to get your VSE app running with software DEP.  Check the bottom of the Data Execution Prevention tab of the Performance Options of System Properties to see if a machine is using software or hardware DEP.  Having said that, I don’t recall any problems with VSE and software DEP, and I also don’t know why the exception wouldn’t work whereas turning it off completely would.  Sorry.

I assume you’ve checked the windows event log to make sure it’s actually DEP that’s killing your app?  I only ask because we’ve encountered some  cases that look a lot like DEP termination but aren’t.  This even included the problem going away once the DEP exclusion was added... which later turned out to be a red herring when the problem returned with the exclusion still in place.  The windows event log usually clears it up definitively.  Also add column “Data Execution Prevention” to the Windows Task Manager process tab if you haven’t already.

--Lorin

We have a rather strange problem. It’s well-established that VisualSmalltalk has issues with Data Execution Prevention. Up until now we’ve always simply asked the customer to list the application as a DEP exception, and this solves the problem. But what appears to be happening here is that, even though the application is listed as an exception, DEP is *still* being applied (and killing the application at launch); it is only if we set DEP to Opt-In rather than Opt-Out that the box allows the application to run.

It seems to be affecting only this one customer, but two different computers on that site (running different OS versions).

Has anyone else ever seen anything like this?

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

To clarify: there are normally two options for DEP; let us call them options 1 and 2. On Windows 7 option 1 is described as “Turn on DEP for essential Windows programs and services only”; option 2 is “Turn on DEP for all programs and services except those I select”.

For any other customer, if you select option 2 and don’t specify any exceptions then our application (as expected) doesn’t run; but if you add our application to the list of exceptions, everything then works fine. This is what you would expect.

But for just this one customer, even if you add our application to the list of exceptions, it still doesn’t work; but if you select DEP option 1 on the box instead, then our application works fine.

So it has to be DEP that is causing the crash, otherwise switching to option 1 wouldn’t make any difference; but (for some reason) the fact that our application is in the DEP exception list is being ignored. The question is why that might be happening.

Hardware DEP or software DEP?  The past conversations in this VSE group, from my recollection, were all about hardware DEP.  Software DEP is quite different under the covers (check Wikipedia) and is only on for Windows components by default, so the admin would have to go out of their way to get your VSE app running with software DEP.  Check the bottom of the Data Execution Prevention tab of the Performance Options of System Properties to see if a machine is using software or hardware DEP.  Having said that, I don’t recall any problems with VSE and software DEP, and I also don’t know why the exception wouldn’t work whereas turning it off completely would.  Sorry.

I assume you’ve checked the windows event log to make sure it’s actually DEP that’s killing your app?  I only ask because we’ve encountered some  cases that look a lot like DEP termination but aren’t.  This even included the problem going away once the DEP exclusion was added... which later turned out to be a red herring when the problem returned with the exclusion still in place.  The windows event log usually clears it up definitively.  Also add column “Data Execution Prevention” to the Windows Task Manager process tab if you haven’t already.

--Lorin

We have a rather strange problem. It’s well-established that VisualSmalltalk has issues with Data Execution Prevention. Up until now we’ve always simply asked the customer to list the application as a DEP exception, and this solves the problem. But what appears to be happening here is that, even though the application is listed as an exception, DEP is *still* being applied (and killing the application at launch); it is only if we set DEP to Opt-In rather than Opt-Out that the box allows the application to run.

It seems to be affecting only this one customer, but two different computers on that site (running different OS versions).

Has anyone else ever seen anything like this?

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***

*** this signature added by listserv *** *** Visit http://www.listserv.dfn.de/archives/vswe-l.html *** *** for archive browsing and VSWE-L membership management ***