Dolphin 5.1 fails "verified for Windows 2003 server" certification

I want to submit my application to the "verified for windows 2003 server"
certiciation. This test is just a subset of the "certified for windows 2003
server" certification.

This is currently not possible because Dolphin crashes without notice when
trying to test criterion 4.1:

4.1 Appropriate Resource Use
The heap, critical sections, and handles can be misused which results in
less reliable
applications and failures with subtle circumstances that impact customers
but may not be
easily reproduced. Each of these items are easily tested to ensure they are
not misused.
Your application must be able to perform all of its primary and secondary
functionality with no critical errors while running under AppVerifier
configured to detect heap corruptions, invalid locks usage (critical section
use), and invalid handle usage.

I testest this with my applications as well as a standard Dolphin 5.1
installation.

This is quite important to fix this because we need this certification for
our application. Also we want to make other certifications like "verified
for Windows xp" or "certified for windows XP".

Some more information and links:
Specification "verified for windows 2003 server":
http://www.veritest.com/downloads/certification/verified_for_windows_server_2003_test_specification_1.0.pdf

Other certifications:
http://www.veritest.com/certification/ms/default.asp?visitor=#Server

Regards

Carsten Haerle

Carsten,

> I want to submit my application to the "verified for windows 2003 server"
> certiciation. This test is just a subset of the "certified for windows 2003
> server" certification.
>
> This is currently not possible because Dolphin crashes without notice when
> trying to test criterion 4.1:

Does it give a crash dump?  Does it crash at startup or shutdown?  Does
it say which type of access (memory, synchronization, handles) are at
fault?  If not, can you run with one check at a time to isolate it?


> 4.1 Appropriate Resource Use
> The heap, critical sections, and handles can be misused which results in
> less reliable
> applications and failures with subtle circumstances that impact customers
> but may not be
> easily reproduced. Each of these items are easily tested to ensure they are
> not misused.

Question gang: how can something that is not easily reproduced also be
easy to test?  I am familiar with methods for building this kind of
testing into a program (heap checks, etc.), but it seems that it would
be fairly tricky to do it from the outside.


> Your application must be able to perform all of its primary and secondary
> functionality

How do they address coverage?  Do you turn the software over to someone
else to pound until it breaks or they get bored?


 > with no critical errors while running under AppVerifier
> configured to detect heap corruptions, invalid locks usage (critical section
> use), and invalid handle usage.

Does invalid locks mean leaving more critical sections than are entered,
or is there more to it?

FWIW, my hunch is that if this indeed detects a problem in Dolphin, it
will turn out to be sequencing of messages or missing/extraneous calls
to default window procedures, etc.  Obviously, it will be great if it
finds something that has gone unnoticed.  However, I suspect it's even
money that the test is wrong.  For example, is it really possible for
something external to monitor Dolphin's memory and correctly understand
the actions of the garbage collector and memory manager?

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

Bill,

1) Testing for heap and handle problems can be done from the outside!
Microsoft provides a tool called "AppVerifier" from Microsoft. The
application runs under the control of this tool. As I understand the tool
hooks into various APIs to test if the tested application behaves well.

Here is how it could work in my opinion: E.g. to test for buffer overflows
the tool intercepts the heap allocation API, then it allocates a slightly
larger memory block itself and surrounds it with special magic values. It
gives this memory block to the application and periodically checks whether
the magic values are overwritten, which would indicate a buffer overflow.
This is just one example. Another example: To check for handle leakage it
would just hook into all handle creation and handle release APIs and check
whether all allocated handles are freed upon application termination.

To your questions what each tests exactly means you have to look into the
AppVerifier documentation or the certification specification.

Concerning Dolphin: The AppVerfied offers many other tests, which sometimes
also produce warnings or errors. These should be fixed also, although their
are not part of the "Verified for" certification but only part of the much
more rigid "Certified for" or "Designed for" certifications. When the heap
and handle tests are performed the application or Dolphin development
environment process starts but just vanishes after a couple of seconds
without notice. I didn't try to activate a debugger (which is supported from
the AppVerified tool) because I don't have the source code for the VM to see
what went wrong.

It should be possible to write 100% Microsoft guidelines compliant
applications with Dolphin and the "Verified for" tests are the lowest
possible certification level.

The "AppVerifier" tool can be downloaded at:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnappcom/html/AppVerifier.asp

The "VeriTest Installation Analyzer" tool can be downloaded at:
http://www.veritest.com/certification/ms/xp/jpn/xp_analyzer.asp

2) How is the test performed at Veritest

Many test cases are well specified, like the installation test cases (e.g.
clean deinstallation, no replacing of system dlls like in Windows 95 days,
proper handling of shared DLLs, and so on) or the heap and handles tests.
Microsoft supplies some tools with which these criteria can be easily tested
like "VeriTest Installation Analyzer"  and "AppVerifier".

One thing is a little bit vague, but this by nature of the problem. What is
"primary functionality" and how to test it? There is not perfect solutions
but as I understand Veritest does the following:

You have to hand the application to Veritest and describe test cases which
cover the "primary functionality". I think they will try these test cases,
and check the documentation to see if your test cases really represent the
"primary functionality" if the application in some sense.

Regards

Carsten

"Bill Schwab" <[hidden email]> schrieb im Newsbeitrag
news:cbq7i5$f3m$[hidden email]...
> Carsten,
>
> > I want to submit my application to the "verified for windows 2003
server"
> > certiciation. This test is just a subset of the "certified for windows
2003
> > server" certification.
> >
> > This is currently not possible because Dolphin crashes without notice
when customers
> > but may not be
> > easily reproduced. Each of these items are easily tested to ensure they
are
> > not misused.
>
> Question gang: how can something that is not easily reproduced also be
> easy to test?  I am familiar with methods for building this kind of
> testing into a program (heap checks, etc.), but it seems that it would
> be fairly tricky to do it from the outside.
>
>
> > Your application must be able to perform all of its primary and
secondary
> > functionality
>
> How do they address coverage?  Do you turn the software over to someone
> else to pound until it breaks or they get bored?
>
>
>  > with no critical errors while running under AppVerifier
> > configured to detect heap corruptions, invalid locks usage (critical
section

"Carsten Haerle" <[hidden email]> wrote in message
news:40e14c57$0$27061$[hidden email]...
> It should be possible to write 100% Microsoft guidelines compliant
> applications with Dolphin and the "Verified for" tests are the lowest
> possible certification level.

that would be nice, but I am not sure if it possible. Dolphin is executed
under VM, which let's say behaves a bit different than common applications.
Maybe it is just few minor gliches that need to be corrected, and maybe this
test is not created with VM-s in mind, and passing it would require major
rewrite of VM with possible performance penalties. I do not know internals
of Dolphin VM, but my gut feeling tells me that these kind of things are
usually not easily and quickly resolved.

rush
--
http://www.templatetamer.com/

The guidelines don't favour a specific programming language. They are
general guidelines which every application under windows should conform too,
examples:

* server applications must not have memory or handle leaks, otherwise the
server will be non-functional after some time
* the application should not have buffer overflows (as far as this can be
tested)
* the application must not free memory and access freed memory afterwards
* one must be able to install and deinstall the application properly
* no system files are replaced
* no unnecessary reboots

The application could be written Visual Basic Script, Java or C and it
doesn't matter which execution model (compiled, semicompiled, interpreted,
VM-based) the appplication or programming language has.

Regards

Carsten Haerle

"rush" <[hidden email]> schrieb im Newsbeitrag
news:cbrs99$lag$[hidden email]...
> "Carsten Haerle" <[hidden email]> wrote in message
> news:40e14c57$0$27061$[hidden email]...
> > It should be possible to write 100% Microsoft guidelines compliant
> > applications with Dolphin and the "Verified for" tests are the lowest
> > possible certification level.
>
> that would be nice, but I am not sure if it possible. Dolphin is executed
> under VM, which let's say behaves a bit different than common
applications.
> Maybe it is just few minor gliches that need to be corrected, and maybe
this

Carsten,

> The guidelines don't favour a specific programming language. They are
> general guidelines which every application under windows should conform too,
> examples:
>
> * server applications must not have memory or handle leaks, otherwise the
> server will be non-functional after some time

True, but how does one identify a leak?  By that I mean, how can an
external entity conclude that an allocated block should have been freed
by a certain point, with the exception of app shutdown?  In the latter
case, while sloppy, the OS takes care of it, so it's not _really_ a leak.



> * the application should not have buffer overflows (as far as this can be
> tested)
> * the application must not free memory and access freed memory afterwards

I'll give you those :)


> * no unnecessary reboots

Tell that one to Microsoft ;)


> The application could be written Visual Basic Script, Java or C and it
> doesn't matter which execution model (compiled, semicompiled, interpreted,
> VM-based) the appplication or programming language has.

That would be ideal, but I'm with Rush in suspecting that this could be
a tool getting confused by things it does not understand.

Have a good one,

Bill

--
Wilhelm K. Schwab, Ph.D.
[hidden email]

"Carsten Haerle" <[hidden email]> wrote in message
news:40e04749$0$149$[hidden email]...
Firstly I should state that we make no claims that Dolphin passes any sort
of Microsoft "test", especially not one associated with a version of Windows
released well after the current version so that we could not even have run
basic tests on it when that version of Dolphin was released. In other words
we
do not warrant that either the Dolphin development itself or applications
created using it will pass any logo compliance tests.

Now onto why it fails to run; it was an interesting debugging task! It turns
out that the crash dump hints at the issue. You should see that the system
crashes due to a failure to perform a C-runtime library call
(setlocale())early on in the startup, which then causes a lot of knock-on
errors in attempting to report the original error. I soon discovered that
this was because when running the in AppVerifier "instrumented" mode, the
call to GetProcAddress in the VM's external call primitive was returning
0/NULL for the proc. address of setlocale(). Closer examination showed that
the module handle for the CRT being passed to GetProcAddress was that of
"verifier.dll" rather than the CRT dll, and this is happening because of the
way the Dolphin VM finds the module handle for the C-runtime library, to
which it is itself linked, that it passes into the image at startup. Really
the instrumentation inserted by the AppVerifier should be transparent to the
application, but as I have commonly found with this sort of tool (another
example being Purify) this does not seem to be the case for complex
applications.

All libraries with the exception of the CRT and kernel are opened using
LoadLibrary, but for chicken-before-egg and multiple build configuration
reasons, the CRT handle is determined by locating the base address of a
function known to be in the module, in this case it was malloc(). Its not
entirely surprising that the AppVerifier replaces references to malloc()
with references to one of its own wrapper functions (in verifier.dll it
seems), and so the module handle was not that of the CRT, and the
GetProcAddress therefore fails. This was very easily corrected by using a
function other than malloc. I've placed a new hotfix VM build at
http//object-arts.com/Lib/Update/Dolphin/5.1/DolphinVM005.zip, and new ToGo
stubs (which include the VM) in the same location in the file ToGoStubs.zip.
Please note that I have not had time to do much testing of the rebuilt VM
yet, and none whatsoever of the stubs. However the change is very small and
I am not anticipating any new issues to arise as a result of it.

Incidentally from my brief run of the AppVerifier it showed up very few
issues, and most of those were calls to obsolete APIs in components external
to Dolphin (such as a calls to GetTextExtentPointW in User32.dll itself, and
a call to RegQueryValueW in ole32.dll) and therefore outside of our control.
It certainly didn't find any memory or lock errors in my tests.

Anyway, please let us know what you find out and if you are successful in
getting your application verified.

>
> This is quite important to fix this because we need this certification for
> our application. Also we want to make other certifications like "verified
> for Windows xp" or "certified for windows XP".

It is not something we have paid any attention to up to now, since we are
resource constrained and confident that the reliability of the Dolphin
environment, and applications developed in a professional manner using it,
is considerably better than average on Windows, and certainly much better
than Microsoft's own development tools in my experience. Also I doubt that a
tool such as AppVerifier designed, I suspect, primarily for testing
traditionaly statically compiled applications, will be able to find out much
about the behaviour of a dynamic system like Dolphin, because it probably
won't intercept all the calls. However I can quite understand how getting a
"pass" might be important for an application for marketing purposes (even if
technically irrelevant), and so it may be higher on your priority list than
has been on ours.

Regards

Blair

Bill Schwab wrote:

Actually, this is pretty easy to test for w/ server apps.
If you run a server through its paces, the memory usage
should eventually settle into a static trough.
If the trough keeps inching upwards with repeated
cycling of the features under test, tada - leakage.

The OS itself is verified like this,
and can last for several *days*.

 
>
> > * the application should not have buffer overflows (as far as this can be
> > tested)
> > * the application must not free memory and access freed memory afterwards
>
> I'll give you those :)


Whoa - can't have these, they're already spoken for.
What would virus writers do for a living if this
was actually addressed?  Oh - you mean third party
apps have to pass this.  Ok, that's different.

Gotta love the caveat - (as far as this can be tested).
Perhaps they just want ideas here, you know, like pass
everything, even if they fail this one, but
  "Buy any company that can do this"


>
> > * no unnecessary reboots
>
> Tell that one to Microsoft ;)


Hey, that's not fair, Bill.
There is a carefully determined definition of
what "necessary" means in this instance.
Something like - "Oh cripes, you're changing
*that* preference? A reboot is obviously required."

The "uneccessary" reboots thing is a recent
innovation, that you should be learning about now,
so your stuff can also become "state of the art".

E.g. here is my current favorite --
  "found new hardware - press enter or click
   to continue with the installation".

Unfortunately, the "new" hardware is discovered
before the newly displaced USB keyboard and mouse.
They smartly avoid this kind of unneccessary reboot.


>
>


If you check the list again, you'll see that
some 15% of the bullets would be really hard
for the OS to just take care of. Ok, kinda hard.
At any rate, it's only the remainder that are subject
to false negatives.  They really are setting examples,
and raising the bar of excellence.  So what if it
amounts to a ton of useless busywork for some people.  

Someone has to lead.


Cheers,

-cstb

> Actually, this is pretty easy to test for w/ server apps.
> If you run a server through its paces, the memory usage
> should eventually settle into a static trough.
> If the trough keeps inching upwards with repeated
> cycling of the features under test, tada - leakage.

I agree that such behavior is very suggestive of a leak, but it could
also be the result of increasing work load (what is to say that
exercising features does not cause long-term work for the machine?) or
perhaps memory management that (for good or bad) grows its heap more
easily than it shrinks it.

I am not arguing with you, but I am suggesting that a one-size-fits-all
test is eventually going to turn up false positives, if only for very
unusual software.


> The OS itself is verified like this,
> and can last for several *days*.

Is that all :)  I have one win2k server that, except when updates
require more frequent reboots, runs for three or more months at a time.
  Eventually, the sockets start getting a little tired - not sure
whether that's me, Dolphin, or Windows??


> Whoa - can't have these, they're already spoken for.
> What would virus writers do for a living if this
> was actually addressed?  Oh - you mean third party
> apps have to pass this.  Ok, that's different.

I am not sure I'm following you.  I meant that clearly these things are
bad, and so I have no problem with trying to eliminate them whenever
possible.  I am not alone in being able to say that my life would be
simpler if buffer overruns were less common.


>>>* no unnecessary reboots
>>
>>Tell that one to Microsoft ;)
>
> Hey, that's not fair, Bill.
> There is a carefully determined definition of
> what "necessary" means in this instance.

Which IMHO, makes the humor all the more effective.


> The "uneccessary" reboots thing is a recent
> innovation, that you should be learning about now,
> so your stuff can also become "state of the art".

I am far more concerned with keeping my software in service, evolving
and robust than in making it "state of the art".  Only time will tell
whether that is good judgement on my part.


> If you check the list again, you'll see that
> some 15% of the bullets would be really hard
> for the OS to just take care of. Ok, kinda hard.
> At any rate, it's only the remainder that are subject
> to false negatives.  They really are setting examples,
> and raising the bar of excellence.  So what if it
> amounts to a ton of useless busywork for some people.  

It's a big deal if you are one of those people being saddled with
useless work.


> Someone has to lead.

Fortunately, there are multiple leaders going in different directions.
It is up to us to decide which group to join, and for which parts of the
journey.

Have a good one,

Bill


--
Wilhelm K. Schwab, Ph.D.
[hidden email]