DejaGnu conforms to the POSIX standard for test frameworks.
POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET.(3)
The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.
As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. DejaGnu supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:
PASS, instead of
XPASS, must also be returned for test cases which were expected to fail and did not. This means that
PASSis in some sense more ambiguous than if
XPASSis also used. For information on
XFAIL, see section Using
FAILmessage is based on the test case only. Other messages are used to indicate a failure of the framework. As with
PASS, POSIX tests must return
XFAILeven if a failure was expected.
FAILbefore a test run can be considered finished. Note that for POSIX, each assertion must produce a test result code. If the test isn't actually run, it must produce
UNRESOLVEDrather than just leaving that test out of the output. This means that you have to be careful when writing tests, to not carelessly use tcl statements like
return---if you alter the flow of control of the tcl code you must insure that every test still produces some result code. Here are some of the ways a test may wind up
ERRORfrom DejaGnu while processing the test, or because there were three or more
ERRORmessages can invalidate the output of the test. This usually requires a human being to examine the output to determine what really happened--and to improve the test case.
The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:
gethostnamewould never work on a target board running only a boot monitor.
DejaGnu uses the same output procedures to produce these messages for
all test suites, and these procedures are already known to conform to
POSIX 1003.3. For a DejaGnu test suite to conform to POSIX
1003.3, you must avoid the
setup_xfail procedure as described in
PASS section above, and you must be careful to return
UNRESOLVED where appropriate, as described in the
UNRESOLVED section above.
Go to the first, previous, next, last section, table of contents.