DejaGnu conforms to the POSIX standard for test frameworks.
POSIX standard 1003.3 defines what a testing framework needs to provide, in order to permit the creation of POSIX conformance test suites. This standard is primarily oriented to running POSIX conformance tests, but its requirements also support testing of features not related to POSIX conformance. POSIX 1003.3 does not specify a particular testing framework, but at this time there is only one other POSIX conforming test framework: TET.(3)
The POSIX documentation refers to assertions. An assertion is a description of behavior. For example, if a standard says "The sun shall shine", a corresponding assertion might be "The sun is shining." A test based on this assertion would pass or fail depending on whether it is daytime or nighttime. It is important to note that the standard being tested is never 1003.3; the standard being tested is some other standard, for which the assertions were written.
As there is no test suite to test testing frameworks for POSIX 1003.3 conformance, verifying conformance to this standard is done by repeatedly reading the standard and experimenting. One of the main things 1003.3 does specify is the set of allowed output messages, and their definitions. Four messages are supported for a required feature of POSIX conforming systems, and a fifth for a conditional feature. DejaGnu supports the use of all five output messages; in this sense a test suite that uses exactly these messages can be considered POSIX conforming. These definitions specify the output of a test case:
PASS
XFAIL
PASS
, instead of XPASS
, must also be returned for test
cases which were expected to fail and did not. This means that
PASS
is in some sense more ambiguous than if XPASS
is also
used. For information on XPASS
and XFAIL
, see
section Using runtest
.
FAIL
FAIL
message is based on the test case only. Other messages are used to
indicate a failure of the framework.
As with PASS
, POSIX tests must return FAIL
rather
than XFAIL
even if a failure was expected.
UNRESOLVED
PASS
or
FAIL
before a test run can be considered finished.
Note that for POSIX, each assertion must produce a test result
code. If the test isn't actually run, it must produce UNRESOLVED
rather than just leaving that test out of the output. This means that
you have to be careful when writing tests, to not carelessly use tcl
statements like return
---if you alter the flow of control of the
tcl code you must insure that every test still produces some result code.
Here are some of the ways a test may wind up UNRESOLVED
:
ERROR
from DejaGnu while processing the test, or because there
were three or more WARNING
messages. Any WARNING
or
ERROR
messages can invalidate the output of the test. This
usually requires a human being to examine the output to
determine what really happened--and to improve the test case.
UNTESTED
The only remaining output message left is intended to test features that are specified by the applicable POSIX standard as conditional:
UNSUPPORTED
gethostname
would never work on a target board running only a
boot monitor.
DejaGnu uses the same output procedures to produce these messages for
all test suites, and these procedures are already known to conform to
POSIX 1003.3. For a DejaGnu test suite to conform to POSIX
1003.3, you must avoid the setup_xfail
procedure as described in
the PASS
section above, and you must be careful to return
UNRESOLVED
where appropriate, as described in the
UNRESOLVED
section above.
Go to the first, previous, next, last section, table of contents.