| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
Note: This section describes an experimental feature which will be part of Autoconf in a forthcoming release. Although we believe Autotest is stabilizing, this documentation describes an interface which might change in the future: do not depend upon Autotest without subscribing to the Autoconf mailing lists. |
It is paradoxical that portable projects depend on nonportable tools to run their test suite. Autoconf by itself is the paragon of this problem: although it aims at perfectly portability, up to 2.13, its test suite was using DejaGNU, a rich and complex testing framework, but which is far from being standard on Unix systems. Worse yet, it was likely to be missing on the most fragile platforms, the very platforms that are most likely to torture Autoconf and exhibit deficiencies.
To circumvent this problem many package maintainers have developed their own testing framework, based on simple shell scripts whose sole output are their exit status: the test succeeded, or failed. In addition, most of these tests share some common patterns, what results in lots of duplicated code, tedious maintenance etc.
Following exactly the same reasoning that yielded to the inception of Autoconf, Autotest provides a test suite generation frame work, based on M4 macros, building a portable shell script. The suite itself is equipped with automatic logging and tracing facilities which greatly diminish the interaction with bug reporters, and simple timing reports.
Autoconf itself has been using Autotest for years, and we do attest that it has considerably improved the strength of the test suite, and the quality of bug reports. Other projects are known to use some generation of Autotest, such as Bison, Free Recode, Free Wdiff, GNU Tar, each of them having different needs, what slowly polishes Autotest as a general testing framework.
Nonetheless, compared to DejaGNU, Autotest is inadequate for interactive tool testing, which is probably its main limitation.
16.1 Using an Autotest Test Suite Autotest and the user 16.2 Writing `testsuite.at' Autotest macros 16.3 Running testsuiteScriptsRunning testsuitescripts16.4 Making testsuiteScriptsUsing autom4te to create testsuite
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
16.1.1 testsuiteScriptsThe concepts of Autotest 16.1.2 Autotest Logs Their contents
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
testsuite Scripts
Generating testing or validation suites using Autotest is rather easy.
The whole validation suite is held in a file to be processed through
autom4te, itself using GNU M4 under the scene, to
produce a stand-alone Bourne shell script which then gets distributed.
Neither autom4te nor GNU M4 are not needed anymore at
the installer end.
Each test of the validation suite should be part of some test group. A test group is a sequence of interwoven tests that ought to be executed together, usually because one test in the group creates data files than a later test in the same group needs to read. Complex test groups make later debugging more tedious. It is much better keeping keep only a few tests per test group, and if you can put only one test per test group, this is just ideal.
For all but the simplest packages, some file such as `testsuite.at' does not fully hold all test sources, as these are often easier to maintain in separate files. Each of these separate files holds a single test group, or a sequence of test groups all addressing some common functionality in the package. In such cases, file `testsuite.at' only initializes the whole validation suite, and sometimes do elementary health checking, before listing include statements for all other test files. The special file `package.m4', containing the identification of the package, is automatically included if found.
The validation scripts that Autotest produces are by convention called
testsuite. When run, testsuite executes each test
group in turn, producing only one summary line per test to say if that
particular test succeeded or failed. At end of all tests, summarizing
counters get printed. If any test failed, one debugging script gets
automatically generated for each test group which failed. These
debugging scripts are named `testsuite.nn', where nn is
the sequence number of the test group. In the ideal situation, none of
the tests fail, and consequently, no debugging script is generated out
of validation.
The automatic generation of debugging scripts for failed test has the purpose of easing the chase for bugs.
It often happens in practice that individual tests in the validation
suite need to get information coming out of the configuration process.
Some of this information, common for all validation suites, is provided
through the file `atconfig', automatically created by
AC_CONFIG_TESTDIR. For configuration informations which your
testing environment specifically needs, you might prepare an optional
file named `atlocal.in', instantiated by AC_CONFIG_FILES.
The configuration process produces `atconfig' and `atlocal'
out of these two input files, and these two produced files are
automatically read by the `testsuite' script.
Here is a diagram showing the relationship between files.
Files used in preparing a software package for distribution:
subfile-1.at ->.
... \
subfile-i.at ---->-- testsuite.at -->.
... / \
subfile-n.at ->' >-- autom4te* -->testsuite
/
[package.m4] ->'
|
Files used in configuring a software package:
.--> atconfig
/
[atlocal.in] --> config.status* --<
\
`--> [atlocal]
|
Files created during the test suite execution:
atconfig -->. .--> testsuite.log
\ /
>-- testsuite* --<
/ \
[atlocal] ->' `--> [testsuite.nn*]
|
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
When run, the test suite creates a log file named after itself, e.g., a
test suite named testsuite creates `testsuite.log'. It
contains a lot of information, usually more than maintainers actually
need, but therefore most of the time it contains all that is needed:
CC for subsequent runs(5). Autoconf faced exactly the same problem, and solved it by asking
users to pass the variable definitions as command line arguments.
Autotest requires this rule too, but has no means to enforce it; the log
then contains a trace of the variables the user changed.
AT_TESTED).
configure,
are appended. It contains the configuration flags and a detailed report
on the configuration itself.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The `testsuite.at' is a Bourne shell script making use of special
Autotest M4 macros. It often contains a call to AT_INIT nears
its beginning followed by one call to m4_include per source file
for tests. Each such included file, or the remainder of
`testsuite.at' if include files are not used, contain a sequence of
test groups. Each test group begins with one call to AT_SETUP,
it contains an arbitrary number of shell commands or calls to
AT_CHECK, and it completes with one call to AT_CLEANUP.
Autotest test suites rely on the PATH to find the tested program.
This saves from generating the absolute paths to the various tools, and
makes it possible to test installed programs. Therefore, knowing what
programs are being exercised is crucial to understand some problems in
the test suite itself, or its occasional misuses. It is a good idea to
also subscribe foreign programs you depend upon, to ease incompatibility
diagnostics.
AT_KEYWORDS.
Several invocations within a test group accumulate new keywords. In other words, don't fear registering several times the same keyword in a test group.
The commands must not redirect the standard output, nor the standard error.
If status, or stdout, or stderr is `ignore', then the corresponding value is not checked.
The special value `expout' for stdout means the expected output of the commands is the content of the file `expout'. If stdout is `stdout', then the standard output of the commands is available for further tests in the file `stdout'. Similarly for stderr with `expout' and `stderr'.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
testsuite Scripts Autotest test suites support the following arguments:
clean Makefile targets.
By default all the tests are performed (or described with `--list') in the default environment first silently, then verbosely, but the environment, set of tests, and verbosity level can be tuned:
The variable AUTOTEST_PATH specifies the testing path to prepend
to PATH. It handles specially relative paths (not starting with
`/'): they are considered to be relative to the top level of the
package being built. All the directories are made absolute, first
starting from the top level build tree, then from the
source tree. For instance `./testsuite
AUTOTEST_PATH=tests:bin' for a `/src/foo-1.0' source package built
in `/tmp/foo' results in `/tmp/foo/tests:/tmp/foo/bin' and
then `/src/foo-1.0/tests:/src/foo-1.0/bin' being prepended to
PATH.
AT_SETUP or AT_KEYWORDS) match all the keywords
of the comma separated list keywords.
Running `./testsuite -k autoupdate,FUNC' will select all the tests tagged with `autoupdate' and `FUNC' (as in `AC_CHECK_FUNC', `AC_FUNC_FNMATCH' etc.) while `./testsuite -k autoupdate -k FUNC' runs all the tests tagged with `autoupdate' or `FUNC'.
| [ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
testsuite Scripts For putting Autotest into movement, you need some configuration and Makefile machinery. We recommend, at least if your package uses deep or shallow hierarchies, that you use `tests/' as the name of the directory holding all your tests and their `Makefile'. Here is a check list of things to do.
AT_PACKAGE_STRING, the
full signature of the package, and AT_PACKAGE_BUGREPORT, the
address to which bug reports should be sent. For sake of completeness,
we suggest that you also define AT_PACKAGE_NAME,
AT_PACKAGE_TARNAME, and AT_PACKAGE_VERSION.
See section 4.1 Initializing configure, for a description of these variables. We
suggest the following Makefile excerpt:
$(srcdir)/package.m4: $(top_srcdir)/configure.ac
{ \
echo '# Signature of the current package.'; \
echo 'm4_define([AT_PACKAGE_NAME], [@PACKAGE_NAME@])'; \
echo 'm4_define([AT_PACKAGE_TARNAME], [@PACKAGE_TARNAME@])'; \
echo 'm4_define([AT_PACKAGE_VERSION], [@PACKAGE_VERSION@])'; \
echo 'm4_define([AT_PACKAGE_STRING], [@PACKAGE_STRING@])'; \
echo 'm4_define([AT_PACKAGE_BUGREPORT], [@PACKAGE_BUGREPORT@])'; \
} >$(srcdir)/package.m4
|
Be sure to distribute `package.m4' and to put it into the source hierarchy: the test suite ought to be shipped!
AC_CONFIG_TESTDIR.
AUTOTEST_PATH to test-path (see section 16.3 Running testsuite Scripts).
AC_CONFIG_FILES command includes substitution for
`tests/atlocal'.
With Automake, here is a minimal example about how to link `make check' with a validation suite.
EXTRA_DIST = testsuite.at testsuite
TESTSUITE = $(srcdir)/testsuite
check-local: atconfig atlocal $(TESTSUITE)
$(SHELL) $(TESTSUITE)
AUTOTEST = $(AUTOM4TE) --language=autotest
$(TESTSUITE): $(srcdir)/testsuite.at
$(AUTOTEST) -I $(srcdir) $@.at -o $@.tmp
mv $@.tmp $@
|
You might want to list explicitly the dependencies, i.e., the list of the files `testsuite.at' includes.
With strict Autoconf, you might need to add lines inspired from the following:
subdir = tests
atconfig: $(top_builddir)/config.status
cd $(top_builddir) && \
$(SHELL) ./config.status $(subdir)/$@
atlocal: $(srcdir)/atlocal.in $(top_builddir)/config.status
cd $(top_builddir) && \
$(SHELL) ./config.status $(subdir)/$@
|
and manage to have `atconfig.in' and $(EXTRA_DIST)
distributed.
| [ << ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |