Go to the first, previous, next, last section, table of contents.


The DejaGnu Implementation

DejaGnu is entirely written in expect, which uses Tcl as a command language. expect serves as a very programmable shell; you can run any program, as with the usual Unix command shells--but once the program is started, your expect script has fully programmable control of its input and output. This does not just apply to the programs under test; expect can also run any auxiliary program, such as diff or sh, with full control over its input and output.

DejaGnu itself is merely a framework for the set of test suites distributed separately for each GNU tool. Future releases of GNU tools will include even more tests, developed throughout the free software community.

runtest is the glue to tie together and manage the test scripts. The runtest program is actually a simple Bourne shell script that locates a copy of the expect shell and then starts the main Tcl code, runtest.exp. runtest.exp itself has these essential functions:

  1. Parse the command line options, load the library files, and load the default configuration files.
  2. Locating the individual test scripts. runtest.exp locates the tests by exploiting a straightforward naming convention based on the string you specify with the `--tool' option.
  3. Providing an extended test environment, by defining additional Tcl procedures beyond those already in expect.
  4. Locating target-dependent functions, to standardize the test environment across a wide variety of test platforms.

Conventions for using tool names

DejaGnu uses `$tool', the name of the tool under test, to tie together the testing configuration in a straightforward but flexible way. If there is only one testsuite for a particular application, then `$tool' is optional.

`$tool' is not used to invoke the tool, since sites that run multiple configurations of a particular tool often call each configuration by a different name. runtest uses the configuration-dependent variables captured in `site.exp' to determine how to call each tool.

runtest uses tool names to find directories containing tests. runtest scans the source directory (specified with --srcdir) for all directories whose names start with the tool name. It is a common practice to put a period after the tool part of the name. For instance, directories that start with `g++.' contain G++ tests. To add a new test, just put it in any directory (create an entirely new directory, if you wish) whose name follows this convention.

A test is any file in an appropriately named subdirectory whose name ends in `.exp' (the conventional way of naming expect scripts). These simple naming conventions make it as simple as possible to install new tests: all you must do is put the test in the right directory.

runtest sorts the tests in each subdirectory by name (using the Tcl lsort command) and runs them in the resulting order.

Initialization module

The initialization module (or "init file") has two purposes: to provide tool and target dependent procedures, and to start up an interactive tool to the point where it is ready to operate. The latter includes establishing communications with the target. All the tests for interactive programs assume that the tool is already running and communicating. Initialization modules for non-interactive programs may only need to supply the support functions.

Each test suite directory must contain (in its `config' subdirectory) a separate initialization module for each target. The appropriate init file is can be named several ways. The prefered name is the os part of the canonical configuration name with .exp as the suffix. An example would be that for an m68k-coff system, the target_os part would be coff. The next way is for system where there are short filenames, or a shortcut is desired to refer to the OS name for that target. This is uses the value of $target_abbrev rather than the target_os.

The final file looked for is simply `default.exp'. If there is only one operating system to support, then this file can be used. It's main purpose is to offer some support for new operating systems, or for unsupported cross targets. The last file looked for is `unknown.exp'. This is usually limited to error handling for unsupported targets. It's whole contents is typically.

perror "Sorry, there is no support for this target"
exit 1

At the beginning of the init file, you must first determine the proper executable name of the tool to execute, since the actual name of the tool to be tested my vary from system to system. Here's an example for the GNU C compiler.

global AR
# look for the archiver ar
if ![info exists AR] {
    set AR [findfile $base_dir/../../binutils/ar $base_dir/../../binutils/ar [tr
ansform ar]]
    verbose "AR defaulting to $AR" 2
}
}

global CFLAGS
if ![info exists CFLAGS] then {
    set CFLAGS ""
}

It is always a good idea to first check the variable, and only set it if it has not yet been defined. Often the proper value of AR is set on the command line that invokes `runtest'.

The findfile procedure takes as it's first argument a file name to look for. The second argument is returned if the file is found, and the third argument is returned if the file is not found. base_dir is set internally by DejaGnu to the top level directory of the object tree.

The transform procedure takes as its argument the native name of a tool (such as `gcc' for the compiler), and returns the name as configured for that tool in the current installation. (For example, a cross-compiling version of GNU CC that generates MIPS code may be installed with a name like mips-idt-ecoff-gcc.)

In a test running native, writing the Tcl code for initialization is usually quite simple. For cross configurations, however, more elaborate instructions are usually needed to describe how to talk to a remote target.

Each initialization module defines up to four procedures with standard names and purposes. The names of these procedures begin with `$tool', the string that identifies tests for a particular tool: $tool_start, $tool_load, $tool_exit, and $tool_version. For example, the start procedure for GDB is called gdb_start. (Since start procedures are used differently for batch and interactive tools, however, runtest itself never calls the start procedure. Init files for interactive tools are expected to end by running the start procedure.)

The initialization module is also a good place to call load_lib to get any collections of utility procedures meant for a family of test cases, and to set up default values for any additional Tcl variables needed for a specific set of tests.

See section Target dependent procedures, for full descriptions of these procedures.

DejaGnu procedures

DejaGnu provides these Tcl procedures for use in test scripts. You can also use any standard expect or Tcl function. These procedures are stored in libraries, which DejaGnu loads at runtime. Here's explanation of the library procedures that get loaded at runtime. All other librarys are optional, and need to be loaded by the testsuite.

Core Internal Procedures

See section A POSIX conforming test framework, for more detailed explanations of the test outcomes (`FAIL', `PASS', `UNTESTED', `UNRESOLVED', `UNSUPPORTED').

perror "string number"
Declares a severe error in the testing framework itself. perror writes in the log files a message beginning with `ERROR', appending the argument string. If the optional number is supplied, then this is used to set the internal count of errors to that value. As a side effect, perror also changes the effect of the next pass or fail command: the test outcome becomes `UNRESOLVED', since an automatic `PASS' or `FAIL' cannot be trusted after a severe error in the test framework. If the optional numeric value is `0', then there are no further side effects to calling this function, and the following test outcome doesn't become `UNRESOLVED'. This can be used for errors with no known side effects.
warning "string number"
Declares detection of a minor error in the test case itself. warning writes in the log files a message beginning with `WARNING', appending the argument string. Use warning rather than error for cases (such as communication failure to be followed by a retry) where the test case can recover from the error. If the optional number is supplied, then this is used to set the internal count of warnings to that value. As a side effect, warning_threshold or more calls to warning in a single test case also changes the effect of the next pass or fail command: the test outcome becomes `UNRESOLVED' since an automatic `PASS' or `FAIL' may not be trustworthy after many warnings. If the optional numeric value is `0', then there are no further side effects to calling this function, and the following test outcome doesn't become `UNRESOLVED'. This can be used for errors with no known side effects.
note "string"
Appends an informational message to the log file. note writes in the log files a message beginning with `NOTE', appending the argument string. Use note sparingly. verbose should be used for most such messages, but in cases where a message is needed in the log file regardless of the verbosity level use note.
pass "string"
Declares a test to have passed. pass writes in the log files a message beginning with `PASS' (or XPASS, if failure was expected), appending the argument string.
fail "string"
Declares a test to have failed. fail writes in the log files a message beginning with `FAIL' (or XFAIL, if failure was expected), appending the argument string.
unresolved "string"
Declares a test to have an unresolved outcome. unresolved writes in the log file a message beginning with `UNRESOLVED', appending the argument string. This usually means the test did not execute as expected, and a human being must go over results to determine if it passed or failed (and to improve the test case).
untested "string"
Declares a test was not run. untested writes in the log file a message beginning with `UNTESTED', appending the argument string. For example, you might use this in a dummy test whose only role is to record that a test does not yet exist for some feature.
unsupported "string"
Declares that a test case depends on some facility that does not exist in the testing environment. unsupported writes in the log file a message beginning with `UNSUPPORTED', appending the argument string.
get_warning_threshold
Returns the current value of warning_threshold. The default value is 3.
set_warning_threshold threshold
Sets the value of warning_threshold. A value of 0 disables it: calls to warning will not turn a `PASS' or `FAIL' into an `UNRESOLVED'.
transform "toolname"
Generates a string for the name of a tool as it was configured and installed, given its native name (as the argument toolname). This makes the assumption that all tools are installed using the same naming conventions: it extrapolates from the invocation name for `runtest'. For example, if you call runtest as `m68k-vxworks-runtest', the result of ` transform "gcc" ' is `m68k-vxworks-gcc'.
ishost "host"
Tests for a particular host environment. If the currently configured host matches the argument string, the result is 1; otherwise the result is 0. host must be a full three-part configure host name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters, using shell syntax, to specify sets of names).
istarget "target"
Tests for a particular target environment. If the currently configured target matches the argument string, the result is 1; otherwise the result is 0. target must be a full three-part configure target name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters, using shell syntax, to specify sets of names). If it is passed a NULL string, then it returns the name of the build canonical configuration.
isbuild "host"
Tests for a particular build host environment. If the currently configured host matches the argument string, the result is 1; otherwise the result is 0. host must be a full three-part configure host name; in particular, you may not use the shorter nicknames supported by configure (but you can use wildcard characters, using shell syntax, to specify sets of names). If it is passed a NULL string, then it returns the name of the build canonical configuration. item is3way "host" Tests for a canadian cross. This is when the tests will be run on a remotly hosted cross compiler. If it is a canadian cross, then the result is 1; otherwise the result is 0.
isnative
Tests whether the current configuration has the same host and target. When it runs in a native configuration this procedure returns a 1; otherwise it returns a 0.
load_lib "library-file"
Loads the file library-file by searching a fixed path built into runtest. If DejaGnu has been installed, it looks in a path starting with the installed library directory. If you are running DejaGnu directly from a source directory, without first running `make install', this path defaults to the current directory. In either case, it then looks in the current directory for a directory called lib. If there are duplicate definitions, the last one loaded takes precedence over the earlier ones.
setup_xfail "config [bugid]"
Declares that the test is expected to fail on a particular set of configurations. The config argument must be a list of full three-part configure target name; in particular, you may not use the shorter nicknames supported by configure (but you can use the common shell wildcard characters to specify sets of names). The bugid argument is optional, and used only in the logging file output; use it as a link to a bug-tracking system such as GNATS (see section `Overview' in Tracking Bugs With GNATS). Once you use setup_xfail, the fail and pass procedures produce the messages `XFAIL' and `XPASS' respectively, allowing you to distinguish expected failures (and unexpected success!) from other test outcomes. Warning: you must clear the expected failure after using setup_xfail in a test case. Any call to pass or fail clears the expected failure implicitly; if the test has some other outcome, e.g. an error, you can call clear_xfail to clear the expected failure explicitly. Otherwise, the expected-failure declaration applies to whatever test runs next, leading to surprising results.
clear_xfail config
Cancel an expected failure (previously declared with setup_xfail) for a particular set of configurations. The config argument is a list of configuration target names. It is only necessary to call clear_xfail if a test case ends without calling either pass or fail, after calling setup_xfail.
verbose [-log] [-n] [--] "string" number
Test cases can use this function to issue helpful messages depending on the number of `--verbose' options on the runtest command line. It prints string if the value of the variable verbose is higher than or equal to the optional number. The default value for number is 1. Use the optional `-log' argument to cause string to always be added to the log file, even if it won't be printed. Use the optional `-n' argument to print string without a trailing newline. Use the optional `--' argument if string begins with "-".

Remote Communication Procedures

`lib/remote.exp' defines these functions, for establishing and managing communications:

Procedures to establish a connection: Each of these procedures tries to establish the connection up to three times before returning. Warnings (if retries will continue) or errors (if the attempt is abandoned) report on communication failures. The result for any of these procedures is either -1, when the connection cannot be established, or the spawn ID returned by the expect command spawn.

It use the value of the connect field in the target_info array (was connectmode as the type of connection to make. Current supported connection types are tip, kermit, telnet, rsh, rlogin, and netdata. If the --reboot option was used on the runtest command line, then the target is rebooted before the connection is made.

remote_open type
Remote Connection Procedure. This is passed host or target. Host or target refers to whether it is a connection to a remote target, or a remote host. This opens the connection to the desired target or host using the default values in the configuration system. It returns that spawn_id of the process that manages the connection. This value can be used in expect or exp_send statements, or passed to other procedures that need the connection process's id. This also sets the fileid field in the target_info array.
remote_close shellid
shellid is value returned by a call to remote_open. This closes the connection to the target so resources can be used by others. This parameter can be left off if the fileid field in the target_info array is set.
telnet hostname port
rlogin hostname
rsh hostname
IP network procedures. hostname refers to the IP address or name (for example, an entry in `/etc/hosts') for this target. The procedure names reflect the Unix utility used to establish a connection. The optional port is used to specify the IP port number. The value of the netport field in the target_info array is used. (was $netport) This value has two parts, the hostname and the port number, seperated by a :. If host or target is used in the hostname field, than the config array is used for all information.
tip port
Serial line procedure. Connect using the Unix utility tip. port must be a name from the tip configuration file `/etc/remote'. Often, this is called `hardwire', or something like `ttya'. This file holds all the configuration data for the serial port. The value of the serial field in the target_info array is used. (was $serialport) If host or target is used in the port field, than the config array is used for all information.
kermit port bps
Serial line procedure. Connect using the program kermit. port is the device name, e.g. `/dev/ttyb'. bps is the line speed to use (in bits per second) for the connection. The value of the serial field in the target_info array is used. (was $serialport) If host or target is used in the port field, than the config array is used for all information.

Procedures to manage a connection:

tip_download spawnid file
Download `file' to the process spawnid (the value returned when the connection was established), using the ~put command under tip. Most often used for single board computers that require downloading programs in ASCII S-records. Returns 1 if an error occurs, 0 otherwise.
exit_remote_shell spawnid
Exits a remote process started by any of the connection procedures. spawnid is the result of the connection procedure that started the remote process.
download file [ spawnid ]
After you establish a connection to a target, you can download programs using this command. download reads in file (object code in S-record format) and writes it to the device controlling this spawnid. (From the point of view of the target, the S-record file comes in via standard input.) If you have more than one target active, you can use the optional argument spawnid to specify an alternative target (the default is the most recently established spawnid.)

Utility Procedures

`lib/utils.exp' defines these utility procedures:

getdirs dir
getdirs dir pattern
Returns a list of all the directories in the single directory dir that match pattern. If you do not specify pattern, getdirs assumes `*'. You may use the common shell wildcard characters in pattern. If no directories match the pattern, then a NULL string is returned.
find dir pattern
Search for files whose names match pattern (using shell wildcard characters for filename expansion). Search subdirectories recursively, starting at dir. The result is the list of files whose names match; if no files match, the result is empty. Filenames in the result include all intervening subdirectory names. If no files match the pattern, then a NULL string is returned.
which binary
Searches the execution path for an executable file binary, like the the BSD which utility. This procedure uses the shell environment variable `PATH'. It returns 0 if the binary is not in the path, or if there is no `PATH' environment variable. If binary is in the path, it returns the full path to binary.
grep filename regexp
grep filename regexp line
Search the file called filename (a fully specified path) for lines that contain a match for regular expression regexp. The result is a list of all the lines that match. If no lines match, the result is an empty string. Specify regexp using the standard regular expression style used by the Unix utility program grep. Use the optional third argument `line' to start lines in the result with the line number in filename. (This argument is simply an option flag; type it just as shown---`line'.)
diff filename filename
Compares the two files and returns a 1 if they match, or a 0 if they don't. If verbose is set, then it'll print the differences to the screen.
slay name
This look in the process tabel for name and send it a unix SIGINT, killing the process.
absolute path
This procedure takes the relative path, and converts it to an absolute path.
psource filename
This sources the file filename, and traps all errors. It also ignores all extraneous output. If there was an error it returns a 1, otherwise it returns a 0.
prune list pattern
Remove elements of the Tcl list list. Elements are fields delimited by spaces. The result is a copy of list, without any elements that match pattern. You can use the common shell wildcard characters to specify pattern.
setenv var val
Sets the variable var to the value val.
unsetenv var
Unsets the environment variable var
getenv var
returns the value of var in the environment if it exists, otherwise it returns NULL.
runtest_file_p runtests testcase
Search runtests for testcase and return 1 if found, 0 if not. runtests is a list of two elements. The first is the pathname of the testsuite expect script running. The second is a copy of what was on the right side of the = if `foo.exp="..."' was specified, or an empty string if no such argument is present. This is used by tools like compilers where each testcase is a file.
prune_system_crud system text
For system system, delete text the host or target operating system might issue that will interfere with pattern matching of program output in text. An example is the message that is printed if a shared library is out of date.

Cross target procedure

`lib/target.exp' defines these utility procedures:

push_target name
This makes the target named name be the current target connection. The value of name is an index into the target_info array and is set in the global config file.
pop_target
This unsets the current target connection.
list_targets
This lists all the supported targets for this architecture.
push_host name
This makes the host named name be the current remote host connection. The value of name is an index into the target_info array and is set in the global config file.
pop_host
This unsets the current host connection. This invokes the compiler as set by CC to compile the file file. The default options for many cross compilation targets are guessed by DejaGnu, and these options can be added to by passing in more parameters as arguments to compile. Optionally, this will also use the value of the cflags field in the target config array. If the host is not the same as the build machines, then then compiler is run on the remote host using execute_anywhere. This produces an archive file. Any parameters passed to archive are used in addition to the default flags. Optionally, this will also use the value of the arflags field in the target config array. If the host is not the same as the build machines, then then archiver is run on the remote host using execute_anywhere. This generates an index for the archive file for systems that aren't POSIX yet. Any parameters passed to ranlib are used in for the flags.
execute_anywhere cmdline
This executes the cmdline on the proper host. This should be used as a replacement for the Tcl command exec as this version utilizes the target config info to execute this command on the build machine or a remote host. All config information for the remote host must be setup to have this command work. If this is a canadian cross, (where we test a cross compiler that runs on a different host then where DejaGnu is running) then a connection is made to the remote host and the command is executed there. It returns either REMOTERROR (for an error) or the output produced when the command was executed. This is used for running the tool to be tested, not a test case.

Debugging Procedures

`lib/debugger.exp' defines these utility procedures:

dumpvars expr
This takes a csh style regular expression (glob rules) and prints the values of the global variable names that match. It is abbreviated as dv
dumplocals expr
This takes a csh style regular expression (glob rules) and prints the values of the local variable names that match. It is abbreviated as dl.
dumprocs expr
This takes a csh style regular expression (glob rules) and prints the body of all procs that match. It is abbreviated as dp
dumpwatch expr
This takes a csh style regular expression (glob rules) and prints all the watchpoints. It is abbreviated as dw.
watchunset var
This breaks program execution when the variable var is unset. It is abbreviated as wu.
watchwrite var
This breaks program execution when the variable var is written. It is abbreviated as ww.
watchread var
This breaks program execution when the variable var is read. It is abbreviated as wr.
watchdel watch
This deletes a the watchpoint for watch. It is abbreviated as wd.
print var
This prints the value of the variable var. It is abbreviated as p.
quit
This makes runtest exit. It is abbreviated as q.
bt
This prints a backtrace of the executed Tcl commands.

Target dependent procedures

Each combination of target and tool requires some target-dependent procedures. The names of these procedures have a common form: the tool name, followed by an underbar `_', and finally a suffix describing the procedure's purpose. For example, a procedure to extract the version from GDB is called `gdb_version'. See section Initialization module, for a discussion of how DejaGnu arranges to find the right procedures for each target.

runtest itself calls only two of these procedures, tool_exit and tool_version; these procedures use no arguments.

The other two procedures, tool_start and tool_load, are only called by the test suites themselves (or by testsuite-specific initialization code); they may take arguments or not, depending on the conventions used within each test suite.

tool_start
Starts a particular tool. For an interactive tool, tool_start starts and initializes the tool, leaving the tool up and running for the test cases; an example is gdb_start, the start function for GDB. For a batch oriented tool, tool_start is optional; the recommended convention is to let tool_start run the tool, leaving the output in a variable called comp_output. Test scripts can then analyze `$comp_output' to determine the test results. An example of this second kind of start function is gcc_start, the start function for GCC. runtest itself does not call tool_start. The initialization module `tool_init.exp' must call tool_start for interactive tools; for batch-oriented tools, each individual test script calls tool_start (or makes other arrangements to run the tool).
tool_load
Loads something into a tool. For an interactive tool, this conditions the tool for a particular test case; for example, gdb_load loads a new executable file into the debugger. For batch oriented tools, tool_load may do nothing--though, for example, the GCC support uses gcc_load to load and run a binary on the target environment. Conventionally, tool_load leaves the output of any program it runs in a variable called `exec_output'. Writing tool_load can be the most complex part of extending DejaGnu to a new tool or a new target, if it requires much communication coding or file downloading. Test scripts call tool_load.
tool_exit
Cleans up (if necessary) before runtest exits. For interactive tools, this usually ends the interactive session. You can also use tool_exit to remove any temporary files left over from the tests. runtest calls tool_exit.
tool_version
Prints the version label and number for tool. This is called by the DejaGnu procedure that prints the final summary report. The output should consist of the full path name used for the tested tool, and its version number. runtest calls tool_version.

The usual convention for return codes from any of these procedures (although it is not required by runtest) is to return 0 if the procedure succeeded, 1 if it failed, and -1 if there was a communication error.

Remote targets supported

The DejaGnu distribution includes support for the following remote targets. You can set the target name and the connect mode in the `site.exp' file (using the Tcl variables `targetname' and `connectmode', respectively), or on the runtest command line (using `--name' and `--connect').

AMD 29000, with UDI protocol
Configure DejaGnu for target `a29k-amd-udi'. (Cygnus configure also recognizes the abbreviation `udi29k'.) Then, to run tests, use the runtest target name to specify whether you want to use a simulator, or a particular hardware board. The particular string to use with `--name' will depend on your UDI setup file, `udi_soc' (if `udi_soc' is not in your working directory, the environment variable `UDICONF' should contain a path to this file). For example, if your UDI setup file includes these lines:
iss   AF_UNIX  *   isstip -r /home/gnu/29k/src/osboot/sim/osboot
mon   AF_UNIX  *   montip -t serial -baud 9600 -com /dev/ttyb
You can use `--name iss' to run tests on the simulator, and `--name mon' to run tests on the 29K hardware. See the manufacturer's manuals for more information on UDI and `udi_soc'. The default connect protocol is `mondfe' with either back end. mondfe is the only shell DejaGnu supports for UDI targets. mondfe is an AMD specific monitor program freely available from AMD. Warning: This target requires GDB version 4.7.2 (or greater). Earlier versions of GDB do not fully support the load command on this target, so DejaGnu has no way to load executable files from the debugger.
Motorola 680x0 boards, a.out or COFF object format
Configure DejaGnu for any remote target matching `m68k-*'. Warning: Most `m68k-*' configurations run all tests only for native testing (when the target is the same as the host). When you specify most of these targets for a cross configuration, you will only be able to use tests that run completely within the host (for example, tests of the binary utilities such as the archiver; or compiler tests that only generate code rather than running it). To run a.out or COFF binaries on a remote M68K, you must configure DejaGnu for a particular target board. `m68k-abug' is an example. (In general for an embedded environment, because it does not have absolute addresses, a.out is not a good choice for output format in any case; most often S-records or Hex-32 are used instead.)
Motorola 68K MVME 135 board running ABug boot monitor
Configure for `m68k-abug-aout' or `m68k-abug-coff' (as a target). This boot monitor can only download S-records; therefore, the DejaGnu tests for this environment require a linker command script to convert either output format to S-records, setting the default addresses for .text, .bss, and .data. With this configuration, the default for `--connect' is `tip'. `tip' is the only communications protocol supported for connecting to `m68k-abug-*' targets. `tip' uses an ASCII downloader (the ~put command) to load S-records into the target board. The `--name' string must be a machine name that tip understands (for example, on some tip implementations it must be an entry from the initialization file for tip; this file is sometimes called `/etc/remote'). See your system documentation for information on how to create new entries in `/etc/remote'. (Some UNIX systems are distributed with at least one default entry with a name resembling `hardwire'; if your system has one, you can edit it, or make a modified copy with a new name.) When you have a working `/etc/remote' entry abugtarget, you should be able to type `tip abugtarget', and get the prompt `135ABUG>' from the board. Use the same abugtarget string with `runtest --name'.
Motorola IDP board running the rom68k boot monitor
This is the same in functionality as the MVME board running the BUG boot monitor. Only the monitor commands and the addresses are different.
VxWorks (Motorola 68K or Intel 960)
Configure DejaGnu for either `m68k-wrs-vxworks' (abbreviated `vxworks68') or `i960-wrs-vxworks' (abbreviated `vxworks960'). Since both targets support IP addressing, specify the network address (for example, a host name from `/etc/hosts') with `--name'. The default connect protocol is `rlogin', but you can use any of `--connect rlogin', `--connect telnet', or `--connect rsh'. Test scripts need no special code to load programs into these targets; since VxWorks supports NFS, all you must do is ensure test programs are on an exported filesystem. When you compile for VxWorks, use the linker `-r' option to make the linker output relocatable--at least if you want to use library routines. Many standard C routines are included in VxWorks; often no additional libraries are needed. See your VxWorks system documentation for additional details.

The files DejaGnu reads

The runtest program used to invoke DejaGnu is a short shell script generated by make during the configuration process. Its main task is to read the main test framework driver, `runtest.exp'.

`runtest.exp', in turn, reads expect code from certain other files, in this order:

  1. Each of the `site.exp' local definition files available. See section Setting runtest defaults, for details.
  2. `lib/utils.exp', a collection of utility procedures. See section DejaGnu procedures, for descriptions of these procedures.
  3. `lib/framework.exp', a file of subroutines meant for runtest itself rather than for general-purpose use in both runtest and test suites.
  4. `debugger.exp', Don Libes' Tcl Debugger. (See A Debugger for Tcl Applications by Don Libes. This paper is distributed with expect in PostScript form as the file `expect/tcl-debug.ps'.)
  5. `lib/remote.exp', a collection of subroutines meant for connecting to remote machines.
  6. `lib/target.exp', a collection of subroutines used for the configuration systems in DejaGnu. These procedures typically manipulate or utilize the configuration system.
  7. An initialization file tool_init.exp. See section Initialization module, for more discussion of init files.

The files DejaGnu writes

runtest always writes two kinds of output files: summary logs and detailed logs. The contents of both of these are determined by your tests.

For troubleshooting, a third kind of output file is useful: use `--debug' to request an output file showing details of what expect is doing internally.

Summary log

runtest always produces a summary output file `tool.sum'. This summary shows the names of all test files run; for each test file, one line of output from each pass command (showing status `PASS' or `XPASS') or fail command (status `FAIL' or `XFAIL'); trailing summary statistics that count passing and failing tests (expected and unexpected); and the full pathname and version number of the tool tested. (All possible outcomes, and all errors, are always reflected in the summary output file, regardless of whether or not you specify `--all'.)

If any of your tests use the procedures unresolved, unsupported, or untested, the summary output also tabulates the corresponding outcomes.

For example, after `runtest --tool binutils', look for a summary log in `binutils.sum'. Normally, runtest writes this file in your current working directory; use the `--outdir' option to select a different directory.

Here is a short sample summary log:

Test Run By rob on Mon May 25 21:40:57 PDT 1992
                === gdb tests ===
Running ./gdb.t00/echo.exp ...
PASS:   Echo test
Running ./gdb.all/help.exp ...
PASS:   help add-symbol-file
PASS:   help aliases
PASS:   help breakpoint "bre" abbreviation
FAIL:   help run "r" abbreviation
Running ./gdb.t10/crossload.exp ...
PASS:   m68k-elf (elf-big) explicit format; loaded
XFAIL:  mips-ecoff (ecoff-bigmips) "ptype v_signed_char" signed
C types 
                === gdb Summary ===
# of expected passes 5
# of expected failures 1
# of unexpected failures 1
/usr/latest/bin/gdb version 4.6.5 -q

Detailed log

runtest also saves a detailed log file `tool.log', showing any output generated by tests as well as the summary output. For example, after `runtest --tool binutils', look for a detailed log in `binutils.log'. Normally, runtest writes this file in your current working directory; use the `--outdir' option to select a different directory.

Here is a brief example showing a detailed log for G++ tests:

Test Run By rob on Mon May 25 21:40:43 PDT 1992

                === g++ tests ===

--- Running ./g++.other/t01-1.exp ---
        PASS:   operate delete

--- Running ./g++.other/t01-2.exp ---
        FAIL:   i960 bug EOF
p0000646.C: In function `int  warn_return_1 ()':
p0000646.C:109: warning: control reaches end of non-void function
p0000646.C: In function `int  warn_return_arg (int)':
p0000646.C:117: warning: control reaches end of non-void function
p0000646.C: In function `int  warn_return_sum (int, int)':
p0000646.C:125: warning: control reaches end of non-void function
p0000646.C: In function `struct foo warn_return_foo ()':
p0000646.C:132: warning: control reaches end of non-void function

--- Running ./g++.other/t01-4.exp ---
        FAIL:   abort
900403_04.C:8: zero width for bit-field `foo'
--- Running ./g++.other/t01-3.exp ---
        FAIL:   segment violation
900519_12.C:9: parse error before `;'
900519_12.C:12: Segmentation violation
/usr/latest/bin/gcc: Internal compiler error: program cc1plus got
fatal signal

                === g++ Summary ===

# of expected passes 1
# of expected failures 3
/usr/ps/bin/g++ version cygnus-2.0.1

Logging expect internal actions

With the `--debug' option, you can request a log file showing the output from expect itself, running in debugging mode. This file (`dbg.log', in the directory where you start runtest) shows each pattern expect considers in analyzing test output.

This file reflects each send command, showing the string sent as input to the tool under test; and each expect command, showing each pattern it compares with the tool output.

The log messages for expect begin with a message of the form

expect: does {tool output} (spawn_id n) match pattern
{expected pattern}?

For every unsuccessful match, expect issues a `no' after this message; if other patterns are specified for the same expect command, they are reflected also, but without the first part of the message (`expect...match pattern').

When expect finds a match, the log for the successful match ends with `yes', followed by a record of the expect variables set to describe a successful match. Here is an excerpt from the debugging log for a GDB test:

send: sent {break gdbme.c:34\n} to spawn id 6
expect: does {} (spawn_id 6) match pattern {Breakpoint.*at.* file
 gdbme.c, line 34.*\(gdb\) $}? no
{.*\(gdb\) $}? no
expect: does {} (spawn_id 0) match pattern {<return>}? no
{\(y or n\) }? no
{buffer_full}? no
{virtual}? no
{memory}? no
{exhausted}? no
{Undefined}? no
{command}? no
break gdbme.c:34
Breakpoint 8 at 0x23d8: file gdbme.c, line 34.
(gdb) expect: does {break gdbme.c:34\r\nBreakpoint 8 at 0x23d8: 
file gdbme.c, line 34.\r\n(gdb) } (spawn_id 6) match pattern
{Breakpoint.*at.* file gdbme.c, line 34.*\(gdb\) $}? yes
expect: set expect_out(0,start) {18}
expect: set expect_out(0,end) {71}
expect: set expect_out(0,string) {Breakpoint 8 at 0x23d8: file
gdbme.c, line 34.\r\n(gdb) }
expect: set expect_out(spawn_id) {6}
expect: set expect_out(buffer) {break gdbme.c:34\r\nBreakpoint 8
at 0x23d8: file gdbme.c, line 34.\r\n(gdb) }
        PASS:   70      0       breakpoint line number in file

This example exhibits three properties of expect and DejaGnu that might be surprising at first glance:


Go to the first, previous, next, last section, table of contents.