» TETware GUI
» Download TET
Information» Datasheet (pdf)
» GUI Help Pages
How to to write a TET C API Test Suite
This tutorial describes how to write a test suite using the TET C language API binding. The source code for the test suite can be found in Appendix A.
This tutorial is designed to illustrate how a test suite can be structured under TET, as well as how individual test cases and their test purposes relate to each other and to the API. The test suite has been deliberately kept simple and realistic. For example, one test purpose compares the returned error code against an expected error code of a failed system call, while another test purpose in the same test case checks the successful execution of the system call.
Small segments of code from the test suite appear in the following sections to help illustrate specific points. Refer to the appropriate section in Appendix A to see the code in its entirety.
Test suites reside in subdirectories of $TET_ROOT (or alternately $TET_SUITE_ROOT, which is an Extended TET feature). The name of the subdirectory and the test suite are the same. The following figure shows the component files of the sample test suite, called C-API:
The make-up of this test suite is similar to the demo test suite which comes with the TET: an install script and cleantool in the bin directory; configuration files for test build, execution, and cleanup; a control file, tet_scen; a result codes file, tet_code; several test cases in a directory structure under the directory ts; and a results directory.
The control file, tet_scen, lists the components of the test suite; and its contents determine the scenarios that can be used in running the test suite. The following figure shows the contents of the control file, tet_scen, for the C-API test suite.
The control file lists five scenarios for the test suite: all (required), chmod, fileno, stat, and uname. Since the test suite is composed of four test cases, one for the chmod system call, one for the fileno system call, one for the stat system call, and one for the uname system call, the control file has been written to allow each test case to be handled as a separate scenario, or for the whole test suite to be run at once with the all scenario.
The lines enclosed in double quotation marks are optional information lines that get passed into the journal file. The lines that begin with a slash or stroke character (/) name the executable test cases associated with each scenario. Note that even though these lines begin with a slash character, their location is relative to the local directory (the root directory for the test suite). In this instance, the test cases are in a subdirectory named ts.
The cleantool is used to remove unwanted files after the build of each test case. It is invoked in the source directory of the test case. In this case it is set to exec make clean to remove unwanted object files as defined in each makefile.
Since most test suites lend themselves to lots of code redundancy, making an effort to group together common functions and variables can greatly simplify the writing and debugging of a test suite. With the C-API test suite, which is very small, no common functions and variables other than the standard ones in tetapi.h were created.
One additional result code was invented, however, which would normally be defined in a test suite specific header file. But because it is only used within one test case in this very small test suite, it is instead defined within uname-tc.c as follows:
Every test case requires some minimum initialization of functions and variables. The fileno-tc test case provides a good illustration of how this initialization can be handled.
After the #include statements, several functions are declared. TET provides the option of naming both a startup and cleanup function. The named startup function will be called before the first test purpose is executed; and the cleanup function will be called after all test purposes have been executed. In this test case, only the cleanup function is named. The cleanup function cleanup() removes files created during the course of the test case.
The stat-tc test case includes a more substantial cleanup function, as well as a startup function. It requires that a file be created before the first test purpose, so this is handled by the startup function; this same file, as well as another file and a directory created during the tests, is then removed in the cleanup function. See Appendix A for a complete code listing of the stat-tc test case.
The fileno-tc test case includes four test purposes, contained in the functions: tp1, tp2, tp3, and tp4. First the functions are declared (including an extra function which is a child process of tp4), as shown above. Then they are listed in the tet_testlist array with the invocable component to which they belong. In this case, each test purpose can be executed individually, so they are assigned to separate invocable components. If, say, tp2 depended on prior execution of tp1, then they would be assigned the same IC number. After the array is set, any test case wide declarations are made. This commonly includes a buffer to use for constructing information lines to be output with tet_infoline().
1.5 Controlling and Recording Test Case Execution Results
Identifying and executing highly specific tests is central to any test case. Each test purpose in a test case typically targets one specific test that is loosely or strongly related to the other test purposes contained in the test case. The central purpose of each of these test purposes is to relay information about the execution of the test for the tester to examine later. This relaying of information can take the form of informational messages describing the test being executed, fatal or non-fatal errors that were encountered, and specific test execution results, such as pass or fail.
The chmod-tc test case contains three test purposes:
Functions tp1 and tp2 are shown here and are described below.
The comments for the code should clarify what is happening on each line. However, it is important to note that a lot of useful diagnostics have been written right into the tests. If any of the system calls fail, whether it is the one being specifically tested or one that the test relies on, that failure will be reported. Also, the tests begin the same, with a message about the test's purpose; and they end the same, with a pass/fail result being reported.
This sort of consistency yields two important benefits:
Some test cases may require user verification of information generated by a test case. An example of this can be found in the uname-tc test case when system specific information is being reported.
Since the information from uname will be different on every machine, the output needs to be reported and then verified. Here the information is simply being printed out for the tester to see and check, but no attempt has been made to interact with the tester to receive verification of the information and then use that verification to set the pass/fail result. Instead, a result code of INSPECT has been used.
Some test purposes require the creation of a child process or execution of a subprogram. The Toolkit provides three interfaces to facilitate this:
An example of their use can be found in test purpose tp4 of the fileno test case:
All the testing is done in the child, so the function tp4() simply calls tet_fork() and ignores the return value. If it needed to do any processing after the call to tet_fork(), it should check that the return value was one of the expected child exit codes before continuing. The arguments to tet_fork() are: a function to be executed in the child; a function to be executed in the parent (in this case no parent processing is required, so a null function pointer TET_NULLFP, defined in tet_api.h, is used); a timeout period in seconds; and, a bitwise OR of the valid child exit codes (in this case the only valid exit code is zero).
The file fileno-t4.c contains the definition of tet_main():
Since test cases often change and/or create data, it is important to cleanup this data before exiting the test case. As explained earlier, one way to do this is to specify a cleanup function with TET's tet_cleanup utility. The cleanup function named in the stat-tc test case provides a good example.
The cleanup function is called when all the test purposes have finished executing. As shown, it simply removes the files and directory that were created during the test.