Monthly Archives: September 2013

tcltest Part 9: Provides Exit Code

The Problem

One of the complains against tcltest is the fact that it does not return an exit code to communicate test pass/fail status.

Consider a simple scenario. In a directory, we have two files: all.tcl, which drives all the tests, and test1.test, which contains the actual tests. The contents of all.tcl is:

package require tcltest

And the contents of test1.test is:

package require tcltest
namespace import ::tcltest::*

test fail_me {} -body { set foo 1 } -result 0
test pass_me {} -body { set foo 1 } -result 1


In the scenario above, one of the tests will fail, but all.tcl will always return with exit code 0. It would be more useful if it returns 0 when all tests passed and 1 when not all tests passed.

The Solution

We looked into the tcltest source code and found an array, tcltest::numTests, which looks what we need. So, we modified all.tcl to see if we can use it:

package require tcltest
parray tcltest::numTests

The result is disappointing:

(irrelevant output omitted)
tcltest::numTests(Failed)  = 0
tcltest::numTests(Passed)  = 0
tcltest::numTests(Skipped) = 0
tcltest::numTests(Total)   = 0

With one failed tests, we thought tcltest::numTests(Failed) should be 1, but it was not. we went back to the source code and dug a little deeper and found tcltest::cleanupTests resets these numbers after it finishes reporting.

Just when we was about to give up, we thought of asking Mr. Google for help. Sure enough, tcltest provides a hook into its tcltest::cleanupTests, which gives us access to the statistics variables before it resets them. Here is the final all.tcl:

package require tcltest

# Hook to determine if any of the tests failed. Then we can exit with
# proper exit code: 0=all passed, 1=one or more failed
proc tcltest::cleanupTestsHook {} {
    variable numTests
    set ::exitCode [expr {$numTests(Failed) > 0}]

exit $exitCode


The trick is to write a hook, tcltest::cleanupTestsHook. Before resetting the test statistics, tcltest::cleanupTests calls the hook function, which gives us the opportunity to access these statistics, and use them to determine the exit code.

Now that all.tcl returns the proper exit code, we can use it to determine pass/fail. Here is an example using the bash shell in Unix-like systems:

if tclsh all.tcl; then
    echo Tests passed
    echo One or more tests failed

For Windows environment:

tclsh all.tcl
if errorlevel 1 echo One or more tests failed


We wish tcltest automatically returns the correct exit code, but it does not. For now, we will stick with this solution and hope that whoever working on tcltest won’t make our code obsolete.