tcltest Part 9: Provides Exit Code

The Problem

One of the complains against tcltest is the fact that it does not return an exit code to communicate test pass/fail status.

Consider a simple scenario. In a directory, we have two files: all.tcl, which drives all the tests, and test1.test, which contains the actual tests. The contents of all.tcl is:

package require tcltest
tcltest::runAllTests

And the contents of test1.test is:

package require tcltest
namespace import ::tcltest::*

test fail_me {} -body { set foo 1 } -result 0
test pass_me {} -body { set foo 1 } -result 1

cleanupTests

In the scenario above, one of the tests will fail, but all.tcl will always return with exit code 0. It would be more useful if it returns 0 when all tests passed and 1 when not all tests passed.

The Solution

We looked into the tcltest source code and found an array, tcltest::numTests, which looks what we need. So, we modified all.tcl to see if we can use it:

package require tcltest
tcltest::runAllTests
parray tcltest::numTests

The result is disappointing:

(irrelevant output omitted)
tcltest::numTests(Failed)  = 0
tcltest::numTests(Passed)  = 0
tcltest::numTests(Skipped) = 0
tcltest::numTests(Total)   = 0

With one failed tests, we thought tcltest::numTests(Failed) should be 1, but it was not. we went back to the source code and dug a little deeper and found tcltest::cleanupTests resets these numbers after it finishes reporting.

Just when we was about to give up, we thought of asking Mr. Google for help. Sure enough, tcltest provides a hook into its tcltest::cleanupTests, which gives us access to the statistics variables before it resets them. Here is the final all.tcl:

package require tcltest

# Hook to determine if any of the tests failed. Then we can exit with
# proper exit code: 0=all passed, 1=one or more failed
proc tcltest::cleanupTestsHook {} {
    variable numTests
    set ::exitCode [expr {$numTests(Failed) > 0}]
}

tcltest::runAllTests
exit $exitCode

Discussion

The trick is to write a hook, tcltest::cleanupTestsHook. Before resetting the test statistics, tcltest::cleanupTests calls the hook function, which gives us the opportunity to access these statistics, and use them to determine the exit code.

Now that all.tcl returns the proper exit code, we can use it to determine pass/fail. Here is an example using the bash shell in Unix-like systems:

#!/bin/bash
if tclsh all.tcl; then
    echo Tests passed
else
    echo One or more tests failed
fi

For Windows environment:

tclsh all.tcl
if errorlevel 1 echo One or more tests failed

Conclusion

We wish tcltest automatically returns the correct exit code, but it does not. For now, we will stick with this solution and hope that whoever working on tcltest won’t make our code obsolete.

4 thoughts on “tcltest Part 9: Provides Exit Code

  1. Pawel

    I have one more question, what if there will be an “error” inside the script? Because this solution is “catching” only “failing” tests, not errored scripts

  2. Hai Post author

    I did some poking around and it looks like tcltest treats error within the script as failed case. I am still thinking of a way to differentiate between the two. On the other hand, you might want to view part 6 if you want to test for error condition.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s