tcltest Part 4: Constraints

The Problem

I want to be able to put constraints on tests such as:

  • Platform constraints: For example, on the Unix platform, I don’t want to run Windows-specific tests.
  • Known bugs: I want to temporarily disable valid tests that are failing until the developers fix the code.
  • Crashed tests: I want to temporarily disabled tests that crashed, until I have time to investigate.
  • Custom constraints: I want to define my own constraints, such as tests which are time consuming, tests that only run on a specific host (presumably because that host has some special setup.)

The Solution

In the previous installment, I discussed of ways to include or exclude tests based on the file names or test names. In this installment, I will discuss the use of the -constraints flag to filter tests. For the purpose of this discussions, we are creating tests which does not do anything useful, so we can concentrate on the constraints instead. For the demonstrative purpose, I created a directory called tcltest_part4 with two files: all.tcl and constraints.test.

# all.tcl
package require tcltest
namespace import ::tcltest::*

configure -verbose {skip start}
eval ::tcltest::configure $::argv

runAllTests
# constraints.test

package require tcltest
namespace import ::tcltest::*

test runOnUnix {}           -constraints unix -body {}
test runOnWindows {}        -constraints win -body {}
test runOnMac {}            -constraints mac -body {}
test runNotOnUnix {}        -constraints tempNotUnix -body {}
test knownBug {}            -constraints knownBug -body {}
test customConstraints {}   -constraints timeConsuming -body {}
test multipleConstraints {} -constraints {timeConsuming unix} -body {}
test runOnlyOnMyLaptop {}   -constraints {[info hostname] == "haiv-mac.local"} -body {}
test runOnlyOnMyLaptop {}   -constraints {[info hostname] != "haiv-mac.local"} -body {}

cleanupTests

Explanations:

  • Lines 6-14: Introduces a number of tests whose bodies are empty so we can concentrate on the -constraints flag.
  • Lines 6-8: These are platform-specific tests. Note that I am using the term “platform” instead of “operating system”. The platform is determined by the $tcl_platform(platform) variable. On the Mac OS X, platform is unix, not mac. My guess is the ‘mac’ platform refers to Mac OS 9 or earlier.
  • Line 9: This test should not be run on the unix platform.
  • Line 10: knownBug is a predefined constraint which disables the test. One common use for this constraint is to disable valid tests which are failing because of bugs in the code. Once the developers fix their code and the test passed again, the test engineer can remove this constraint, thus enable them. Many people will point out that this is a way to tamper with test statistics by disabling failed tests. While it is true, this constraint is still useful in some cases such as when the bug’s severity and priority are low, but the cost of fixing them are high, so developers want to postpone fixing it. Meanwhiles, it does not make any sense to see the test keeps failing.
  • Line 11: This line introduces a custom constraint, timeConsuming is a constraint I made up, it can be anything. In practice, I often use this constraint to disable those tests that take too long to run and not required to run on a regular basis.
  • Line 12: A test can have more than one constraint.
  • Lines 13-14: Now this is something different. Instead of tokens identifying test constraints we introduce constraints in the form of an expression. tcltest will evaluate these expressions, and if they are true, the test will run. Line 13 defines a test which will only run on my laptop (perhaps because it meets a certain setup condition). Line 14 is the opposite of line 13.

Now that we discuss the constraints, let see the run output on my Mac OS X laptop:

$ tclsh all.tcl 
Tests running in interp:  /usr/bin/tclsh
Tests located in:  /Users/haiv/src/tcl/tcltest_part4
Tests running in:  /Users/haiv/src/tcl/tcltest_part4
Temporary files stored in /Users/haiv/src/tcl/tcltest_part4
Test files run in separate interpreters
Running tests that match:  *
Skipping test files that match:  l.*.test
Only running test files that match:  *.test
Tests began at Sun Apr 03 09:14:12 PDT 2011
constraints.test
---- runOnUnix start
++++ runOnWindows SKIPPED: win
++++ runOnMac SKIPPED: mac
++++ runNotOnUnix SKIPPED: tempNotUnix
++++ knownBug SKIPPED: knownBug
++++ customConstraints SKIPPED: timeConsuming
++++ multipleConstraints SKIPPED: timeConsuming
---- runOnlyOnMyLaptop start
++++ runOnlyOnMyLaptop SKIPPED: [info hostname] != "haiv-mac.local"

Tests ended at Sun Apr 03 09:14:12 PDT 2011
all.tcl:	Total	9	Passed	2	Skipped	7	Failed	0
Sourced 1 Test Files.
Number of tests skipped for each constraint:
	1	[info hostname] != "haiv-mac.local"
	1	knownBug
	1	mac
	1	tempNotUnix
	2	timeConsuming
	1	win

At the end of the output, tcltest lists 7 skipped tests and their reasons such as knownBug, tempNotUnix, or win (platform). What if I want to run the time-consuming tests? I can enable these timeConsuming tests by adding the -constraints flag to the command line:

$ tclsh all.tcl -constraints timeConsuming
... (irrelevant output snipped)
---- runOnUnix start
++++ runOnWindows SKIPPED: win
++++ runOnMac SKIPPED: mac
++++ runNotOnUnix SKIPPED: tempNotUnix
++++ knownBug SKIPPED: knownBug
---- customConstraints start
---- multipleConstraints start
---- runOnlyOnMyLaptop start
++++ runOnlyOnMyLaptop SKIPPED: [info hostname] != "haiv-mac.local"

Tests ended at Sun Apr 03 09:19:26 PDT 2011
all.tcl:	Total	9	Passed	4	Skipped	5	Failed	0
Sourced 1 Test Files.
Number of tests skipped for each constraint:
	1	[info hostname] != "haiv-mac.local"
	1	knownBug
	1	mac
	1	tempNotUnix
	1	win

Notice this time around, the two timeConsuming tests are included in the run, reducing number of skipped tests from 7 down to 5. However, there are times I want to run only these two tests. I can accomplish this requirement by introducing the –limitconstraints flag:

$ tclsh all.tcl -constraints timeConsuming -limitconstraints true

I can also enable more than one constraints by grouping the constraints within the single- or double-quote characters:

$ tclsh all.tcl -constraints "timeConsuming tempNotUnix" -limitconstraints true

Predefined Constraints

The documentation for tcltest 2.2.5 lists the following predefined constraints. Please consult this manual for their meanings and usage.

  • Platforms: unix, win, mac, unixOrWin, macOrWin, macOrUnix, tempNotWin, tempNotMac
  • Operating systems: nt 95 98
  • Disabling crashed tests: unixCrash, winCrash, macCrash
  • Other constraints: emptyTest, knownBug, nonPortable, userInteraction, interactive, nonBlockFiles, asyncPipeClose, unixExecs, hasIsoLocale, root, notRoot, eformat, stdio, singleTestInterp

Custom Constraints

Below is a list of suggested custom constraints.

  • Time consuming tests: timeConsuming
  • Run only on Mac OS X: {$tcl_platform(os) == “Darwin”}
  • Never on Sunday: {[clock format [clock seconds] -format “%a”] != “Sun”}
  • On the first day of each month: {clock format [clock seconds] -format “%d”] == “01”}
  • Only for i386 architecture: {$tcl_platform(machine) == “i386”}
  • For a specific user: {$tcl_platform(user) == “test1user”}
  • for a specific Tcl version or later: {[info tclversion] >= “8.5”}
  • Only when a specific function is found: {[info procs myFunction] == “myFunction”}

For constraints that are very complex, we can write a function, which returns true (1) or false (0) and pass that into the test constraints:

proc myComplexConstraint {} {
	# code ...
}

...

test myTest {} -constraints [myComplexConstraint] -body { ... }

What’s Next?

So far, we only test functions that return values such as sum, square. Sometimes, we need to test those functions which writes output to the standard output device. The next installment will discuss this feature of tcltest.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s