tcltest Part 3 – Include and Exclude Tests

Posted on April 1, 2011

0


The Problem

Sometimes I want to run just a subset of the tests. Some other times, I want to skip some specific tests.

The Solution

tcltest came with a configure command which customizes many aspects of the test invocation. Among those are the ability to include or exclude based on file names or test names. Below is a list of useful configure commands and their shortcuts:

  • configure -file patternList
  • configure -notfile patternList
  • configure -match patternList
  • configure -skip patternList
  • matchFiles patternList = shortcut for configure -file
  • skipFiles patternList = shortcut for configure -notfile
  • match patternList = shortcut for configure -match
  • skip patternList = shortcut for configure -skip

Dynamically Configure Test Runs

Before I move all to explain the configure command, let’s modify the all.tcl main script to allow us the ability to configure the test runs using command line. Here is the revived all.tcl:

package require tcltest
namespace import ::tcltest::*

if {$argc != 0} {
    foreach {action arg} $::argv {
        if {[string match -* $action]} {
            configure $action $arg
        } else {
            $action $arg
        }
    }
}

runAllTests
Explanation
  • Lines 4-12: The script assumes the command-line parameters are for configuration.
  • Line 5: I assume that command-line parameters come in pairs of action and argument.
  • Lines 6-7: If the action starts with a dash, I assume it is part of the configure command, so I invoke the configure command accordingly.
  • Lines 8-9: If the action does not start with a dash, it must be a shortcut command, invoke that action accordingly.

Now that I updated all.tcl, I am ready to have some fun with test filtering.

Run Only Selective Files

In this installment, I assume the same settings as in part 2, with the exception of all.tcl, which was modified as discussed above. That means our tests directory has two test files: square.test and sum.test. To run tests in one or more files:

tclsh all.tcl -file sum.test # Run only tests in sum.test
tclsh all.tcl matchFiles sum.test # Same as above

The above command will only run tests in sum.test and ignore other test files. This filter is especially good when you are developing test and want to run only those tests you are developing. You can choose more than one files by using wildcards or by listing several patterns at once:

# Run all files with names start with s
tclsh all.tcl -file 's*.test' 

# List more than one files
tclsh all.tcl -file 'su*.test sq*.test'

Note that the configure command and its shortcuts will override previous settings. The following command will only run tests in sum.test, but not in square.test:

tclsh all.tcl -file square.test -file sum.test

Skipping Files

Here are some examples of skipping files:

# Will run all, but sum.test
tclsh all.tcl -notfile sum.test 

# Same as above
tclsh all.tcl skipFiles sum.test 

# Skips all files with names start with a or b
tclsh all.tcl -notfile 'a*.test b*.test'

Run Based on Test Names

While previous commands filter based on file names, in this section and the next, I will filter based on the name of the tests instead. These filters work regardless of which files the tests reside in.

# Run only tests whose names start with 'square_'
tclsh all.tcl -match 'square_*' 

# Same as above
tclsh all.tcl match 'square_*' 

# Only tests with Negative or Zero in their names
tclsh all.tcl -match '*Negative* *Zero*' 

# Tests names that end with Zero
tclsh all.tcl -match '*Zero'

Skip Tests Based on Test Names

While the previous section “filter-in” based on test names, this section will “filter-out” instead.

# Skip tests that start with sum
tclsh all.tcl -skip 'sum*' 

# Skip tests that start with sum
tclsh all.tcl skip 'sum*' 

# Skip tests that contains either Positive or Negative in their names
tclsh all.tcl -skip '*Positve* *Negative*'

Combine the Filters

Finally, I can mix and match filters:

# Run tests in the file square.test, but skipping over those tests
# whose names contain 'Zero'
tclsh all.tcl -file square.test -skip '*Zero*'

# Skip the file sum.test, also skip tests with 'Negative' in their names
tclsh all.tcl -skip '*Negative*' -notfile sum.test

What’s Next?

So far, I only touch a little on test filtering. In m next installment, I will discuss test constrains. Better yet, please subscribe to my blog to make sure you won’t miss my posts. For reference, please review my previous posts:

About these ads
Posted in: Programming, Tcl