Monthly Archives: March 2011

tcltest Part 2 – Multi-Module Test Suite

The Problem

Now that I created my first tcltest module, I want to break up my tests into several modules. I also want to separate my tests from my source code.

The Solution

In this tutorial, I will pick up from the last time. Recall that in the previous post, I created the first test module, but both the source code (sum.tcl) and the tests are in the same directory. While this arrangement is fine for small projects, as they grow, I will want to separate the source code from the test code to improve code management.

From the previous tutorial, the directory contains 3 files:

  • sum.tcl – The software under test (SUT)
  • all.tcl – The “main” test script
  • sum.test – The test module which tests function sum

To separate the source code from the tests, we start by creating two sub directories: src and tests and move the files to these directories according to their roles:

mkdir src tests
mv sum.tcl src
mv all.tcl sum.test tests

Next, we go into the tests directory and attempt to run the tests again:

cd tests
tclsh all.tcl

This time, we ran into a error because the file sum.tcl is no longer in the same directory with the test module:

Test file error: couldn't read file "sum.tcl": no such file or directory
    while executing
"source sum.tcl"
    (file "/home/haiv/src/tcl/tcltest_part2/tests/sum.test" line 7)

To fix it, modify line 7 of sum.test so it reads:

source ../src/sum.tcl

Now, run the tests again and everything should work as before. At this point, we successfully separate the tests from the source code. From now on, our test modules will go into the tests directory and the source code will go into the src directory.

Next, we are adding a new function: square, which returns the square of a number. The new function resides in src/square.tcl:

proc square {x} {
    expr {$x * $x} 
}

Our next step is to add the tests for function square. Again, for the sake of brevity, i will ommit the test descriptions, but encourage you to add them to enhance test readablity.

# square.test

package require tcltest
namespace import ::tcltest::*

# Software under test
source ../src/square.tcl

test square_ofZeroExpectsZero {
    Test: [square 0] == 0
} -body {
    square 0
} -result 0

test square_ofNegativeExpectsPositive {} -body {
    square -9
} -result 81

test square_ofPositiveExpectsPositive {} -body {
    square 19
} -result 361

cleanupTests

Notes: This test module is very similar to the sum.test module so it is tempting to copy and paste. Please keep in mind that when you copy and paste, be sure to check the code. Here is the tests output:

Tests running in interp:  /usr/bin/tclsh
Tests located in:  /home/haiv/src/tcl/tcltest_part2/tests
Tests running in:  /home/haiv/src/tcl/tcltest_part2/tests
Temporary files stored in /home/haiv/src/tcl/tcltest_part2/tests
Test files run in separate interpreters
Running tests that match:  *
Skipping test files that match:  l.*.test
Only running test files that match:  *.test
Tests began at Mon Mar 28 07:13:18 PM PDT 2011
square.test
sum.test

Tests ended at Mon Mar 28 07:13:18 PM PDT 2011
all.tcl:	Total	6	Passed	6	Skipped	0	Failed	0
Sourced 2 Test Files.

Note that the output now shows 2 files: square.test and sum.test. Also note that square.test precedes sum.test: tcltest run them in alphabetical order of file names. However, do not assume that order: in general, unit test cases should be independent of each other.

Summary

This tutorial shows you how to separate the tests from the source code for better code management. It also demonstrate a multi-module test suite. In the next installment, I am going to go into test filters.

Getting Started with tcltest

The Problem

I want to use tcltest to unit test my code, but don’t know how to start.

The Solution

This tutorial will guide you through a step-by-step process on how to get started with tcltest

Installation

Installation is in in the scope of this tutorial, I assume that you have Tcl 8.4 or 8.5 installed. Chances are, you have tcltest package installed as well. To check, type the following command to your terminal:

tclsh
package require tcltest
exit

If you don’t see any error messages, then you are set. Otherwise, please install Tcl 8.4 or later, and tcltest.

Your Software under Test

In this tutorial, I assume the software under test (SUT) is a set of functions in a file called sum.tcl:

proc sum {a b} {
    expr {$a + $b}
}

Create a “main” Test Script

The first step is to create a main script. This script will drive all the tests in the current directory. I name this file all.tcl:

package require tcltest
namespace import ::tcltest::*
runAllTests

Next, test this main script:

tclsh all.tcl

The output should look similar to this:

Tests running in interp:  /usr/bin/tclsh
Tests located in:  /Users/haiv/src/tcl/tcltest_getting_started
Tests running in:  /Users/haiv/src/tcl/tcltest_getting_started
Temporary files stored in /Users/haiv/src/tcl/tcltest_getting_started
Test files run in separate interpreters
Running tests that match:  *
Skipping test files that match:  l.*.test
Only running test files that match:  *.test
Tests began at Mon Mar 28 09:37:40 PDT 2011
Error:  No test files remain after applying your match and skip patterns!

Tests ended at Mon Mar 28 09:37:40 PDT 2011
all.tcl:	Total	0	Passed	0	Skipped	0	Failed	0
Sourced 0 Test Files.

Note the next-to-last line of the output: it said we have 0 tests, so writing a test is the next task.

Writing the First Test Module

In this tutorial, I will write one test file (module) per function. However, you can organize your tests in any way you want. Let’s create the first test module to test the sum function and call it sum.test. Note that by default, tcltest will look for files with .test extension and assume them to be a test module. The contents of sum.test look like this:

# sum.test

package require tcltest
namespace import ::tcltest::*

# Software under test
source sum.tcl

test sum_addTwoZerosExpectZero {
    Test: [sum 0 0] == 0
} -body {
    sum 0 0
} -result 0

test sum_addTwoPositiveNumbers {} -body {
    sum 4 9
} -result 13

test sum_addPositiveToNegative {} -body {
    sum -95 72
} -result -23

cleanupTests

Explanation:

  • Line 3,4: Use the tcltest package. The namespace import line allows you to skip the name space. For example, you can use test instead of tcltest::test
  • Line 7: Includes the software under test into your test module.
  • Lines 9-13: Our first test case starts with the ‘test’ command. The first argument is the name of the test, followed by the test description (line 10), the test body (line 12). We will discuss more about test case in the next section.
  • Lines 15-21: Defines two more test cases. For the sake of brevity, I ommit the test descriptions
  • Line 23: This line is required to tally all the test results for this module.

About a Test Case

In unit testing, each test case should be small, concise, and only test 1 aspect of the software. That way, when a test failed, we know exactly what is failed and hopefully, why it faied and how to fix it. The test description follows the test name. I recommend not to leave the description empty, unless you think the test name is descriptive enough.

Running the Tests

Now that we have our first test case written, it is time to run it. From the terminal issue the following command:

tclsh all.tcl

Output:

Tests running in interp:  /usr/bin/tclsh
Tests located in:  /Users/haiv/src/tcl/tcltest_getting_started
Tests running in:  /Users/haiv/src/tcl/tcltest_getting_started
Temporary files stored in /Users/haiv/src/tcl/tcltest_getting_started
Test files run in separate interpreters
Running tests that match:  *
Skipping test files that match:  l.*.test
Only running test files that match:  *.test
Tests began at Mon Mar 28 15:10:03 PDT 2011
sum.test

Tests ended at Mon Mar 28 15:10:03 PDT 2011
all.tcl:	Total	3	Passed	3	Skipped	0	Failed	0
Sourced 1 Test Files.

Note that now the sumary line shows 3 tests and it was passing. You are now free to add more test cases to the sum.test module or create a new module. That concludes my tutorial. In the next installment, I am going to talk about organizing the tests and adding more test modules to the mix.

How to Fix tclsh Line Editing Problem

The problem

One sentence: command line editing sucks in Linux and Mac. On Windows, you can use the up and down arrows, among other things, to recall last lines and edit. On Linux and Mac OS X, these keys don’t work, making line editing difficult.

The Solutions

In my previous article, I mentioned using tkcon to get around this limitation. Today, I am going to talk about a different mechanism: tclreadline. This is a Tcl package which makes use of GNU readline. To install this package on Debian-family of Linux, which includes Debian, Ubuntu, Kbuntu, Xubuntu, and Linuxmint, issue the following command line from the terminal:

sudo apt-get install tclreadline

Next, follow the instruction in the usage section from the tclreadline page. Now, every time you start tclsh, you can use up/down errors, tab-completion, and many other features that GNU readline provides.

Tcl – How to Locate Include File to Source

The Problem

In your Tcl script, you want to source (include) another script that resides the in the same directory. However, all is fine if your scripts are in the current directory. If you execute the main script from a different directory, Tcl complains that the included script is not found.

To demonstrate this problem, create a directory called demo from your home directory and cd to it:

mkdir ~/demo
cd ~/demo

In this directory, create two files:

# library.tcl
proc greet {} {
    puts "Hello, world"
}
# main.tcl
source library.tcl
greet

Now, execute the main script:

tclsh main.tcl

The script works as expected and produces “Hello, world”. However, if you cd to a different directory, say the home directory, and execute that same main script, the result is different:

$ tclsh ~/demo/main.tcl 
couldn't read file "library.tcl": no such file or directory
    while executing
"source library.tcl"
    (file "/home/haiv/demo/main.tcl" line 2)

The problem is the source statement in line 2 of main.tcl looks for the file library.tcl in the current directory, which explains why the it works the first time, but fails the second. The next section will discuss the solution to make it work in both cases.

The Solution

In order for main.tcl to find library.tcl, it needs to look in the directory where main.tcl is and here is the trick:

# main.tcl
source [file dirname [info script]]/library.tcl
greet

The info script command returns the location of the currect script (main.tcl). Next, the file dirname command extracts the directory part. Finally, we append /library.tcl to give the full path to library.tcl. So now, it does not matter where you execute the script from, the main script will always find its library to include

The Revised Solution

I would like to thank michaelhinds for this suggestion:

# main.tcl
source [file join [file dirname [info script]] library.tcl]
greet

This time, we use the file join command to ensure cross-platform compatibility instead of hardcoding using the forward slash (/).

Squeeze Multiple Blank Lines within vim

The Problem

I want to squeeze multiple blank lines into just one while editing in vim. I often receive source code which the author used multiple blank lines to separate functions. I don’t mind if functions or code block are one or two lines apart, but I have seen sources where functions are four or more blank lines from each other. This is annoying and I want to squeeze many of them into one.

The Solutions

The obvious solution is to use vim’s search-and-replace feature, as discussed in this post. However, my regular expression skill is very basic and I tend to forget the syntax when I need it. Hence, I devised my own solution, one which I have better luck memorizing.

My solution involes the cat command:

:%!cat -s

This short little command applies ‘cat -s’ on my entire file contents, which squeezes multiple blank lines down to one. Note that this solution will only work on Unix-like systems such as BSD, Linux, and Mac OS X. It does not work on Windows. This is one of the shortcoming of this method compare to the previous one.

Building TclCurl on Mac OS X Snow Leopard

The Problem

I want to install TclCurl on my MacBook Pro running Snow Leopard, but the only installation option that I can find was to build it via MacPorts. However, I don’t want MacPorts since it will be pulling in too many components. All I want is to build TclCurl and install it. I did download the source code and attempted to build it, but ran into compiler’s error. This post will details the steps I made in order to overcome this problem.

The Procedure

To successfully build and install TclCurl, follow these steps:

  1. Download the source and unpack
  2. Configure and build
  3. Modify the source
  4. Build again and install

Download the Source and Unpack

After searching for the keyword “TclCurl”, I arrived at the TclCurl download page. Here, I downloaded the only version available at the time, version 7.19.6 under the “Unix and Linux” heading.

Next, I opened a terminal, navigate to the directory containing the source and issue the following command:

tar xf TclCurl-7.19.6.tar.gz
cd TclCurl-7.19.6

If you downloaded a different version, replace the version numbers in those commands accordingly.

Configure and build

Now that I was in the TclCurl source directory, I could build and install it:

./configure
make

However, make did not completed successfully due to a compiler error:

gcc -DPACKAGE_NAME=\"TclCurl\" -DPACKAGE_TARNAME=\"tclcurl\" -DPACKAGE_VERSION=\"7.19.6\" -DPACKAGE_STRING=\"TclCurl\ 7.19.6\" -DPACKAGE_BUGREPORT=\"\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DNO_VALUES_H=1 -DHAVE_LIMITS_H=1 -DHAVE_SYS_PARAM_H=1 -DUSE_THREAD_ALLOC=1 -D_REENTRANT=1 -D_THREAD_SAFE=1 -DTCL_THREADS=1 -DMODULE_SCOPE=extern\ __attribute__\(\(__visibility__\(\"hidden\"\)\)\) -DTCL_WIDE_INT_IS_LONG=1 -DUSE_TCL_STUBS=1  -I/usr/include -I"/System/Library/Frameworks/Tcl.framework/Headers"    -pipe  -Os -Wall -fno-common   -c `echo ./generic/tclcurl.c` -o tclcurl.o
./generic/tclcurl.c: In function ‘curlSetOpts’:
./generic/tclcurl.c:455: error: ‘ulong’ undeclared (first use in this function)
./generic/tclcurl.c:455: error: (Each undeclared identifier is reported only once
./generic/tclcurl.c:455: error: for each function it appears in.)
./generic/tclcurl.c:455: error: expected ‘;’ before ‘protocolMask’
./generic/tclcurl.c:1780: error: ‘protocolMask’ undeclared (first use in this function)

...

Did you notice the line that said ‘ulong’ undeclared? This is where we need to fix for the Mac OS X platform.

Modify the Source

After reviewing the code, I fired up my favorite text editor and added the following block near the beginning of the file generic/tclcurl.h and saved:

#ifdef __APPLE__
#define ulong unsigned long
#endif

Build again and Install

After making the necessary modification. I am ready to build again and install:

make
sudo make install

Finally, I can test to see if the TclCurl package has been installed correctly. I ran tclsh and issued the following command:

package require TclCurl

Voila! no error. That means the package is installed and ready for use. Now I can resume my work, which needs TclCurl.

Upgrade My Mac’s Hard Drive

My 4.5 year-old MacBook is getting slower every day due to its age against the new OS and applications. To give it a new lease in life, I replaced its 500GB hard drive with a 120GB SSD from OWC. This post discusses the upgrade process.

Before the Upgrade

Unwind Applications

This step disconnects the applications from their services. For example, I de-authorized my iTunes account after one last sync with my iPhone. I also disconnected from Dropbox, quit Mail. I quitted all applications which connects to the internet to ensure their data not to change during the back up. I also took note of the Dropbox’s user name and password, along with the password to my 1Password data file. These information are crucial for bringing my system up after the upgrade.

Prepare Applications List

This step is crucial. I made a list of applications I installed on my MacBook. I also hunt down all the license keys to allow me to reinstall later. Since I store my license keys in 1Password, this step is relatively simple. I then prioritize my applications list: priority means I must have them. 1Password and Backblaze back up falls into this category. Priority 2 means I should have them for my every day tasks: Aperture, Path Finder, and Alfred are examples in this category. Applications that do not fall into these two categories will be installed only when absolutely needed.

Back up Data

Next, I disconnected my internet connection to ensure no data change during the back up. I then created a bootable clone of my hard drive using SuperDuper. This is the last snapshot of my old hard drive. This step also means that after upgrade, I will have two copies of my old hard drive. I will keep one of those copies in case I ever need any old data from it.

Upgrade and Post Upgrade

I will not discuss the actual replacement of the drives as there are several how-to videos showing this procedure–just search for them.

Install the Operating System and Updates

After installing the new drive, I installed the operating system from the disc. Then connect to the internet and run software update. This step might spans several minutes.

Restore Dropbox and 1Password

Since I stored my 1Password’s data in Dropbox, I connected to the internet, downloaded Dropbox, installed it, copied the Dropbox data from the old drive to the new, and brought Dropbox online. Next, I downloaded and installed 1Password. Amazingly, the first time I launched 1Password, the application found its data from the Dropbox folder and asked me if I want to use it. After saying yes, I got all my 1Password back as if the upgrade never occurred. I have the 1Password team to thank for that.

Restore Other Applications and Data

Most applications store their data in ~/Library/Application Support, so after re-installing the application and before the first launch, I copied the data from the old drive over to the new drive at the same location and applications will pick up where they left off. Many people suggested to use Apple’s application migrations, but I don’t want to use it because I need the absolute control of which application to move. Besides, I am more comfortable with this level of details. If you are not familiar with the file structure of Mac OS X, you should use application migration.

Lessons Learned

  • Backup, backup, backup. My drive were backed up by three means: The Backblaze online service, SuperDuper and Time Machine. Over the years, I had lost many valuable family photos and videos due to the lack of backing up. I made that mistake twice (yes, its dumb.) With the new SSD, I keep all three back up methods. The Backblaze backup is especially useful because it is offsite.
  • Preparation ensures smooth transition. By taking notes of the essential services’s user names and passwords, I was able to pick up quickly. By learning the data location of my applications, I avoided headaches of not having my data migrated.
  • Only installed application when absolutely necessary. This time around, I not install applications just because it is cool. This will save me from bloating my hard drive and operating system, slowing down my already-aging MacBook.
  • 1Password is a life-saver, especially for storing my applications’ license. Some applications, such as Hazel, store their keys in a file, which 1Password can also handle via the attachment feature. Go 1Password!

Tcl Full-Feature Stack with struct::stack

The problem

You need a stack solution which provides more than just push and pop for your Tcl script.

The Solutions

In the last post, I discussed the use of Tclx package’s lvarpop and lvarpush to turn an ordinary list into a stack. In this installment, I will discuss the struct::stack package which provides more features.

Using the struct::stack Package

package require struct::stack

# Create a stack and push 3 items
struct::stack myStack
puts "\nCreate a stack and push 3 items"
myStack push peter paul mary

# The stack is just an ordinary Tcl list. Show it.
puts "Stack contents: [myStack get]";  # mary paul peter
puts "Stack size: [myStack size]";     # 3

# Peek and pop some items and display:
puts "\nPeek and Pop:"
puts "Peek 1 item: [myStack peek]";    # mary
puts "Peek 2 items: [myStack peek 2]"; # mary paul
puts "Pop 1 item: [myStack pop]";      # mary
puts "Pop 2 items: [myStack pop 2]";   # paul peter
puts "Stack size: [myStack size]";     # 0

# Stack managements
puts "\nStack Management:"
myStack push mickey michael peter davy
myStack push john paul george ringo
puts "Stack contents: [myStack get]";  # ringo ... mickey
puts "Stack size: [myStack size]";     # 8

puts "Reduce to 3 items, which discards: [myStack trim 3]"
puts "Stack contents: [myStack get]";  # peter michael mickey
puts "Stack size: [myStack size]";     # 3

myStack clear
puts "Clear the Stack"
puts "Stack size: [myStack size]";     # 0

# Done, destroy the stack to reclaim memory
myStack destroy

The Output:

Create a stack and push 3 items
Stack contents: mary paul peter
Stack size: 3

Peek and Pop:
Peek 1 item: mary
Peek 2 items: mary paul
Pop 1 item: mary
Pop 2 items: paul peter
Stack size: 0

Stack Management:
Stack contents: ringo george paul john davy peter michael mickey
Stack size: 8
Reduce to 3 items, which discards: ringo george paul john davy
Stack contents: peter michael mickey
Stack size: 3
Clear the Stack
Stack size: 0

Discussion

  • Line 4: create a new stack and name it ‘myStack’
  • Lines 6: push 3 items into the stack ‘myStack’.
  • Line 9: the get command returns the contents of the stack, with the left-most item being the “top of stack”.
  • Lines 13-18: demonstrate the peek and pop command. Note that the peek command does not alter the stack. Also note the order of items when peeking and popping more than one items.
  • Lines 27-28: the stack at this time has 8 items, the trim command reduce its size down to 3 and return the discarded items. Note the order of the discarded items. The next line show the contents of the stack, which only has 3 items as the result of the trim command.
  • Lines 31-33: demonstrate the clear command to reset the contents of the stack.

Conclusion

The struct::stack package offers the full-feature stack data type which you can use in your Tcl script. I hope this post helps you to get yourself familiar with the stack and consequently feel comfortable using it instead of rolling your own solution.

Tcl: Use Tclx for Lightweight Stack

The problem

You need a lightweight stack for use in your Tcl script. All you need
are push and pop, and nothing fancy.

The Solutions

There are several solutions to this problem. The first is to use the
struct::stack package, which will provide you with a complete stack
implementation. The second is to use the Tclx package, which provides
two commands: lvarpop and lvarpush. The third solution is to role your
own. However, I advice to go with the first solutions unless you have a
good reason to reinvent the wheel.

The first solution is ideal when you need to use a full-feature
stack. The struct::stack package provides the following operations:
create, destroy, clear, get (the whole contents), trim (shrink to a
certain size), get size, and of course, pop and push. Since one of the
operation is destroy, I believe the package use some sort of
object-oriented design to implement the class. For a lightweight use,
the overhead could be noticeable.

The second solution, using lvarpop and lvarpush from Tclx is more
suitable for lightweight use. In this case, the “stack” really is a
normal Tcl list. In this post, I will talk about these two commands and
save discussion of the full-feature struct::stack package to a future
post.

Using lvarpop and lvarpush

Before you can use these two commands, you need to include the Tclx
package in your script. The following is an example script to
demonstrates these two commands.

# Demo: use of lvarpop and lvarpush to pop and push a stack
package require Tclx

# Create a stack and push 3 items
lvarpush myStack peter
lvarpush myStack paul
lvarpush myStack mary

# The stack is just an ordinary Tcl list. Show it.
puts "Stack: $myStack";                 # {mary paul peter}
puts "Items count: [llength $myStack]"; # 3

# Pop some items and display:
puts "First pop: [lvarpop myStack]";    # mary
puts "Second pop: [lvarpop myStack]";   # paul
puts "Third pop: [lvarpop myStack]";    # peter
puts "Fourth pop: [lvarpop myStack]";   # no error, empty

The Output:

Stack: mary paul peter
Items count: 3

First pop: mary
Second pop: paul
Third pop: peter
Fourth pop: 

Discussion

  • Line 2: includes the Tclx package, which provides lvarpop and
    lvarpush
  • Lines 5-7: Push 3 items into the stack ‘myStack’.
  • Lines 10-11: Since the stack is an ordiary list, we can print it
    out and use llength to get the length (stack size).
  • Lines 14-16: Pops the stack using lvarpop to pop the stack.
  • Line 17: Note that popping an empty stack does not cause any
    error. So before popping, you might want to use llength to check the
    stack size.