A simpler Tcl For Loop

Problem

The for loop (not foreach) in Tcl has a tedious syntax. For example:

for {set i 0} {$i < 10} {incr i} {
   puts $i
}

That is a lot to type for a simple loop. There must be a better way.

The Solution

While exploring the ubiquitous Tclx package, I found it has just what we need in the form of the loop command whose syntax is:

loop var first limit ?incr? command

Note the fourth parameter, the crement value is optional and default to 1. This command simplifies the above code block as:

loop i 0 10 {
   puts $i
}

Note that the values of i in both cases go from 0 to 9, not 10. Like the for command, the loop command allows the loop counter to skip or to go backward by chaging the increment value:

# Skip count: 0 2 4 6 8
loop i 0 10 2 {
   puts $i
}

# Count backward: 10 .. 1
loop i 10 0 -1 {
   puts $i
}

Discussion

Besides making coding loop easier, the loop command also offers another advantage: the limit is evaluated only once, thus improve performance.

The loop command does have one disadvantage over the built-in for command: to use it, you must include the Tclx package. If your script does not use Tclx, this inclusion will incur some small performance penalty up front as the script will have to load Tclx into memory. The Tclx package also increases your script’s memory foot print; depend on your script’s size, the increase could be significant. However, if your script does need Tclx for other reasons, you will get the loop command for free.

Finally, if you want a do .. until or do .. while loop, please read my post titled Does Tcl Has Do Loop?.

Hide Irrelevant Files from Eclipse’s Script Explorer

The Problem

In Eclipse, I would like to hide irrelevant files such as object files (*.o or *.obj) from the Script Explorer (the file hierarchy).

I have asked this question in superuser.com, but received no answer. So, I set out to find the answer myself.

The Solution

A trip to Google turns up this page: Creating resource filters, which solves my problem. The solution is not obvious and initial search for terms such as Eclipse hide files does not turn up useful information. This is the reason I create the blog to remind myself and to help others.

I wish there is a common set of filters which works for all projects, but for now, I have to set these filters for all of my projects.

Use subst to Deal with Long Path Names

The Problem

I want to shorten the paths to a directory that is buried deep in the file system. In my daily work, I often need to cd into such directory as long as:

C:\projects\tests\utilities\fileutil

Although Windows command line has file name completion, it is still a long and tedious process to change into and out of these directories.

The Solution

There are several solutions, such as using environment variable to point to the destination directory, or to create alias to cd there (via doskey). My favorite solution uses subst to substitute the long path with a drive letter:

subst F: C:\projects\tests\utilities\fileutil

From this point on, I can switch to this long directory using the shorter drive alias. Instead of

cd C:\projects\tests\utilities\fileutil

I can get to it by changing the drive:

F:

By using a drive to access a deeply buried directory, I also get around the problem of 260-character path name limit in Windows.

Profiling Your Tcl – Output to CSV

This is the last part of the Tcl profiler series. In part one, I presented a 5-line tool which profiles your Tcl script. In part two, I refined it to show sorted timings and number of calls. In this part, I am going to create a tool to output the profiler information to a CSV file, useful for using it with other analysis tools such as Excel.

The Tool

Save the following script to a file and call it profile_me_v3.

#!/usr/bin/env tclsh

# File: profile_me_v3

namespace eval profiler_utils {
    proc getHeaders {info} {
        set headers {function}
        foreach {k v} $info {
            lappend headers $k
        }
        return [join $headers ,]
    }
    
    proc getValues {function info} {
        set values $function
        foreach {k v} $info {
            lappend values $v
        }
        return [join $values ,]
    }

    proc reportStatistics {} {
        set outfile [open "profile_me_v3.csv" w]
        set needHeaders 0
        foreach {function info} [::profiler::dump] {
            if {$needHeaders == 0} {
                puts $outfile [getHeaders $info]
            }
            incr needHeaders
            puts $outfile [getValues $function $info]
        }
        close $outfile
    }
}

package require profiler
profiler::init

# Call the script
set scriptName [lindex $::argv 0]
set ::argv [lrange $::argv 1 end]
source $scriptName

# Lastly, display profiler info
profiler_utils::reportStatistics

Discussion

After profiling a script, function reportStatistics (line 22-33) opens a CSV file and and writes to it the headers (line 27) and one line for each function (line 30). The variable needHeaders is used to ensure the headers are written only once.

Note that the function ::profiler::dump returns a list of alternate values: the function name and a sublist (info). The sublist, in turn, contains name/value pairs of profiler info.

Note also that function reportStatistics hard-codes its output to a file named “profile_me_v3.csv” for the sake of simplicity. As an exercise, you might want to change the file name based on the scriptName variable.

Below is the contents of the CSV file:

function,callCount,callerDist,compileTime,totalRuntime,averageRuntime,stddevRuntime,covpercentRuntime,descendantTime,averageDescendantTime,descendants
::call_all_tests,1,GLOBAL 1,133870,133870,0,0,0,45792,45792,::test1 ::test2
::tcl::clock::add,0,,0,0,0,0,0,0,0,
::tcl::clock::format,0,,0,0,0,0,0,0,0,
::tcl::clock::scan,0,,0,0,0,0,0,0,0,
::test1,6,GLOBAL 5 ::call_all_tests 1,121579,233013,38835,40568,104.5,0,0,
::test2,11,GLOBAL 10 ::call_all_tests 1,24335,228723,20793,1412,6.8,0,0,

When viewed in a spreadsheet application, the above CSV might look like this:

Profiling Your Tcl – Show Profiler’s Summary

Introduction

This is the second part of my three-part series showing how simple it is to profile a Tcl script. If you have not done so, please read part one before continuing with this article.

In part one, I created a simple profiler tool which offer immediate usefulness. The output of this tool, while useful, leave much to be desired. In this article, I will update my tool to answer these questions:

• Which functions take the most time to run?
• Which functions was called the most?
• Which functions which was defined, but never get called?

The Solution

Below is the version 2 of the profiler tool, save it to a file called profile_me_v2, make it executable and copy it to a directory in the path:

#!/usr/bin/env tclsh

# File: profile_me_v2

namespace eval profiler_utils {
    proc printProfilerInfo {sortKey} {
        puts "\nFunctions that are sorted by $sortKey:"
        set list [lsort -decreasing -integer -index 1 \
            [::profiler::sortFunctions $sortKey]]
        
        foreach entry $list {
            set functionName [lindex $entry 0]
            set functionInfo [lindex $entry 1]
            puts [format "- %10d - %s" $functionInfo $functionName]
        }
    }
    
    proc reportStatistics {} {
        printProfilerInfo totalRuntime
        printProfilerInfo avgRuntime
        printProfilerInfo exclusiveRuntime
        printProfilerInfo avgExclusiveRuntime
        printProfilerInfo calls
    }
}

package require profiler
profiler::init

# Call the script
set scriptName [lindex $::argv 0]
set ::argv [lrange $::argv 1 end]
source $scriptName

# Lastly, display profiler info
puts "\n[string repeat - 40]"
profiler_utils::reportStatistics

Discussion
This script introduces two functions: reportStatistics and printProfilerInfo. Function reportStatistics, called at the end of the script, prints lists of functions, sorted by a few criteria such as total run time, average run time, or number of calls. Function printProfilerInfo does the actual reporting work.
Lines 8-9 obtains a list of functions and measurements (such as total run time). Since this list is sorted in ascending order, I re-sort it in decreasing order to suit my purpose.
Lines 11-15 show these functions, along with their measurements
Using the Tool
You can run profile_me_v2 as follow:
profile_me_v2 example_script.tcl

One interesting note: if example_script.tcl takes on command-line parameters, you can specify them as followed:
profile_me_v2 example_script.tcl argument1 argument2 ...

The output:
$ profile_me_v2 example_script.tcl 
This is test1
This is test1
This is test1
This is test1
This is test1
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test1
This is test2

----------------------------------------

Functions that are sorted by totalRuntime:
-     393935 - ::test1
-     229716 - ::test2
-     134194 - ::call_all_tests
-          0 - ::tcl::clock::scan
-          0 - ::tcl::clock::format
-          0 - ::tcl::clock::add

Functions that are sorted by avgRuntime:
-      78787 - ::test1
-      22971 - ::test2

Functions that are sorted by exclusiveRuntime:
-     393935 - ::test1
-     229716 - ::test2
-      88286 - ::call_all_tests
-          0 - ::tcl::clock::add
-          0 - ::tcl::clock::format
-          0 - ::tcl::clock::scan

Functions that are sorted by avgExclusiveRuntime:
-      88286 - ::call_all_tests
-      65655 - ::test1
-      20883 - ::test2

Functions that are sorted by calls:
-         11 - ::test2
-          6 - ::test1
-          1 - ::call_all_tests
-          0 - ::tcl::clock::scan
-          0 - ::tcl::clock::format
-          0 - ::tcl::clock::add


Conclusion
Version two offers some improvements over the first version. For example, it shows that function ::test1 takes the longest runtime, or function ::tcl::clock::add never gets called.
The output from this version is much more useful than that of version one. However, a CSV output will be most useful, especially when coupled with other analysis tools such as Microsoft Excel. In part three of the series, I will create the CSV output.
					

Simple Profile for Your Tcl Script

The Problem

You want to profile your Tcl script to determine which functions need optimization, but profiling seems to be a complicated process.

The Overview

Profiling in Tcl is very simple, so simple that it does not take more than five lines of code. What’s more, you don’t have to modify your script. In a series of three articles, I will show you how to profile your Tcl scripts. The first part (this article) will show the “lazy-man” approach: a five-liner profiler tool. The second article will go a little more in depth to customize the profiler’s output in about 30 lines of code. Finally, the third article exports the profiler’s information to CSV format for use with other programs such as Microsoft Excel.

The Solution

We begin our journey by creating a profiler tool. This tools is written also in Tcl and about five lines long. Save the following lines in a file called profile_me:

#!/usr/bin/env tclsh

# File: profile_me
package require profiler
profiler::init

# Call the script
source $::argv

# Display profiler info
puts "\n[string repeat - 40]"
puts [::profiler::print]

The next step is to make it executable:

chmod +x profile_me

Finally, copy this tool to a directory in your path to make it accessible from any where. Now you are ready to profile your script.

Let’s assume that you have a Tcl script called example_script.tcl:

#!/usr/bin/env tclsh

proc test1 {} {
    puts "This is test1"
}

proc test2 {} {
    puts "This is test2"
}

proc call_all_tests {} {
    test1
    test2
}

# main
for {set i 0} {$i < 5} {incr i} { test1 }
for {set i 0} {$i < 10} {incr i} { test2 }
call_all_tests

To start profiling your Tcl script, issue the following command:

profile_me example_script.tcl

The output:

This is test1
This is test1
This is test1
This is test1
This is test1
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test1
This is test2

----------------------------------------
Profiling information for ::call_all_tests
============================================================
            Total calls:  1
    Caller distribution:
  GLOBAL:  1
           Compile time:  143706
          Total runtime:  143706
        Average runtime:  0
          Runtime StDev:  0
         Runtime cov(%):  0
  Total descendant time:  50718
Average descendant time:  50718
Descendants:
  ::test1:  1
  ::test2:  1

Profiling information for ::tcl::clock::add
============================================================
            Total calls:  0

Profiling information for ::tcl::clock::format
============================================================
            Total calls:  0

Profiling information for ::tcl::clock::scan
============================================================
            Total calls:  0

Profiling information for ::test1
============================================================
            Total calls:  6
    Caller distribution:
  ::call_all_tests:  1
  GLOBAL:  5
           Compile time:  150958
          Total runtime:  269793
        Average runtime:  44965
          Runtime StDev:  51950
         Runtime cov(%):  115.5
  Total descendant time:  0
Average descendant time:  0
Descendants:
  none

Profiling information for ::test2
============================================================
            Total calls:  11
    Caller distribution:
  ::call_all_tests:  1
  GLOBAL:  10
           Compile time:  26261
          Total runtime:  454414
        Average runtime:  41310
          Runtime StDev:  46538
         Runtime cov(%):  112.7
  Total descendant time:  0
Average descendant time:  0
Descendants:
  none

Conclusion

Profiling a Tcl script is very simple. Our tool, profile_me, demonstrates that you don’t have write a lot of code to accomplish your objective. The output of profile_me is useful right away: it shows the total runtime for each function, along with the number of times the function was called. Based on this information, you can prioritize which function to optimize.

Due to its simplicity, profile_me does not display summary of information, such as which function takes the longest to run, which function is called the most. In part two, I will address these issues.

A Solution for tclsh Editing Woes

The Problem

People who runs tclsh under Windows can use the up arrow, among other keys, to recall last commands issued. However, on the Mac and Linux environments, the up arrow only gives them ^[[A, which make for a frustrating experience. How do we get the editing capabilities that Windows users enjoy?

The Solution

One of the solution which I like is to use tkcon, tk-based Tcl shell, which offer excellent command line editing, among other features. On my Mac, tkcon is in /usr/bin/tkcon, the the location may vary. To determine if tkcon is installed in your system, issue the following command:

which tkcon

If you don’t see any output, you will need to install it. On the Mac, head to http://tkcon.sourceforge.net/, download it and follow the instruction to install. On Linux, the instructions are different, depending on which distribution so you should search for it.

Once installed, you can start tkcon using the following command:

tkcon &

The ampersand will launch tkcon in the background and return control immediately to the terminal. On my Mac, tkcon pops under other windows, so after launching, I have to task switch to it using the command + tab key combination. I hope you enjoy using tkcon as I do.

Does Tcl Have Do Loop?

When I first started scripting with Tcl, I wish it has C’s do loop, that is, do … until and do … while. After asking my co-workers, the conclusion was Tcl does have do loop. It turns out that Tcl indeed does not have do loop, but it does provide package which does: the control package.

Below is a sample script which uses the control package and demonstrates the two different do loops

#!/usr/bin/env tclsh

# Demonstrates do loop

package require control

# Example 1: Prints 1 to 10 using do..until
set i 0
control::do {
    incr i
    puts -nonewline "$i "
} until {$i >= 10}
puts ""

# Example 2: Same output using do..while
set i 0
control::do {
    incr i
    puts -nonewline "$i "
} while {$i < 10}
puts ""

The first step is to include the control package (line 5). After that, you can use the do loop as you would in C.

Python List Assignment Caveat

The Problem

In Python, the assignment operator (=) assign the reference to the list instead of making copy of it. The following interaction will demonstrate this concept:

>>> original = [1,2,3]
>>> reference = original # assign the reference
>>> reference
[1, 2, 3]
>>> reference.append(4)
>>> reference
[1, 2, 3, 4]
>>> original
[1, 2, 3, 4]
>>>

• In line 1, we create a new list, original, with three elements.
• In line 2, we assign this list to a new variable, reference.
• In line 5, we append a new value to reference.
• What happens to the original list? Line 8 and 9 might bring shock to those who are new to Python. What happened?

The answer is the assignment in line 2 assigns the reference to the list so both original and reference will now share the same list and that changes made to one will reflect in the other. So, how do we really make copy of the list instead of sharing the reference?

The Solutions

To make a new copy of a list, there are two different approaches: using the list() function and using the sub list mechanism. The following interactive session will demonstrate the list() function:

>>> original = [1,2,3]
>>> copy = list(original)
>>> copy.append(4)
>>> copy
[1, 2, 3, 4]
>>> original
[1, 2, 3]
>>> 

In line 2, we use the list function to create a copy of the original and assign that list to the variable copy. Any action on copy at that point on will not affect the original list (line 6 and 7)

The following interactive session will demonstrate the sub list feature:

>>> original = [1,2,3]
>>> copy = original[:]
>>> copy.append(4)
>>> copy
[1, 2, 3, 4]
>>> original
[1, 2, 3]
>>> 

Discussion

while both methods accomlish the same goal, many argues that the sub list method is faster. However, the list() function wins point for being easier to understand, especially to new comers.

Bash – Log to Screen and File Simultaneously

The Problem

In my bash script, I would like to print both the the standard output (typically the screen) and a file.

The Solution

Below is a simple script which demonstrate the solution.

#!/bin/sh

# whatis: Demo script that prints to both screen and a file

{
    echo Logging demo
    echo Output will go both the screen and logging.log
    # Other lines which might produce output here
    
} | tee logging.log

Discussion

Normally, the output from a bash script goes to the standard output, typically the screen. To redirect the output to both the standard output and a file, we employ the tee program. By surrounding the block of code with curly braces, we redirect the whole block, not just individual lines.