Monthly Archives: January 2011

A simpler Tcl For Loop

Problem

The for loop (not foreach) in Tcl has a tedious syntax. For example:


for {set i 0} {$i < 10} {incr i} {
   puts $i
}

That is a lot to type for a simple loop. There must be a better way.

The Solution

While exploring the ubiquitous Tclx package, I found it has just what we need in the form of the loop command whose syntax is:

loop var first limit ?incr? command

Note the fourth parameter, the crement value is optional and default to 1. This command simplifies the above code block as:


loop i 0 10 {
   puts $i
}

Note that the values of i in both cases go from 0 to 9, not 10. Like the for command, the loop command allows the loop counter to skip or to go backward by chaging the increment value:


# Skip count: 0 2 4 6 8
loop i 0 10 2 {
   puts $i
}

# Count backward: 10 .. 1
loop i 10 0 -1 {
   puts $i
}

Discussion

Besides making coding loop easier, the loop command also offers another advantage: the limit is evaluated only once, thus improve performance.

The loop command does have one disadvantage over the built-in for command: to use it, you must include the Tclx package. If your script does not use Tclx, this inclusion will incur some small performance penalty up front as the script will have to load Tclx into memory. The Tclx package also increases your script’s memory foot print; depend on your script’s size, the increase could be significant. However, if your script does need Tclx for other reasons, you will get the loop command for free.

Finally, if you want a do .. until or do .. while loop, please read my post titled Does Tcl Has Do Loop?.

Advertisements

Hide Irrelevant Files from Eclipse’s Script Explorer

The Problem

In Eclipse, I would like to hide irrelevant files such as object files (*.o or *.obj) from the Script Explorer (the file hierarchy).

I have asked this question in superuser.com, but received no answer. So, I set out to find the answer myself.

The Solution

A trip to Google turns up this page: Creating resource filters, which solves my problem. The solution is not obvious and initial search for terms such as Eclipse hide files does not turn up useful information. This is the reason I create the blog to remind myself and to help others.

I wish there is a common set of filters which works for all projects, but for now, I have to set these filters for all of my projects.

Use subst to Deal with Long Path Names

The Problem

I want to shorten the paths to a directory that is buried deep in the file system. In my daily work, I often need to cd into such directory as long as:

C:\projects\tests\utilities\fileutil

Although Windows command line has file name completion, it is still a long and tedious process to change into and out of these directories.

The Solution

There are several solutions, such as using environment variable to point to the destination directory, or to create alias to cd there (via doskey). My favorite solution uses subst to substitute the long path with a drive letter:

subst F: C:\projects\tests\utilities\fileutil

From this point on, I can switch to this long directory using the shorter drive alias. Instead of

cd C:\projects\tests\utilities\fileutil

I can get to it by changing the drive:

F:

By using a drive to access a deeply buried directory, I also get around the problem of 260-character path name limit in Windows.

Profiling Your Tcl – Output to CSV

This is the last part of the Tcl profiler series. In part one, I presented a 5-line tool which profiles your Tcl script. In part two, I refined it to show sorted timings and number of calls. In this part, I am going to create a tool to output the profiler information to a CSV file, useful for using it with other analysis tools such as Excel.

The Tool

Save the following script to a file and call it profile_me_v3.

#!/usr/bin/env tclsh

# File: profile_me_v3

namespace eval profiler_utils {
    proc getHeaders {info} {
        set headers {function}
        foreach {k v} $info {
            lappend headers $k
        }
        return [join $headers ,]
    }
    
    proc getValues {function info} {
        set values $function
        foreach {k v} $info {
            lappend values $v
        }
        return [join $values ,]
    }

    proc reportStatistics {} {
        set outfile [open "profile_me_v3.csv" w]
        set needHeaders 0
        foreach {function info} [::profiler::dump] {
            if {$needHeaders == 0} {
                puts $outfile [getHeaders $info]
            }
            incr needHeaders
            puts $outfile [getValues $function $info]
        }
        close $outfile
    }
}

package require profiler
profiler::init

# Call the script
set scriptName [lindex $::argv 0]
set ::argv [lrange $::argv 1 end]
source $scriptName

# Lastly, display profiler info
profiler_utils::reportStatistics

Discussion

After profiling a script, function reportStatistics (line 22-33) opens a CSV file and and writes to it the headers (line 27) and one line for each function (line 30). The variable needHeaders is used to ensure the headers are written only once.

Note that the function ::profiler::dump returns a list of alternate values: the function name and a sublist (info). The sublist, in turn, contains name/value pairs of profiler info.

Note also that function reportStatistics hard-codes its output to a file named “profile_me_v3.csv” for the sake of simplicity. As an exercise, you might want to change the file name based on the scriptName variable.

Below is the contents of the CSV file:

function,callCount,callerDist,compileTime,totalRuntime,averageRuntime,stddevRuntime,covpercentRuntime,descendantTime,averageDescendantTime,descendants
::call_all_tests,1,GLOBAL 1,133870,133870,0,0,0,45792,45792,::test1 ::test2
::tcl::clock::add,0,,0,0,0,0,0,0,0,
::tcl::clock::format,0,,0,0,0,0,0,0,0,
::tcl::clock::scan,0,,0,0,0,0,0,0,0,
::test1,6,GLOBAL 5 ::call_all_tests 1,121579,233013,38835,40568,104.5,0,0,
::test2,11,GLOBAL 10 ::call_all_tests 1,24335,228723,20793,1412,6.8,0,0,

When viewed in a spreadsheet application, the above CSV might look like this:

Profiling Your Tcl – Show Profiler’s Summary

Introduction

This is the second part of my three-part series showing how simple it is to profile a Tcl script. If you have not done so, please read part one before continuing with this article.

In part one, I created a simple profiler tool which offer immediate usefulness. The output of this tool, while useful, leave much to be desired. In this article, I will update my tool to answer these questions:

  • Which functions take the most time to run?
  • Which functions was called the most?
  • Which functions which was defined, but never get called?

The Solution

Below is the version 2 of the profiler tool, save it to a file called profile_me_v2, make it executable and copy it to a directory in the path:

#!/usr/bin/env tclsh

# File: profile_me_v2

namespace eval profiler_utils {
    proc printProfilerInfo {sortKey} {
        puts "\nFunctions that are sorted by $sortKey:"
        set list [lsort -decreasing -integer -index 1 \
            [::profiler::sortFunctions $sortKey]]
        
        foreach entry $list {
            set functionName [lindex $entry 0]
            set functionInfo [lindex $entry 1]
            puts [format "- %10d - %s" $functionInfo $functionName]
        }
    }
    
    proc reportStatistics {} {
        printProfilerInfo totalRuntime
        printProfilerInfo avgRuntime
        printProfilerInfo exclusiveRuntime
        printProfilerInfo avgExclusiveRuntime
        printProfilerInfo calls
    }
}

package require profiler
profiler::init

# Call the script
set scriptName [lindex $::argv 0]
set ::argv [lrange $::argv 1 end]
source $scriptName

# Lastly, display profiler info
puts "\n[string repeat - 40]"
profiler_utils::reportStatistics

Discussion

This script introduces two functions: reportStatistics and printProfilerInfo. Function reportStatistics, called at the end of the script, prints lists of functions, sorted by a few criteria such as total run time, average run time, or number of calls. Function printProfilerInfo does the actual reporting work.

Lines 8-9 obtains a list of functions and measurements (such as total run time). Since this list is sorted in ascending order, I re-sort it in decreasing order to suit my purpose.

Lines 11-15 show these functions, along with their measurements

Using the Tool

You can run profile_me_v2 as follow:

profile_me_v2 example_script.tcl

One interesting note: if example_script.tcl takes on command-line parameters, you can specify them as followed:

profile_me_v2 example_script.tcl argument1 argument2 ...

The output:

$ profile_me_v2 example_script.tcl 
This is test1
This is test1
This is test1
This is test1
This is test1
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test1
This is test2

----------------------------------------

Functions that are sorted by totalRuntime:
-     393935 - ::test1
-     229716 - ::test2
-     134194 - ::call_all_tests
-          0 - ::tcl::clock::scan
-          0 - ::tcl::clock::format
-          0 - ::tcl::clock::add

Functions that are sorted by avgRuntime:
-      78787 - ::test1
-      22971 - ::test2

Functions that are sorted by exclusiveRuntime:
-     393935 - ::test1
-     229716 - ::test2
-      88286 - ::call_all_tests
-          0 - ::tcl::clock::add
-          0 - ::tcl::clock::format
-          0 - ::tcl::clock::scan

Functions that are sorted by avgExclusiveRuntime:
-      88286 - ::call_all_tests
-      65655 - ::test1
-      20883 - ::test2

Functions that are sorted by calls:
-         11 - ::test2
-          6 - ::test1
-          1 - ::call_all_tests
-          0 - ::tcl::clock::scan
-          0 - ::tcl::clock::format
-          0 - ::tcl::clock::add

Conclusion

Version two offers some improvements over the first version. For example, it shows that function ::test1 takes the longest runtime, or function ::tcl::clock::add never gets called.

The output from this version is much more useful than that of version one. However, a CSV output will be most useful, especially when coupled with other analysis tools such as Microsoft Excel. In part three of the series, I will create the CSV output.

Simple Profile for Your Tcl Script

The Problem

You want to profile your Tcl script to determine which functions need optimization, but profiling seems to be a complicated process.

The Overview

Profiling in Tcl is very simple, so simple that it does not take more than five lines of code. What’s more, you don’t have to modify your script. In a series of three articles, I will show you how to profile your Tcl scripts. The first part (this article) will show the “lazy-man” approach: a five-liner profiler tool. The second article will go a little more in depth to customize the profiler’s output in about 30 lines of code. Finally, the third article exports the profiler’s information to CSV format for use with other programs such as Microsoft Excel.

The Solution

We begin our journey by creating a profiler tool. This tools is written also in Tcl and about five lines long. Save the following lines in a file called profile_me:

#!/usr/bin/env tclsh

# File: profile_me
package require profiler
profiler::init

# Call the script
source $::argv

# Display profiler info
puts "\n[string repeat - 40]"
puts [::profiler::print]

The next step is to make it executable:

chmod +x profile_me

Finally, copy this tool to a directory in your path to make it accessible from any where. Now you are ready to profile your script.

Let’s assume that you have a Tcl script called example_script.tcl:

#!/usr/bin/env tclsh

proc test1 {} {
    puts "This is test1"
}

proc test2 {} {
    puts "This is test2"
}

proc call_all_tests {} {
    test1
    test2
}

# main
for {set i 0} {$i < 5} {incr i} { test1 }
for {set i 0} {$i < 10} {incr i} { test2 }
call_all_tests

To start profiling your Tcl script, issue the following command:

profile_me example_script.tcl

The output:

This is test1
This is test1
This is test1
This is test1
This is test1
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test2
This is test1
This is test2

----------------------------------------
Profiling information for ::call_all_tests
============================================================
            Total calls:  1
    Caller distribution:
  GLOBAL:  1
           Compile time:  143706
          Total runtime:  143706
        Average runtime:  0
          Runtime StDev:  0
         Runtime cov(%):  0
  Total descendant time:  50718
Average descendant time:  50718
Descendants:
  ::test1:  1
  ::test2:  1

Profiling information for ::tcl::clock::add
============================================================
            Total calls:  0

Profiling information for ::tcl::clock::format
============================================================
            Total calls:  0

Profiling information for ::tcl::clock::scan
============================================================
            Total calls:  0

Profiling information for ::test1
============================================================
            Total calls:  6
    Caller distribution:
  ::call_all_tests:  1
  GLOBAL:  5
           Compile time:  150958
          Total runtime:  269793
        Average runtime:  44965
          Runtime StDev:  51950
         Runtime cov(%):  115.5
  Total descendant time:  0
Average descendant time:  0
Descendants:
  none

Profiling information for ::test2
============================================================
            Total calls:  11
    Caller distribution:
  ::call_all_tests:  1
  GLOBAL:  10
           Compile time:  26261
          Total runtime:  454414
        Average runtime:  41310
          Runtime StDev:  46538
         Runtime cov(%):  112.7
  Total descendant time:  0
Average descendant time:  0
Descendants:
  none

Conclusion

Profiling a Tcl script is very simple. Our tool, profile_me, demonstrates that you don’t have write a lot of code to accomplish your objective. The output of profile_me is useful right away: it shows the total runtime for each function, along with the number of times the function was called. Based on this information, you can prioritize which function to optimize.

Due to its simplicity, profile_me does not display summary of information, such as which function takes the longest to run, which function is called the most. In part two, I will address these issues.

A Solution for tclsh Editing Woes

The Problem

People who runs tclsh under Windows can use the up arrow, among other keys, to recall last commands issued. However, on the Mac and Linux environments, the up arrow only gives them ^[[A, which make for a frustrating experience. How do we get the editing capabilities that Windows users enjoy?

The Solution

One of the solution which I like is to use tkcon, tk-based Tcl shell, which offer excellent command line editing, among other features. On my Mac, tkcon is in /usr/bin/tkcon, the the location may vary. To determine if tkcon is installed in your system, issue the following command:

which tkcon

If you don’t see any output, you will need to install it. On the Mac, head to http://tkcon.sourceforge.net/, download it and follow the instruction to install. On Linux, the instructions are different, depending on which distribution so you should search for it.

Once installed, you can start tkcon using the following command:

tkcon &

The ampersand will launch tkcon in the background and return control immediately to the terminal. On my Mac, tkcon pops under other windows, so after launching, I have to task switch to it using the command + tab key combination. I hope you enjoy using tkcon as I do.