\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#86: Improved Debugger Support}
\date{February 08, 2002}
\author{Peter MacDonald, Peter MacDonald}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#86}&\textbf{Improved Debugger Support}\\\hline
Author&

Peter MacDonald \tipmail{peter}{browsex.com}
\par
Peter MacDonald \tipmail{peter}{pdqi.com}
\\
Created&Friday, \(\text{8}^{\text{th}}\) February 2002\\
Type&Project\\
Tcl Version&\(8.7\)\\
State&Draft\\
Vote&Pending\\
Version&\tipversion$Revision: 1.26 $\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes the storage by Tcl of source code file-name and line-numbering information, making it available at script execution time. It also adds additional \textbf{trace} and \textbf{info} subcommands to make it easier for a debugger to control a Tcl script much as \textit{gdb} can control a C program.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Rationale}

Currently, although Tcl provides quite reasonable information to users in error traces, the line numbers within those traces are always relative to the evaluation context containing them (often the procedure, but not always) and not to the script file containing the procedure. This is substantially different to virtually every other computer language and makes correlating errors with the source line that caused them much more difficult. This also makes coupling a Tcl interpreter to an external debugging tool more difficult. This TIP proposes adding new interfaces to the Tcl core to make such debugging activity easier.

A new \textbf{trace execution} option enables Tcl to track line number and source file information associated with statements being executed and call a single callback. A new \textit{'info line} option provides access to line number information. As a result, it becomes a simple matter to implement a debugger for Tcl, in Tcl. Furthermore, the implementation also serves as example usage of the C interface, enabling similar capabilities at the lower level.

A simple Tcl debugger, \textit{tgdb}, written in Tcl and emulating \textit{gdb}, is included with this TIP to demonstrate the use of this interface. \textit{tgdb} runs and controls a Tcl application in a sub-interp using \textbf{trace execution} and \textbf{interp alias}. It supports breakpoints on lines/procs/vars/etc, single-stepping, up/down stack and evals. It is designed to work both as a commandline and a slave process (see \textit{Reference Implementation}).

Finally, upon error within a procedure, the file path and absolute (as opposed to relative) line number are printed out when available, even in the case where called from an after or callback invocation. Aside from aiding the user in more easily locating and dealing with errors, the message is machine parseable For example: automatically bring the user into an editor at the offending line.

\section{Specification}

A new \textbf{execution} subcommand to the \textbf{trace} command.

\begin{quote}
\textbf{trace execution} \textit{target} ?\textit{level}?
\end{quote}

This arranges for an execution trace to be setup for commands at nesting \textit{level} or above, thereby providing a simple Tcl interface for tracing commands to say, implement a debugger. With no arguments, the current target is returned. If target is the empty string, the execution trace is removed. The \textit{target} argument is assumed to be a command string to be executed. When level is not specified, it defaults to 0, meaning trace all commands. For each traced command, the following data will be produced:
\begin{itemize}
\item{}
linenumber

The line number the instruction begins on.
\item{}
filename

The fully normalized file name.
\item{}
nestlevel

The nesting level of the command.
\item{}
stacklevel

The stack call level as per \textbf{info level}.
\item{}
curnsproc

The current fully qualified namespace/function.
\item{}
cmdname

The fully qualified command name of the command to be invoked.
\item{}
command

The command line to be executed including arguments.
\item{}
flags

Integer bit flags, currently bit 1 is set for breakpoint.
\end{itemize}

The target is presumed to be a valid Tcl command onto which is appended the above arguments before evaluation. Any return from the command other than a normal return results in the command not being executed. As with all traces, execution tracing is disabled within a trace handler.

Second, a new \textbf{line} subcommand to \textbf{info} gives access to the file path and line number information. It takes subcommands of its own in turn:
\begin{itemize}
\item{}
\textbf{info line current}
\begin{description}
\item[Returns a list with two items]
the line number and file name of the current statement.
\end{description}
\item{}
\textbf{info line number} \textit{proc} ?\textit{line}?

Get/set the line number for the start of \textit{proc}. In get mode, returns the definition line number in the message body.
\item{}
\textbf{info line level} \textit{number}

Like the info level command, but returns the line number and file name from which the call at level number originates. For use with \textit{trace execution}.
\item{}
\textbf{info line file} ?\textit{proc} ?\textit{file}??

Get/set the sourced file path for \textit{proc}. If \textit{proc} is not specified, dump all known sourced file paths.
\item{}
\textbf{info line find} \textit{file line}

Given a file path and line number, return the \textit{proc} name containing line number \textit{line}. A new nonstatic procedure \textit{TclFindProcByLine()} provides this function.
\item{}
\textbf{info line relativeerror} ?\textit{bool}?

Set to 1 to disable absolute line number and file path on a procedure error. This demotes procedure traceback errors to the same format as all other traceback errors, that is, using the relative the line number and file name.
\end{itemize}

These exhibit the following behavior:
\begin{itemize}
\item{}
\textit{What does this do when you redefine a proc?}

You get the values from the latest definition.
\item{}
\textit{What about when you use interp aliases?}

You get an error, as it is not considered a proc.
\item{}
\textit{And if proc itself gets redefined by someone's special debugger?}

If the definition is not the result of a source, the file/line come back as an empty string.
\end{itemize}

Third, a new \textit{info} subcommand \textit{return}.
\begin{itemize}
\item{}
\textbf{info return}

When in \textit{trace execution} mode, returns the saved last result of the previously executed command. Otherwise returns an empty string. Commands executed as part of a trace handler do not affect or change the saved last result. 
\end{itemize}

Forth, an additional flag option \textit{debug} to \textit{trace add variable}
\begin{itemize}
\item{}
\textbf{trace add variable \{read write unset debug\} command}

Used in conjuction with \textit{read}, \textit{write}, and \textit{unset}, this option allows a debugger to set read/write traces that will not trigger the execution trace. In other words, it specifies that the command is debugger code that is not to be traced. Normally, debugger code is entered via the \textbf{trace execution} handler and so has tracing disabled. This just provides a similar feature for \textbf{trace variable}
\end{itemize}

Fifth, a new \textbf{breakpoint} subcommand to the \textbf{trace} command.

\begin{quote}
\textbf{trace breakpoint} ??\textit{line file} ?\textit{level} ...??
\end{quote}

The \textit{trace breakpoint} manages a list of breakpoints that cause an \textit{execution trace} to trigger, even when the nestlevel is exceeded. With no arguments it returns a ternery list of all breakpoints in sets of the triples: line, file, and state. With two arguments, the current state for the breakpoint is returned. With three or more arguments, new breakpoints are created. If created with a state of zero, the breakpoint is considered inactive. Setting the state of a breakpoint to the empty string effectively deletes the breakpoint. A state set to an N greater than zero triggers every Nth time.

\section{Changes}

Sourced file paths are stored per interp in a hash table. File/line numbering information is also stored in the \textit{Interp}, \textit{Proc}, \textit{After}, and \textit{CallFrame} structures. Newline counting/shifting code was added to \textit{proc}, \textit{while}, \textit{for}, \textit{foreach}, and \textit{if}. All but the non-trivial code is active only when the new TRACE\_LINE\_NUMBERS interp flag is active, which is the case when using \textit{trace execution}.

Most new variables within Interp are in the struct subfield sourceInfo of type \textit{Tcl\_SourceInfo}, which can be retrieved via the new \textit{Tcl\_GetSourceInfo(interp)} stubbed/public call.

\section{Overhead/Impact}

The runtime impact to Tcl should be modest: a few 10's of kilobytes of memory, even for moderately large programs. Most of the space impact occurs in storing the file paths. A typical example from a large system:
\begingroup\small\begin{verbatim}
  100 sourced files * 100 bytes = 10K.
\end{verbatim}
\endgroup

The other space overhead adds up to several words (8 bytes on a 32-bit platform) per defined procedure, plus an additional words in the \textit{Interp} structure.

Runtime processing overhead should be negligible.

However, there have been no benchmarks done to validate these assertions.

\section{Reference Implementation}

This patch is against Tcl 8.4.9 and represents a complete rework of the approach.

\url{http://pdqi.com/download/tclline-8.4.9.diff.gz}

There is a simple demonstration debugger script: \textit{tgdb.tcl}.

\url{http://pdqi.com/download/tgdb.tcl}

\section{Previous/Old Reference Implementation}

\url{http://pdqi.com/download/tclline-cvs.diff.gz} --- Patch against CVS head.

\url{http://pdqi.com/download/tclline-8.4.6.diff.gz} --- Patch against Tcl 8.4.6

The CVS patch was against the CVS head is as of June 13/2004. These have been lightly tested against numerous small Tcl programs.

There is also an initial version of a debugger: \textit{tgdb}.

\url{http://pdqi.com/download/tgdb-2.0.tar.gz}

\textit{tgdb} emulates the basic commands of \textit{gdb} (\textit{s}, \textit{n}, \textit{c}, \textit{f}, \textit{bt}, \textit{break}, \textit{info locals}, etc). This newest version also supports watchpoints and display variables. With \textit{load} and \textit{run} commands added, \textit{tgdb} should probably work even with \textit{emacs} and \textit{ddd}.

An additional package \textit{pdqi} provides \textit{tdb}, a GUI front-end to \textit{gdb}, modified to also work with \textit{tgdb}.

\section{Possible Future Enhancements}

Build and store a line number table internally during parse?

Line number lookup via the source string. A simple way to implement this might be to lookup string against the \textit{codePtr-\tiprangle{}source+bestSrcOffset} as returned by \textit{GetSrcInfoForPc()}.

Add special handling for eval. Cases like \textit{eval \$str} should eventually be changed to report a line number of 0 (or more likely the line number of the original statement) for all statements with any argument involving a sub-eval. 

Possibly implement character offsets within a line.

\section{Notes}

A test has been added to the tests/trace.test. A utility \textit{trcline.tcl} is provided that the test uses to provide some measure of the accuracy of the line number tracing.

\section{Comments and Feedback}

Jeff Hobbs asked what about \textbf{interp alias}, etc.
\begin{itemize}
\item{}
Updated TIP to document cases
\end{itemize}

Jeff Hobbs notes filename storage is inefficient and finalization
\begin{itemize}
\item{}
Code changed to just increment ref count
\item{}
\textbf{TODO:} What needs to be done for \textit{Tcl\_Finalize}?
\end{itemize}

Neil Madden/Stephen Trier comment on info subcommand names \textit{line}, \textit{file} and \textit{proc} and possible future uses for \textit{line}
\begin{itemize}
\item{}
Changed to a single subcommand \textit{line} and use sub-sub commands.
\item{}
Additional subsubcommands can easily be added.
\end{itemize}

Donal Fellows writes: Is there a way to do an equivalent of \textit{\#line} directives in C
\begin{itemize}
\item{}
we can now set line number etc of a proc. Is that enough?
\end{itemize}

Donald Porter notes that changing Tcl\_Parse breaks binary compatibility
\begin{itemize}
\item{}
Move all parse variables to Interp and save/restore values on entry/exit to Tcl\_EvalEx and TclCompileScript.
\end{itemize}

Donald Porter notes that the hash table should be per Interp
\begin{itemize}
\item{}
Code changed to move hash table to Interp.
\end{itemize}

Mo DeJong notes: file path should be used in place of file name
\begin{itemize}
\item{}
TIP updated to use path where appropriate
\end{itemize}

Mo DeJong suggests to maybe use \textit{TclpObjNormalizePath(fileName)}
\begin{itemize}
\item{}
No action yet
\end{itemize}

Donal Fellows objects to no support for \textbf{proc}s in subevals and Andreas Kupries suggests defining a line number \textit{Tcl\_Token} type.
\begin{itemize}
\item{}
Add support for \textbf{proc} in subeval by addition to \textit{ResolvedCmdName}
\item{}
This is now fixed.
\end{itemize}

Donal Fellows asks if trace is disabled in the execution handler, how tracing to a sub-interp would work, and clarification on the purpose and use of trace variable \{debug\}.
\begin{itemize}
\item{}
The documentation was updated to clarify these points.
\end{itemize}

\section{Copyright}

This document has been placed in the public domain.

\textit{tgdb} and \textit{pdqi} have a BSD copyright by Peter MacDonald and PDQ Interfaces Inc.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
