\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#90: Enable [return -code] in Control Structure Procs}
\date{March 15, 2002}
\author{Don Porter, Donal K. Fellows}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#90}&\textbf{Enable [return -code] in Control Structure Procs}\\\hline
Author&

Don Porter \tipmail{dgp}{users.sf.net}
\par
Donal K. Fellows \tipmail{fellowsd}{cs.man.ac.uk}
\\
Created&Friday, \(\text{15}^{\text{th}}\) March 2002\\
Type&Project\\
Tcl Version&\(8.5\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.39 $\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP analyzes existing limitations on the coding of control structure commands as \textit{proc}s, and presents expanded forms of \textit{catch} and \textit{return} to remove those limitations.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Background}

It is a distinguishing feature of Tcl that everything is a command, including control structure functionality that in many other languages are part of the language itself, such as \textit{if}, \textit{for}, and \textit{switch}. The command interface of Tcl, including both a return code and a result, allows extensions to create their own control structure commands.

Control structure commands have the feature that one or more of their arguments is a script, often called a \textit{body}, meant to be evaluated in the caller's context. The control structure command exists to control whether, when, in what context, or how many times that script is evaluated. When the body is evaluated, however, it is intended to behave as if it were interpreted directly in the place of the control structure command.

The built-in commands of Tcl provide the ability for scripts themselves to define new commands. Notably, the \textit{proc} command makes this possible. In addition, other commands such as \textit{catch}, \textit{return}, \textit{uplevel}, and \textit{upvar} offer enough control and access to the caller's context that it is possible to create new control structure commands for Tcl, entirely at the script level.

Almost.

There is one limitation that separates control structure commands created by \textit{proc} from those created in C by a direct call to \textit{Tcl\_Create(Obj)Command}. It is most easily seen in the following example that compares the built-in command \textit{while} to the command \textit{control::do} created by \textit{proc} in the control package of tcllib.
\begingroup\small\begin{verbatim}
  % package require control
  % proc a {} {while 1 {return -code error}}
  % proc b {} {control::do {return -code error} while 1}
  % catch a
  1
  % catch b
  0
\end{verbatim}
\endgroup

The control structure command \textit{control::do} fails to evaluate \textit{return -code error} in such a way that it acts the same as if \textit{return -code error} was evaluated directly within proc \textit{b}.

\section{Analysis}

There are two deficiencies in Tcl's built-in commands that lead to this incapacity in control structure commands defined by \textit{proc}.

First, \textit{catch} is not able to capture the information. Consider:
\begingroup\small\begin{verbatim}
   %  set code [catch {
          return -code error -errorinfo foo -errorcode bar baz
      } message]
\end{verbatim}
\endgroup

After evaluation, \textit{code} contains ``2'' (\textit{TCL\_RETURN}), and \textit{message} contains ``baz'', but the other values are locked away in internal fields of the \textit{Tcl\_Interp} structure as \textit{interp-\tiprangle{}returnCode}, \textit{interp-\tiprangle{}errorCode}, and \textit{interp-\tiprangle{}errorInfo}. The ``-errorcode'' and ``-errorinfo'' values will be copied to the global variables ``::errorCode'' and ``::errorInfo'', respectively, but there will be no way at the script level to get at the \textit{interp-\tiprangle{}returnCode} value which was the value of the original ``-code'' option.

Second, even if the information were available, there is no built-in command in Tcl that can be evaluated within the body of a proc to make the proc itself act as if it were the command \textit{return -code}. Stated another way, it is not possible to create a command with \textit{proc} that behaves exactly the same as \textit{return -code}. Because of that, it is also not possible to create a command with \textit{proc} that behaves exactly the same as \textit{while}, \textit{if}, etc. --- any command that evaluates any of its arguments as a script in the caller's context.

This is a curious, and likely unintentional, limitation. Tcl goes to great lengths to be sure I can create my own \textit{break} replacement with \textit{proc}.
\begingroup\small\begin{verbatim}
 proc myBreak {} {return -code break}
\end{verbatim}
\endgroup

It would be a welcome completion of Tcl's set of built-in commands to be able to create a replacement for every one of them using \textit{proc}.

\section{Specification}

The \textit{return} command shall have syntax:
\begingroup\small\begin{verbatim}
 return ?option value ...? ?result?
\end{verbatim}
\endgroup

There can be any number of \textit{option value} pairs, and any value at all is acceptable for an \textit{option} argument. The legal values of a \textit{value} argument are limited for some \textit{option}s, as follows:

\begin{quote}
the \textit{value} after a ``-code'' must be either an integer (32-bit only), or one of the strings, ``ok'', ``error'', ``return'', ``break'', or ``continue'', just as in the 8.4 spec for \textit{return}. The default \textit{value} for the ``-code'' option is ``0''.
\end{quote}

\begin{quote}
the \textit{value} after a ``-level'' must be a non-negative integer. The default \textit{value} for the ``-level'' option is ``1''.
\end{quote}

\begin{quote}
the \textit{value} after a ``-options'' must be a dictionary (\cite{tip111}). The default \textit{value} for the ``-options'' option is an empty dictionary.
\end{quote}

The keys and values in the dictionary \textit{value} of the ``-options'' option are pulled out and treated as additional \textit{option value} arguments to the \textit{return} command. Note that this ``-options'' option for option expansion is offered only because Tcl itself has no syntax for argument expansion, as observed many, many times before (for example, \cite{tip103}).

The \textit{result} argument, if any, is stored in the interp as the result of the \textit{return} command. In default operation, this becomes the result of the procedure in which the \textit{return} command is evaluated.

The return code of the \textit{return} command is determined by the \textit{value}s of the ``-code'' and ``-level'' options. If the \textit{value} of the ``-level'' option is non-zero, then the return code of \textit{return} is TCL\_RETURN. If the \textit{value} of the ``-level'' option is ``0'', then the return code of \textit{return} is the \textit{value} of the ``-code'' option, translated from string, as needed. In this way,
\begingroup\small\begin{verbatim}
 return -level 0 -code break
\end{verbatim}
\endgroup

is a synonym for
\begingroup\small\begin{verbatim}
 break
\end{verbatim}
\endgroup

while
\begingroup\small\begin{verbatim}
 return -code break
\end{verbatim}
\endgroup

spelled out with defaults filled in as:
\begingroup\small\begin{verbatim}
 return -level 1 -code break
\end{verbatim}
\endgroup

continues to function as before, causing the procedure in which the \textit{return} is evaluated to return the TCL\_BREAK return code.

All \textit{option value} arguments to \textit{return} are stored in a return options dictionary kept in the interp, just as the \textit{result} argument gets stored in the result of the interp.

The TclUpdateReturnInfo() function is modified, so that each level of procedure returning decrements the value of the ``-level'' key in the return options dictionary. When the value of the ``-level'' key reaches ``0'', the return code from the current procedure will be the value of the ``-code'' key in the return options dictionary. Otherwise, the return code of the current procedure will be TCL\_RETURN.

In this way,
\begingroup\small\begin{verbatim}
 return -level 2 -code ok
\end{verbatim}
\endgroup

is equivalent to
\begingroup\small\begin{verbatim}
 return -code return
\end{verbatim}
\endgroup

and should (absent some intervening \textit{catch}) cause a normal return to the caller's caller. Likewise,
\begingroup\small\begin{verbatim}
 return -level 3 -code ok
\end{verbatim}
\endgroup

would cause a normal return to the caller's caller's caller (again absent an intervening \textit{catch}), something that can't currently be accomplished.

The \textit{catch} command shall have syntax:
\begingroup\small\begin{verbatim}
 catch script ?resultVar? ?optionsVar?
\end{verbatim}
\endgroup

The new argument \textit{optionsVar}, if present, will be the name of a variable in which a dictionary of return options should be stored. The return options stored in that dictionary are exactly those needed so that the evaluation of
\begingroup\small\begin{verbatim}
 catch $script result options
 return -options $options $result
\end{verbatim}
\endgroup

is completely indistinguishable (except for the existence and values of variables ``result'' and ``options'') from the direct evaluation of \textit{\$script} by the interpreter. In particular, any values of the ``::errorCode'' and ``::errorInfo'' variables are the same as if there were never a \textit{catch} in the first place.

In addition, when the result of \textit{catch} is TCL\_ERROR, the value in the \textit{errorLine} field of the \textit{Interp} struct will be stored as the value of the ``-errorline'' key in the return options dictionary.

This specification may seem a bit complex, but it makes possible very simple solutions to the problems posed above.

\section{Examples}

First lets revisit the analysis:
\begingroup\small\begin{verbatim}
   %  set code [catch {
          return -code error -errorinfo foo -errorcode bar baz
      } message options]
\end{verbatim}
\endgroup

After evaluation, \textit{code} contains ``2'' (\textit{TCL\_RETURN}), \textit{message} contains ``baz'', and now \textit{options} contains:
\begingroup\small\begin{verbatim}
 -errorcode bar -errorinfo foo -code 1 -level 1
\end{verbatim}
\endgroup

So, the \textit{options} variable now contains the information that was previously inaccessible. We can now
\begingroup\small\begin{verbatim}
 return -options $options $message
\end{verbatim}
\endgroup

to get the same results as if the \textit{catch} had never been there in the first place.

In 8.4 Tcl, it is not possible to implement a replacement for the \textit{return} command as a proc. After this proposal, such a replacement is:
\begingroup\small\begin{verbatim}
 proc myReturn args {
     set result ""
     if {[llength $args] % 2} {
         set result [lindex $args end]
         set args [lrange $args 0 end-1]
     }
     set options [eval [list dict create -level 1] $args]
     dict incr options -level
     return -options $options $result
 }
\end{verbatim}
\endgroup

In every way \textit{myReturn} should be an equivalent to \textit{return}.

The new ability to exactly reproduce stack traces makes a \textit{catch} of large scripts more attractive. For example, a procedure that allocates some resource, then performs operations, and finally frees the resource before returning. In order to be sure the resource is freed, we must \textit{catch} any errors that might cause the procedure to return before the freeing of the resource. The solution looks like:
\begingroup\small\begin{verbatim}
 proc doSomething {} {
     set resource [allocate]
     catch {
          # Arbitrarily long script of operations
     } result options
     deallocate $resource
     return -options $options $result
 }
\end{verbatim}
\endgroup

With that structure, we are confident the resource is always freed, but any error or exception will be presented to the caller exactly as if it had never been caught in the first place.

Here are two examples of how to use the new features in a control structure proc. The essence of a control structure command is its ability to evaluate a script in the caller's context, preserving the illusion that no additional stack frame was ever used. So, a proc replacement for \textit{eval} illustrates the technique.

The first approach assumes one knows the internal details of how the \textit{uplevel} command adds to the stack trace. This is straightforward, but will require a rewrite if \textit{uplevel} ever changes how it manipulates the stack trace.
\begingroup\small\begin{verbatim}
 proc myEval script {
     if {[catch {uplevel 1 $script} result options] == 1} {
         set stack [dict get $options -errorinfo]
         regsub {\s+invoked from within\s+"uplevel 1 \$script"$} $stack {} stack
         regsub {\("uplevel" body line (\d+)\)$} $stack [subst -nobackslashes \
                 {("[lindex [info level 0] 0]" body line \1)}] stack
         dict set options -errorinfo $stack
     }
     dict incr options -level
     return -options $options $result
 }
\end{verbatim}
\endgroup

A second, more robust solution is possible, but requires a bit more context gymnastics.
\begingroup\small\begin{verbatim}
 namespace eval control {
     proc eval script {
         variable result
         variable options
         set code [uplevel 1 \
                 [list ::catch $script [namespace which -variable result] \
                         [namespace which -variable options]]]
         if {$code == 1} {
             set line [dict get $options -errorline]
             dict append options -errorinfo \
                     "\n    (\"[lindex [info level 0] 0]\" body line $line)"
         }
         dict incr options -level
         return -options $options $result
     }
 }
\end{verbatim}
\endgroup

Note that in the second solution we did not have to strip away the contributions of \textit{uplevel} to the stack trace, because we captured the stack trace before \textit{uplevel} added anything. Then we could add our own information (drawing in part on the new ``-errorline'' value available to us now at the script level).

We confirm that either approach solves the original problem:
\begingroup\small\begin{verbatim}
 % proc a {} {eval {return -code error}}
 % proc b {} {myEval {return -code error}}
 % proc c {} {control::eval {return -code error}}
 % catch a
 1
 % catch b
 1
 % catch c
 1
\end{verbatim}
\endgroup

Finally, the new features make possible a utility command that can be of use to people making simple control structure commands, or doing simple wrapping, where there is no need to augment the stack trace, or to treat any return codes in a special way:
\begingroup\small\begin{verbatim}
 namespace eval control {
     proc ascaller script {
         if {[info level] < 2} {
             return -code error \
                     "[lindex [info level 0] 0] called outside a proc"
         }
         variable result
         variable options
         set code [uplevel 2 \
                 [list ::catch $script   [namespace which -variable result] \
                                         [namespace which -variable options]]]
         if {$code == 0} {
             return $result
         }
         dict incr options -level 2
         return -options $options $result
     }
 }
\end{verbatim}
\endgroup

Within a proc, \textit{ascaller \$script} will take care of all aspects of evaluating \textit{\$script} in the caller context, and exiting as appropriate for all non-TCL\_OK return codes.

\section{Extensibility}

The \textit{return -code} command has always accepted any integer value as a valid argument, allowing package and application authors to define their own new return codes as needed by their own control structure commands. Now that \textit{return} will accept any \textit{option} argument, and \textit{catch} can capture all \textit{option value} argument pairs passed to the caught \textit{return} command, package and application authors now have the ability to augment their custom return codes with additional data. Some prefix convention should be established to avoid key name conflicts in the return options dictionary.

\section{Potential Concerns}

Reviewers of drafts of this TIP wondered whether the new ``-level'' option to \textit{return} raised the possibility of trouble with an attempt to return more levels than beyond the top of the call stack. 

It should be understood that \textit{return -level N} does not take any shortcut past the intervening levels. Each level of the call stack gets a TCL\_RETURN return code, and a ``-level'' value, dropping by one each step up the stack. Any level in the stack might choose to \textit{catch} the TCL\_RETURN and treat it as it wishes. This is exactly the way the existing \textit{return -code return} is handled. Normally, it would cause a normal return to the caller's caller, but if the caller chooses to 'catch' it, then the caller has control.

At the toplevel we run out of callers. Then the question becomes how is a TCL\_RETURN code at toplevel handled?
\begingroup\small\begin{verbatim}
 % return -level 0       ;# same as a TCL_OK at toplevel
 % return -level 1       ;# same as [return]
 % return -level 2       ;# same as [return -code return]
 command returned bad code: 2
\end{verbatim}
\endgroup

From the C level, \textit{Tcl\_AllowExceptions()} can be used to modify this toplevel behavior.

The following proc will produce the same results as above, but from any level in the call stack (absent an intervening \textit{catch}):
\begingroup\small\begin{verbatim}
 % proc escape level {
       set x [info level]
       incr x $level
       return -level $x
   }
 % escape 0
 % escape 1
 % escape 2
 command returned bad code: 2
\end{verbatim}
\endgroup

Another concern was whether this proposal gave slave interpreters any new powers over their masters. The return code from evaluation of an untrusted script in a slave interpreter should always be wrapped in a \textit{catch} already, lest a TCL\_ERROR in the script blow the stack. Given that, the only thing this proposal does is give the \textit{catch} command more information to use to decide how to handle the misbehaving script.

\section{Compatibility}

It is the author's belief that this proposal is completely compatible with prior Tcl 8.X releases. Any error-free script that ran before, should continue to run with the same results. At the C level, only internal changes are made, and no new interfaces are defined. Any extension or embedding C program that sticks to the public stubs interface should see no visible change. 

\section{Prototype}

This proposal is implemented by Tcl Patch 531640 at SourceForge.

The prototype covers all described functionality, but might be further improved with more substantial bytecompiling of [return].

\section{Future considerations}

The main reason the global variables \textit{::errorInfo} and \textit{::errorCode} exist is to give the script level access to stack and error code information following the \textit{catch} of a script that raises an error. After this proposal, the \textit{catch} command itself provides access to that information, so the global variables are not required. One can imagine deprecating them, asking users of Tcl 8.5 to stop writing code that accesses them. They could still have apparent existence, to satisfy the needs of scripts written for earlier Tcl 8.X releases, by means of read traces. In time, Tcl 9 could either continue the read trace scheme, or not provide these global variables at all.

One part of Tcl itself that currently makes use of the \textit{::errorCode} and \textit{::errorInfo} variables is the \textit{bgerror} command. Currently, \textit{bgerror} accepts exactly one argument, the error message. To make use of stack or error code information, \textit{bgerror} must retrieve them from the global variables. The proper values of these global variables are re-set by \textit{Tcl\_BackgroundError()} prior to evaluation of \textit{bgerror}.

As an alternative, \textit{Tcl\_BackgroundError()} could first attempt to call \textit{bgerror} with \textit{two} arguments, first the message, then a dictionary of options. If that call returned TCL\_ERROR, then a second attempt could be made with a single message argument. In that way, cleaner \textit{bgerror} commands that get all data from arguments could be supported, while still keeping support for those \textit{bgerror} commands that were defined for single argument use.

It has been noted several times that the processing of the value of \textit{::errorInfo} is rather difficult because it is an arbitrary string with no documented structure. A different, more structured way of representing stack trace information would be an improvement. This proposal does not propose an alternative, but because it offers an extensible dictionary for storing arbitrary return options data, it does provide an infrastructure where such approaches might be tried out.

\section{Acknowledgments}

This proposal is a synthesis of ideas from many sources. As best I can recall, major contributions came from Joe English, Andreas Leitgeb, Reinhard Max, and Kevin Kenny. If you like the idea, give them some credit; it you don't, blame me for combining the ideas badly.

\section{See also}

Documentation for tcllib's control package: \url{http://tcllib.sf.net/doc/control.html}

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\begin{thebibliography}{TIP \#2}
\addcontentsline{toc}{section}{References}
\bibitem[TIP \#103]{tip103}
Peter Spjuth,
Donal K. Fellows,
Andreas Leitgeb,
\emph{Argument Expansion Command},
on-line at \url{http://purl.org/tcl/tip/103.html}
\bibitem[TIP \#111]{tip111}
Donal K. Fellows,
David S. Cargo,
\emph{Dictionary Values and Manipulators},
on-line at \url{http://purl.org/tcl/tip/111.html}
\end{thebibliography}
\end{document}
