\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#148: Correct [list]-Quoting of the '\#' Character}
\date{August 08, 2003}
\author{Don Porter}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#148}&\textbf{Correct [list]-Quoting of the '\#' Character}\\\hline
Author&

Don Porter \tipmail{dgp}{users.sf.net}
\\
Created&Friday, \(\text{8}^{\text{th}}\) August 2003\\
Type&Project\\
Tcl Version&\(8.5\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.5 $\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes the correction of a long-standing bug in the [list]-quoting of the \textit{\#} character.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Background}

Tcl has a bug in its [list]-quoting function. The bug is recorded as Tcl Bug 489537 (\url{http://sf.net/tracker/?func=detail&aid=489537&group_id=10894&atid=110894}).

Briefly, one of the documented functions of [list] is quoting of arguments so the result can be passed to [eval] with each list element becoming exactly one word of the command to be evaluated. Consider the example script:
\begingroup\small\begin{verbatim}
 # FILE: demo.tcl
 set cmdName [lindex $argv 0]
 proc $cmdName {} {puts Success!}
 set command [list $cmdName]
 puts "Evaluating \[$command]..."
 eval $command
\end{verbatim}
\endgroup

This script expects one argument on the command line, and should write \textit{Success!} to stdout. This script works correctly for most input:
\begingroup\small\begin{verbatim}
 $ tclsh demo.tcl foo
 Evaluating [foo]...
 Success!
 $ tclsh demo.tcl "with space"
 Evaluating [{with space}]...
 Success!
\end{verbatim}
\endgroup

But it fails for any argument beginning with the \textit{\#} character:
\begingroup\small\begin{verbatim}
 $ tclsh demo.tcl #bar
 Evaluating [#bar]...
\end{verbatim}
\endgroup

This is because, contrary to the documentation, [list] does not quote the leading \textit{\#} in a manner to make the list safe for passing to [eval]. The Tcl parser sees the unquoted \textit{\#} as the start of a comment.

Starting in Tcl 8.3, optimizations for evaluation of \textit{pure lists} were added, making inconsistency due to \textit{Tcl\_ObjType} shimmering a new symptom of this bug. If we adapt the example script to remove the [puts] (so that a \textit{pure list} is maintained):
\begingroup\small\begin{verbatim}
 # FILE: demo2.tcl
 set cmdName [lindex $argv 0]
 proc $cmdName {} {puts Success!}
 set command [list $cmdName]
 eval $command
\end{verbatim}
\endgroup

We get a script that actually works with the troublesome input:
\begingroup\small\begin{verbatim}
 $ tclsh demo2.tcl #bar
 Success!
\end{verbatim}
\endgroup

This bug in [list]-quoting is present in all released versions of Tcl since and including Tcl 7.4. It may go back further.

There is no question that Tcl's behavior disagrees with its documentation on this point. I believe the documentation to be correct. From that viewpoint, this is a bug, not requiring a TIP for fixing. Because the bug has been around for so long, though, it seems prudent to make the TIP proposal, if only as fair warning to those who might have bug-dependent scripts to fix. The particular fix proposed also adds a single \textit{\#define} to Tcl's public header file.

A large number of tests have been added to the Tcl test suite in the HEAD, demonstrating this bug in several ways.

\section{Proposal}

\textit{Tcl\_ConvertCountedElement()} is modified to have the default behavior of quoting any leading \textit{\#} character in a list element. With this default quoting, any string representation of a list generated by Tcl will not begin with the \textit{\#} character, so cannot be mis-parsed as a comment.

\textit{Tcl\_ConvertCountedElement()} is also modified to recognize a new bit flag value in its \textit{flags} argument, \textit{TCL\_DONT\_QUOTE\_HASH}, which is defined in Tcl's public header file so that it may be used by extensions. When the \textit{TCL\_DONT\_QUOTE\_HASH} bit is set in the \textit{flags} argument, \textit{Tcl\_ConvertCountedElement()} will not quote the leading \textit{\#} character. Quoting of the leading \textit{\#} character is only necessary for the first element of a list. Those callers of \textit{Tcl\_ConvertCountedElement()} that can be sure they are not generating the first element of a list can pass in the \textit{TCL\_DONT\_QUOTE\_HASH} bit to produce the simplest quoting required. The behavior of the \textit{TCL\_DONT\_QUOTE\_HASH} bit flag is added to the documentation. The \textit{Tcl\_ConvertElement()} routine is similarly modified (trivially, since it is just a wrapper).

All callers of \textit{Tcl\_ConvertCountedElement()} in the Tcl source code are modified to use the \textit{TCL\_DONT\_QUOTE\_HASH} flag as appropriate, so that Tcl continues to generate as simple string representations of lists as possible that do not suffer from Bug 489537.

\section{Prototype}

A patch implementing this proposal is attached to Tcl Bug 489537 at SourceForge.

\section{Compatibility}

After acceptance of this patch, the string representation of some lists will change, though as little as possible while still fixing the bug. Scripts that perform string comparisons on lists may see different results. Notably, a test in a test suite that has a test body that generates a list, and then has an expected result as a string may see new test failures. The minimal quoting changes should keep this incompatibility to a minimum, but it may happen.

Notably there are no such compatibility problems in either the Tcl or Tk test suites. Any such incompatibility in other test suites can be easily remedied by using [list] to generate the expected result.

\section{Scope}

It has been observed by some Tcl users that [list] is used for two conceptually distinct purposes. First, adding quoting to list elements as required, so that element boundaries can be re-discovered from the string representation. Second, adding quoting so that the string representation can be passed to [eval] with the original element boundaries becoming the argument boundaries in the evaluation. One can imagine a Tcl where these two functions were separated. However, this TIP does not propose such a separation, and further arguments on that point are out of scope, and should be considered in another TIP, if at all.

\section{Acknowledgements}

The author acknowledges the discovery of this bug by Joe English, analysis by Donal Fellows, and a first draft patch from Miguel Sofer.

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
