\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#424: Improving [exec]}
\date{July 07, 2013}
\author{Alexandre Ferrieux}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#424}&\textbf{Improving [exec]}\\\hline
Author&

Alexandre Ferrieux \tipmail{alexandre.ferrieux}{gmail.com}
\\
Created&Sunday, \(\text{7}^{\text{th}}\) July 2013\\
Type&Project\\
Tcl Version&\(8.7\)\\
State&Draft\\
Vote&Pending\\
Version&\tipversion$Revision: 1.8 $\\
Keywords&Tcl, subprocess, execution\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This extension overcomes day-1 limitations of [\textbf{exec}]'s syntax, allowing for unconstrained arguments to commands, and opening the path to more exotic redirections.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Summary Change}

Replace:
\begingroup\small\begin{verbatim}
   exec foo bar baz > file
\end{verbatim}
\endgroup

With:
\begingroup\small\begin{verbatim}
   exec | {foo bar baz} > file
\end{verbatim}
\endgroup

\section{Rationale}

For decades people have rightfully complained about the stubborn limitation of \textbf{exec} that prevents it from using commands or args resembling a redirection. It's not just Quoting Hell; it is simply impossible to spawn the equivalent of Bourne Shell's ``echo \(\backslash\)\tiprangle{}'' from pure Tcl (i.e., without resorting to another shell).

The reason (excuse?) for this is an unfortunate design choice: stick as closely as possible to the Bourne Shell's syntax, which indeed seamlessly intertwines commands, arguments, and redirects. This is unfortunate, because it overlooks a key difference between the two shells:
\begin{itemize}
\item{}
In Bourne Shell, since everything is about spawning commands, redirects are expected everywhere; hence their quoting is ubiquitous, and part of the language.
\item{}
In Tcl, spawning processes is only a tiny part of the story. Consequently, redirect chars (\tiplangle{}\tiprangle{}\tipbar{}) are not special, and deserve no core-language quoting rules.
\end{itemize}

In this situation, it would have been possible to add an \textbf{exec}-specific layer of quoting, just for these characters. But as usual, the quoting char itself (typically ``\textbf{\(\backslash\)}'') would have itself needed quoting (``\textbf{\(\backslash\)\(\backslash\)}''), which would have overburdened the backslash density of all but the simplest pipelines...

More importantly, the realization that this was Really Wrong came fairly late in Tcl's life; or at least late enough to consider any incompatible fix out of the question.

So \textbf{exec} can be \textit{extended}, not \textit{fixed}.

A few such extensions have been suggested over the years, but none reached critical mass. A possible interpretation of this is that they were considered too ``disruptive'' --- while necessary only for a corner case.

The current proposal addresses all the above concerns. Here are its design goals by decreasing importance:
\begin{enumerate}
\item{}
Current \textbf{exec}'s unescapable warts should disappear

(Yeah, take care of that corner case.)
\setcounter{enumi}{1}
\item{}
Current \textbf{exec}'s mapping to \textbf{open \tipbar{}} should be carried over

(This part of \textbf{exec}'s design was Good)
\setcounter{enumi}{2}
\item{}
Simple pipelines should give easy-to-read lines (like current \textbf{exec})

(No disruption, Ma'am)
\setcounter{enumi}{3}
\item{}
Shell-ish advanced redirections like ``\textbf{3\tiprangle{}\&5}'' should be supported
\begin{description}
\item[(Not just the corner case]
you get a free lunch too)
\end{description}
\end{enumerate}

\section{Definition}
\begin{itemize}
\item{}
Extend \textbf{exec} ``from its error space'', by reserving a single pipe character passed as its first argument:
\begingroup\small\begin{verbatim}
       exec | ...    ;# activates the new syntax
       open "|| ..." ;# same in [open]
\end{verbatim}
\endgroup
\item{}
Once the new syntax is unambiguously introduced, parse the rest as follows:
\begingroup\small\begin{verbatim}
       exec | $cmd1 {*}$redirs1 | ... | $cmdN {*}$redirsN ?&?
       open "|[list | $cmd1 {*}$redirs1 | ... ]"
\end{verbatim}
\endgroup

where:
\begin{itemize}
\item{}
\textbf{\$cmd}\textit{K} and \textbf{\$redirs}\textit{K} are lists
\item{}
\textbf{\$cmd}\textit{K} is a simple command-and-args, no extras
\item{}
\textbf{\$redirs}\textit{K} is a list of current exec redirection operators
\end{itemize}
\end{itemize}

Examples:
\begingroup\small\begin{verbatim}
      exec | {echo >} ;# this returns ">"
      exec | {cmd "<funny>xml</funny>"} 2>@ $ch < /dev/null | {cmd2 arg} >&2
\end{verbatim}
\endgroup

Goals reached:
\begin{enumerate}
\item{}
Unescapable warts are gone because the \textbf{\$cmd} vs \textbf{\$redir} status is positional, not content-based: each command-and args is a separate sublist, with no in-band encoding of redirections.
\setcounter{enumi}{1}
\item{}
The above mapping is consistent with the existing \textbf{open \tipbar{}[list foo bar]} logic. It respects the invariant saying, for \textbf{open \tipbar{}}, that \textbf{[string range \$openarg 1 end]} is always the list that would be passed, expanded, to \textbf{exec}. And it is handy to type \textbf{open ``\tipbar{}\tipbar{} \{foo \tiprangle{}\} \tiprangle{} file''}
\setcounter{enumi}{2}
\item{}
Simple pipelines are simple.

\textbf{exec \tipbar{} \$cmd1 \tipbar{} \$cmd2 \tipbar{} \$cmd3 \tiprangle{} file}
\setcounter{enumi}{3}
\item{}
Advanced redirections are imaginable since the redirection subsyntax now lives on its own. For example, with a putative ``NUMBER\tiprangle{}@'' family of operators, one could define a nonlinear pipe graph:
\begingroup\small\begin{verbatim}
      lassign [chan pipe] pr pw
      exec | {demuxer ...} 3>@ $pw | {filter ...} | {muxer ...} 3<@ $pr
\end{verbatim}
\endgroup
\end{enumerate}

 The definition of these advanced operators will be hosted by another TIP.

\section{TL;DR}

This very conservative syntax, in addition to preserving the overall style and density of current \textbf{exec}, overcomes all the limitations and reaches Bourne Shell power.

Moreover, it leverages the existing internals, so a nearly free side-effect is that it works with \textbf{pid} and \textbf{close} just like current \textbf{exec} does.

\section{Rejected Alternatives}
\begin{itemize}
\item{}
Replace the leading ``\textbf{\tipbar{}}'' in \textbf{exec \tipbar{}} by \textbf{-{}-extended}
\item{}
Use a different toplevel command name.

\textbf{exec2}...
\end{itemize}

\section{Reference Implementation}

Branch ``tip-improve-exec'' on core.tcl.tk holds the implementation.

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
