\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#304: A Standalone [chan pipe] Primitive for Advanced Child IPC}
\date{February 07, 2007}
\author{Alexandre Ferrieux}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#304}&\textbf{A Standalone [chan pipe] Primitive for Advanced Child IPC}\\\hline
Author&

Alexandre Ferrieux \tipmail{alexandre.ferrieux}{gmail.com}
\\
Created&Wednesday, \(\text{7}^{\text{th}}\) February 2007\\
Type&Project\\
Tcl Version&\(8.6\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.15 $\\
Keywords&Tcl, exec, process, subprocess, pipeline, channel\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

Currently, it is not easy to get both (separate) dataflows from the stdout and stderr of a child. BLT's \textbf{bgexec} does this in an extension, and could be added to the core. But the point of this TIP is to show that a much smaller code addition can provide a lower-level primitive with much more potential than \textbf{bgexec}'s: a \textit{standalone pipe} creation tool like TclX's \textbf{pipe} command.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Background}

Getting back both stdout and stderr from a child has long been an FAQ on \url{news:comp.lang.tcl}, to the point that \textbf{bgexec} has been offered in an extension, BLT, whose main job is very remote from IPC. Now this has been a problem for many, who didn't want to have the problems of distributing a script depending on an extension. Moreover, \textbf{bgexec} does not scale up, in that it cannot bring back the separate stderrs of all four children in:
\begingroup\small\begin{verbatim}
	set ff [open "|a | b | c | d" r]
\end{verbatim}
\endgroup

A popular workaround for script-only purists is to spawn an external ``pump'' like \textit{cat} in an [\textbf{open} ... r+], and redirect the wanted stderr to the write side of the pump. Its output can then be monitored through the read side:
\begingroup\small\begin{verbatim}
	set pump [open "|cat" r+]
	set f1 [open "|cmd args 2>@ $pump" r]
	fileevent $f1 readable got_stdout
	fileevent $pump readable got_stderr
\end{verbatim}
\endgroup

Now this is all but elegant of course, difficult to deploy on Windows (where you need an extra cat.exe), and not especially efficient since the ``pump'' consumes context switches and memory bandwidth only to emulate a \textit{single} OS pipe when Tcl is forced to create \textit{two} of them via [\textbf{open} ... r+].

For this latter performance issue, a better alternative is a named pipe. But it is even harder to create on Windows, and it is a nightmare to handle its lifecycle properly (it doesn't die automagically with the creating process; blocks on open() if other side is not ready).

\section{Proposed Change}

All this points to the obvious solution: steal TclX's \textbf{pipe} command, which wraps the OS's pipe()/CreatePipe() syscall, yielding two Tcl channels wrapping the read and write side of the underlying pipe.

TclX's \textbf{pipe} command allows two syntaxes:
\begingroup\small\begin{verbatim}
	pipe pr pw
	foreach {pr pw} [pipe] break
\end{verbatim}
\endgroup

Its application to following stderr is straightfoward:
\begingroup\small\begin{verbatim}
	pipe pr pw
	set f1 [open "|cmd args 2>@ $pw" r]
	fileevent $f1 readable got_stdout
	fileevent $pr readable got_stderr
\end{verbatim}
\endgroup

\subsection{Specification}

The basic functionality is to return a pair of channels, so the script-level API does just that:

\begin{quote}
\textbf{chan pipe}
\end{quote}

Create a pipe, and return a list of two channels wrapping either side of the pipe. The first element is the side opened for reading, the second for writing:
\begingroup\small\begin{verbatim}
	lassign [chan pipe] pr pw
\end{verbatim}
\endgroup

Subsequently, anything written to \$pw will be readable on \$pr. We purposefully drop the other [pipe pr pw] syntax from TclX, for the sake of minimality.

\subsection{C Interface}

A counterpart to \textbf{chan pipe} is added to Tcl's C API:

\begin{quote}
int \textbf{Tcl\_CreatePipe}(Tcl\_Interp *\textit{interp}, Tcl\_Channel *\textit{rchan}, Tcl\_Channel *\textit{wchan}, int \textit{flags})
\end{quote}

The \textit{interp} argument is used for reporting errors and registering the channels created, the \textit{rchan} and \textit{wchan} arguments point to variables into which to place the channels for each end of the pipeline, and the \textit{flags} argument is reserved for future expansion.

\section{Discussion}

It has been in TclX for years, so it is fireproof. The code can be directly copied from there, and is dead simple, just calling Tcl\_MakeFileChannel and Tcl\_RegisterChannel on both fds/handles. Just the low-level syscall changes between unix and windows.

\subsection{Additional Benefits}

It should also be noted that this primitive allows to deprecate the use of [\textbf{open} ``\tipbar{}...''] and \textbf{pid} themselves !
\begingroup\small\begin{verbatim}
	lassign [chan pipe] p1r p1w
	lassign [chan pipe] p2r p2w
	set pids [exec cmd $args >@ $p1w 2>@ $p2w &]
	fileevent $p1r readable got_stdout
	fileevent $p2r readable got_stderr
\end{verbatim}
\endgroup

Taking this idea one step further, one can even deprecate the ``\tipbar{}'' in \textbf{exec}, with:
\begingroup\small\begin{verbatim}
	exec a | b | c &
\end{verbatim}
\endgroup

being rewritten as:
\begingroup\small\begin{verbatim}
	exec a >@ $p1w &
	exec b <@ $p1r >@ $p2w &
	exec c <@ $p2r &
\end{verbatim}
\endgroup

(I'm not crazy about actually deprecating ol'good and concise idioms in favour of the tedious lines above; however as an orthogonality worshipper I like to think it would be theoretically possible to reimplement them in pure Tcl thanks to the added power of one tiny extra primitive)

A more serious advantage, is that it allows the ``half-close'' of a command pipeline (detailed in \cite{tip301} which the current TIP makes partly obsolete). Indeed
\begingroup\small\begin{verbatim}
	lassign [chan pipe] pr pw
	lassign [chan pipe] qr qw
	exec a <@$pr >@$qw &
\end{verbatim}
\endgroup

allows to close just the read side with
\begingroup\small\begin{verbatim}
	close $pw
\end{verbatim}
\endgroup

and still read all the output of the command up to its EOF:
\begingroup\small\begin{verbatim}
	while {[gets $qr line]>=0} {...}
\end{verbatim}
\endgroup

which is notoriously impossible with
\begingroup\small\begin{verbatim}
	set pq [open |a r+]
\end{verbatim}
\endgroup

\section{Directions for Future Work}

On unix, there's no reason to limit the use of unnamed pipes to stdout and stderr. An arbitrary pipe topology can be set up between several childs with descriptors 3,4,5... For this we could imagine a natural extension of \textbf{exec}'s \textbf{2\tiprangle{}@} to \textbf{3\tiprangle{}@}, \textbf{4\tiplangle{}@}, etc.
\begingroup\small\begin{verbatim}
	exec foo >@ $p 3>@ $q 4<@ $r &
	exec bar <@ $q >@ $r 4<@ $p &
\end{verbatim}
\endgroup

Of course, such uses are extreme, but useful in complicated IPC setups to achieve much better performance (through direct point-to-point pipes) than would a simpler ``star'' or ``blackboard'' topology (where all children write back to a central message routing process).

\section{Reference Implementation}

A patch against the Tcl HEAD (8.6) is located at \url{http://sf.net/tracker/?func=detail&aid=1978495&group_id=10894&atid=310894}

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\begin{thebibliography}{TIP \#1}
\addcontentsline{toc}{section}{References}
\bibitem[TIP \#301]{tip301}
Alexandre Ferrieux,
\emph{Split Bidirectional Channels For Half-Close},
on-line at \url{http://purl.org/tcl/tip/301.html}
\end{thebibliography}
\end{document}
