\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#396: Symmetric Coroutines, Multiple Args, and yieldto}
\date{February 11, 2012}
\author{Kevin Kenny}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#396}&\textbf{Symmetric Coroutines, Multiple Args, and yieldto}\\\hline
Author&

Kevin Kenny \tipmail{kennykb}{acm.org}
\\
Created&Saturday, \(\text{11}^{\text{th}}\) February 2012\\
Type&Project\\
Tcl Version&\(8.6\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.8 $\\
Keywords&coroutine, yield, yieldto\\
Obsoletes&372\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

A new command, \textbf{yieldto}, is proposed that allows a corouting to suspend its execution and tailcall into an arbitrary command. If the new command is another coroutine's resume command, we obtain symmetric coroutines. If the new command is the \textbf{return} command, we obtain the ability for a coroutine to present an unusual status (e.g., \textit{error}, \textit{break}, \textit{continue} or \textit{return}) to its caller. The \textbf{yieldto} command also marshals the arguments as a list when the yielding coroutine is next invoked, allowing for the transmission of multiple arguments.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Rationale}

Several TIPS (at least \cite{tip372}, \cite{tip373}, \cite{tip375}, and \cite{tip383}) have been advanced to propose various improvements to the coroutine control transfer provided by the \textbf{yield} command. This TIP attempts to distill the requirements of these TIPs into an irreducible minimum for implementation in 8.6 (resolving a blocking issue for an 8.6 release).

This TIP intentionally leaves out of scope some of the more complex or controversial issues, such as enhancements to \textbf{info args} and \textbf{info default}, unusual return from a `yield` operation. and code injection into coroutines. It is believed that all of these can be added later, without introducing needless incompatibilities into the basic mechanisms of coroutine construction, invocation, and yielding.

Requirements that are thought to be essential for this TIP include:

\textbf{The ability for a coroutine invocation to implement any argument signature that an Tcl command can implement.} A coroutine invocation must be able to accept multiple arguments, and to allow for call-by-name (or rather, call-by-quasi-value-result, see below) parameter transmission.

\textbf{The ability to return an unusual status.} A coroutine invocation must be able to return an error status or another unusual status (e.g., \textit{break}, \textit{continue} or \textit{return}) to its caller, and to perform a tailcall.

\textbf{Support for symmetric coroutines.} Although it is well known that asymmetric coroutines (such as Tcl 8.6 implements today) and symmetric coroutines have equivalent power, the implementation of symmetric coroutines in a system that supports only asymmetric ones is possible only by coding a separate scheduler that allows an active coroutine to detach with a request to resume another. If symmetric coroutines are not implemented directly, it is likely that multiple incompatible schedulers will spring up in user code, greatly impeding a later unification.

\section{Proposal}

The new command:

\begin{quote}
\textbf{yieldto} \textit{cmd} ?\textit{arg1...}?
\end{quote}

shall accept one or more arguments:

\begin{quote}
\textit{cmd} --- The name of a command to invoke in place of the current coroutine invocation.
\end{quote}

\begin{quote}
\textit{arg1...} --- The arguments to pass to the given command.
\end{quote}

It shall have the following effects:
\begin{enumerate}
\item{}
The \textit{cmd} argument shall be resolved in the current coroutine's context, resulting in a command to invoke. If resolution fails, the error is presented in the coroutine's context.
\setcounter{enumi}{1}
\item{}
The current coroutine shall suspend its execution in the same way as with the \textbf{yield} command.
\setcounter{enumi}{2}
\item{}
The command that invokes the coroutine shall be placed into a state such that it will accept multiple arguments when it is next invoked, rather than the single argument demanded by \textbf{yield}. 
\setcounter{enumi}{3}
\item{}
The command and arguments shall be invoked in just the same way as if they had been called directly from the coroutine's caller. The given command replaces the coroutine invocation on the runtime stack. Data and status returned from the given command are returned to the context that invoked the coroutine.
\setcounter{enumi}{4}
\item{}
When the coroutine is resumed, any arguments passed into the coroutine command are assembled into a list and returned as the value of the \textbf{yieldto} command.
\end{enumerate}

In other words, \textbf{yieldto} means ``suspend the current coroutine and \textbf{tailcall} the given command: \textbf{yieldto} is to \textbf{yield} as \textbf{tailcall} is to \textbf{return}.'' In addition, \textbf{yieldto} causes the current coroutine to accept multiple arguments on its next invocation.

\section{Relationship with the Earlier Proposals}

\cite{tip372} proposes a \textbf{yieldm} command that allows a coroutine to yield and subsequently accept multiple arguments when resumed. The requested functionality of \cite{tip372} can be layered trivially atop this proposal: a one-line implementation of the \textbf{yieldm} command would be:

\begin{quote}
interp alias \{\} yieldm \{\} yieldto return -level 0
\end{quote}

\cite{tip373} proposed \textbf{yieldto} together with a separate \textbf{yieldset} command. The latter allowed a coroutine to designate a set of arguments and defaults. The advantage over simply passing the arguments as a list was that error messages for incorrect numbers of arguments could be generated automatically, and that \textbf{info args} and \textbf{info body} could introspect into the desired argument list. Since the error message generation can be done readily by auxiliary procedures, and the introspection is something of a nicety, this proposal defers the implementation of \textbf{yieldset}.

\cite{tip375}, which replaced \cite{tip373}, proposed a \textbf{yieldto} command that is the same as the current proposal's, except that it could transmit only a single argument when the coroutine was resumed. As such, it was incomplete as it stands.

\cite{tip387} proposed a unified syntax for all of the above TIPs. The proposed syntax was quite complex, and its only advantage over the current proposal is that is allowed for introspection using \textbf{info args} and \textbf{info body} and for automatic generation of error messages for incorrect arguments. Since these are regarded as something of a nicety, the author of the current proposal believes that their consideration can be deferred in favour of the current proposal.

The related \cite{tip383} addresses a different set of issues: injecting code into a suspended coroutine for the purpose of debugging. Its implementation can be decided on independently of this TIP.

\section{Examples:}

\subsection{1. Multiple arguments and error returns.}

Let us assume that there is a coroutine \textbf{foo} that wishes to accept at each invocation two arguments \textbf{bar} and \textbf{grill}. It therefore needs to accept multiple arguments, and to check the number of arguments, returning an error to its caller if they are incorrect. Code structured like the following can serve both purposes:
\begingroup\small\begin{verbatim}
 # presume that $value is the value to return to the last invocation
 for {set args [yieldto return -level 0 $value]} \
     {[llength $args] != 2} \
     {set args [yieldto return -level 0 -code error \
                    -errorCode {MYCORO WRONGNUMARGS} \
                    "wrong # args, should be \"foo bar grill\""]} {
         # do nothing
 }
 lassign $args bar grill
\end{verbatim}
\endgroup

\subsection{2. Symmetric coroutines.}

It may not be obvious from the foregoing discussion that the original purpose of \textbf{yieldto} was imagined to be passing of control between peer coroutines. For instance, if we assume that there are two coroutines, \textbf{producer} and \textbf{consumer}, and that calling \textbf{producer} returns a string while calling \textbf{consumer} accepts a string and returns nothing, then each coroutine may yield to the other:

\textit{Code in} \textbf{producer}:
\begingroup\small\begin{verbatim}
 yieldto consumer $string
\end{verbatim}
\endgroup

\textit{Code in} \textbf{consumer}:
\begingroup\small\begin{verbatim}
 lassign [yieldto producer] string
\end{verbatim}
\endgroup

\subsection{3. Call by name.}

It turns out that \textbf{yieldto} allows for a rough simulation of Tcl's call by name, although it requires explicit transmission of values between coroutines, rather than a simple \textbf{uplevel} operation. Let us assume that a caller invokes a coroutine with the name of an array, and that the coroutine wishes to set elements of that array. A procedure like the following will give approximately the desired result.
\begingroup\small\begin{verbatim}
 proc remote-set {varName value} {
     lassign [yieldto remote-set-worker $varName $value [info coroutine]] \
         status result
     return -code $status $result
 }
 proc remote-set-worker {varName value coro} {
     tailcall $coro \
         [catch {uplevel 1 [list set $varName $value]} result] \
         $result
 }
\end{verbatim}
\endgroup

(Proper transmission of the options dictionary from \textbf{catch} is omitted for the sake of simplicity.)

Let's look at what happens when code in the coroutine \textit{mycoro} does
\begingroup\small\begin{verbatim}
 remote-set array(key) value
\end{verbatim}
\endgroup
\begin{enumerate}
\item{}
The \textbf{remote-set} procedure yields to the command
\begingroup\small\begin{verbatim}
         remote-set-worker array(key) value mycoro
\end{verbatim}
\endgroup
\setcounter{enumi}{1}
\item{}
The \textbf{remote-set-worker}, executing in the calling coroutine, invokes
\begingroup\small\begin{verbatim}
         set array(key) value
\end{verbatim}
\endgroup

in its caller's scope, capturing the status and return value. Presuming that the variable was set successfully, It then invokes
\begingroup\small\begin{verbatim}
         mycoro 0 value
\end{verbatim}
\endgroup

as a tailcall, re-entering the \textit{mycoro} coroutine. The two-element \tiprangle{} list, \{0 value\}, becomes the return value of \textbf{yieldto}
\setcounter{enumi}{2}
\item{}
The \textbf{remote-set} procedure returns the same status and result that the \textbf{set} command returned.
\end{enumerate}

This technique actually shows how arbitrary commands may be executed in a calling coroutine. The way this is done opens some interesting possibilities for tools such as debugging interactors. For instance, a \textbf{trace} callback might invoke a debugging coroutine to interact with the user, allow the user to execute arbitrary commands to probe the state of the code being debugged, and then transparently resume at the place that the code left off, without otherwise disturbing its stack.

\section{Implementation notes}

The \textbf{yieldto return} idiom is expected to be used widely, since it is the way to present unusual status codes, and internally is used in the implementation of \textbf{yieldm}. For this reason, the implementation may contain code to optimize it specially.

\section{Acknowledgments}

The implementation of this TIP, and most of the details of its specification, are actually the brainchild of Miguel Sofer, who expresses a continued interest in seeing it implemented while being unwilling at the present time to shepherd it through the sometimes acrimonious discussions on tcl-core. To him belongs all the credit for the ideas: any errors in the specification as presented here are the author's.

Thanks are due to Donal Fellows, Joe English, Andreas Leitgeb, and Lars Hellström for making needed corrections to the initial version of this TIP.

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\begin{thebibliography}{TIP \#5}
\addcontentsline{toc}{section}{References}
\bibitem[TIP \#372]{tip372}
Colin McCormack,
\emph{Multi-argument Yield for Coroutines},
on-line at \url{http://purl.org/tcl/tip/372.html}
\bibitem[TIP \#373]{tip373}
Miguel Sofer,
\emph{Improved Yielding Support for Coroutines},
on-line at \url{http://purl.org/tcl/tip/373.html}
\bibitem[TIP \#375]{tip375}
Miguel Sofer,
\emph{Symmetric Coroutines and Yieldto},
on-line at \url{http://purl.org/tcl/tip/375.html}
\bibitem[TIP \#383]{tip383}
Alexandre Ferrieux,
Miguel Sofer,
\emph{Injecting Code into Suspended Coroutines},
on-line at \url{http://purl.org/tcl/tip/383.html}
\bibitem[TIP \#387]{tip387}
Lars Hellström,
\emph{Unified Yield Command Syntax},
on-line at \url{http://purl.org/tcl/tip/387.html}
\end{thebibliography}
\end{document}
