\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#112: Ensembles are Namespaces are Commands}
\date{October 10, 2002}
\author{Donal K. Fellows}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#112}&\textbf{Ensembles are Namespaces are Commands}\\\hline
Author&

Donal K. Fellows \tipmail{donal.k.fellows}{man.ac.uk}
\\
Created&Thursday, \(\text{10}^{\text{th}}\) October 2002\\
Type&Project\\
Tcl Version&\(8.5\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 2.28 $\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes unifying the concept of ensembles (from [Incr Tcl]) with namespaces and commands. It also adds control of command rewriting to allow for more efficient support for object systems like Snit.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Rationale}

Tcl's subcommand-style command collections (e.g. \textit{array}, \textit{info}, \textit{string}, \textit{interp}, etc.) are a very intuitive and popular way of structuring collections of related commands. However, it is quite awkward to write Tcl code that behaves that way. Users of [Incr Tcl] have access to ensembles which provide that, but it would be a very useful feature for many other uses too.

At the same time, it is becoming clear that many applications want to commonly refer to commands inside other namespaces directly (instead of through the [namespace import] mechanism) but the syntax for doing this is verbose and not as elegant as it might be.

I believe that the same solution can address these two problems in one go, and make the language stronger and more usable for it.

Furthermore, by giving the programmer control over the mapping from the ensemble subcommands to their implementing commands, we can build a simple class system on the cheap since we can, in effect, import commands from elsewhere into the ensemble. By extending the mapping so that it allows the specification of not just the implementing command, but also some leading arguments to that command (similar to what you can do with the [interp alias] mechanism) this becomes a very powerful mechanism indeed.

Finally, a sophisticated extra capability is the addition of an unknown subcommand callback to allow the creator of the ensemble to specify a customized strategy for handling subcommands that are not recognized by the ensemble machinery. This allows an ensemble to be dynamically updated to include those subcommands that the user asks for, which is a potential route for doing things like using an ensemble as a wrapper round a Tk widget.

\section{Proposed Change}

I propose to add a new subcommand to the [namespace] command, \textit{ensemble}, that creates and manipulates the ensemble command for a namespace. Each namespace may have any number of ensemble commands associated with it, with the default name of an ensemble command being the fully-qualified name of the namespace itself, though it will be legal to rename ensemble commands (anyone wanting to track such events should use the [trace] command.) Tcl will not create an ensemble command for any namespace by default.

The [namespace ensemble] command will have the following subcommands:
\begin{description}
\item[create]
For creating a new ensemble for the \textit{current} namespace. The command takes a list of option value pairs (as defined below) to set the ensemble up with. As stated above, the default command name is exactly the name of the namespace, but any other name may be supplied (via the -command option); if it does not start with a namespace separator, the name will be taken as being relative to the current namespace. If another command with the same name exists, it is deleted as part of the creation of the ensemble.
\item[configure]
For reading and writing the configuration of a particular ensemble. The command takes an argument specifying the name of the ensemble to configure, and then works with either no extra arguments (when it retrieves the entire ensemble configuration), one extra argument (the name of the option to retrieve), or a list of option value pairs to configure the ensemble with. This command returns an error if applied to anything that is not an ensemble.
\item[exists]
This subcommand (which takes a single argument) tests whether a command (with the given name) exists and is an ensemble. This command only returns an error if the wrong number of arguments are supplied.
\end{description}

The options available for creation and configuring are:
\begin{description}
\item[-subcommands]
This option (if non-empty) specifies a list of ensemble subcommands that the ensemble supports. It does not need to be sorted. Each command is mapped according to the dictionary in the \textit{-map} option (if a non-empty map is present and the command has a mapping) or to the correspondingly named command (which is not required to exist at the time this is specified) in the context namespace.
\item[-map]
This option (if non-empty) specifies a dictionary that maps from ensemble subcommand names to lists of arguments to substitute into the ensemble invokation in place of the ensemble command name and the subcommand name. See the \textit{-subcommands} option for the meaning of this option when that option is also non-empty. If this option is empty and the \textit{-subcommands} option is empty too, the namespace will use the exported commands of the namespace as its command set, dynamically determining them (subject to cacheing) every time the ensemble is \textit{invoked}. Note that the words in the dictionary values that give the commands to map to (as opposed to any arguments to them) are always resolved (if not absolute) relative to the namespace which is running [namespace ensemble create] or [namespace ensemble configure].
\item[-prefixes]
This boolean option (which is on by default) controls whether unambiguous prefixes of ensemble commands are recognized as if they were the fully specified ensemble command names.
\item[-unknown]
This provides (when non-empty) a partial command to handle the case where an ensemble subcommand is not recognized and would otherwise generate an error. When empty (the default) an error (in the style of \textit{Tcl\_GetIndexFromObj}) is generated whenever the ensemble is unable to determine how to implement a particular subcommand.

See \textit{Unknown Handler Behaviour} below for details of how the ensemble interacts with its unknown handler.
\end{description}

Ensemble creation takes an extra option value pair:
\begin{description}
\item[-command]
This option allows you to override what the name of the ensemble to create is.
\end{description}

Ensemble configuring allows the querying of an extra read-only option:
\begin{description}
\item[-namespace]
This \textit{read-only} option allows the retrieval of the name of the namespace that the ensemble was created in.
\end{description}

Given an ensemble command created by the above mechanism, calling the command will first of all match the subcommand to its implementing command (or command/argument list, as derived from the dictionary) in a manner that will be recognizably similar to that enforced by \textit{Tcl\_GetIndexFromObj()} (unless the \textit{-unknown} option override this behaviour.) For details of how the ensemble determines what its subcommands are, please see the \textit{-subcommands} and \textit{-map} options above. Then the ensemble command will rewrite the command and arguments so that the ensemble command and subcommand are replaced by the implementing command and any specified arguments, with the resulting word list being fed to \textit{Tcl\_EvalObjv()} for execution. Note that this does not increase the stack depth in terms of [uplevel], and that the implementing command may itself be an ensemble command.

Note that if the namespace for an ensemble is deleted, the ensemble will also be deleted though there is no lifetime constraint in the other direction (i.e. using [rename \$theEnsemble \{\}] will not cause the namespace to vanish.) Obviously, any ensemble which has the global namespace backing it up will have the same natural lifespan as the hosting Tcl interpreter.

\section{Unknown Handler Behaviour}

If an unknown handler is specified for an ensemble, that handler is called when the ensemble command would otherwise return an error due to it being unable to decide which subcommand to invoke. The exact conditions under which that occurs are controlled by the \textit{-subcommands}, \textit{-map} and \textit{-prefixes} options as described above.

To execute the unknown handler, the ensemble mechanism takes the specified -unknown option and appends each argument of the attempted ensemble command invocation (including the ensemble command itself, expressed as a fully qualified name). It invokes the resulting command in the scope of the attempted call. If the execution of the unknown handler terminates normally, the ensemble engine reparses the subcommand (as described below) and tries to dispatch it again, which is ideal for when the ensemble's configuration has been updated by the unknown subcommand handler. Any other kind of termination of the unknown handler is treated as an error.

The result of the unknown handler is expected to be a list (it is an error if it is not). If the list is an empty list, the ensemble command attempts to look up the original subcommand again and, if it is not found this time, an error will be generated just as if the -unknown handler was not there (i.e. for any particular invokation of an ensemble, its unknown handler will be called at most once.) This makes it easy for the unknown handler to update the ensemble or its backing namespace so as to provide an implementation of the desired subcommand and reparse.

When the result is a non-empty list, the words of that list are used to replace the ensemble command and subcommand, just as if they had been looked up in the -map. It is up to the unknown handler to supply all namespace qualifiers if the implementing subcommand is not in the namespace of the caller of the ensemble command. Also note that when ensemble commands are chained (e.g. if you make one of the commands that implement an ensemble subcommand into an ensemble, in a manner similar to the text widget's \textit{tag} and \textit{mark} subcommands) then the rewrite happens in the context of the caller of the outermost ensemble. That is to say that ensembles do not in themselves place any namespace contexts on the Tcl call stack.

Where an empty -unknown handler is given (the default), the ensemble command will generate an error message based on the list of commands that the ensemble has defined (formatted similarly to the error message from \textit{Tcl\_GetIndexFromObj()}). This is the error that will be thrown when the subcommand is still not recognized during reparsing. It is also an error for an -unknown handler to delete its namespace.

\section{Examples}
\begingroup\small\begin{verbatim}
namespace eval carrot {
   namespace export foo bar potato
   namespace ensemble create     ;# Creates command ::carrot

   proc foo {} {puts 1}          ;# Exported
   proc bar {} {puts 2}          ;# Exported
   proc boo {} {puts 3}          ;# Not exported

   namespace eval turnip {       ;# Not exported
      namespace export alpha
      proc alpha {} {puts 4}     ;# Exported
      proc beta {} {puts 5}      ;# Not exported
      namespace ensemble create
   }

   namespace eval potato {       ;# Exported
      proc north {x} {puts 6,$x} ;# Not exported
      proc south {x} {puts 7,$x} ;# Not exported
      # Notice we resolve names locally, and Tcl metachars are not special
      namespace ensemble create -map {
         north {north [$meta[$chars}
      }
   }
}


carrot foo                       ;# Prints 1
carrot bar                       ;# Prints 2
carrot b                         ;# Also prints 2 ("boo" not exported)
carrot ?                         ;# ERROR: Alternatives "bar", "foo" and "potato"
carrot potato                    ;# ERROR: Missing argument
carrot potato ?                  ;# ERROR: Try "north" instead
carrot potato north              ;# Prints 6,[$meta[$chars
carrot turnip alpha              ;# ERROR: "turnip" not known
carrot::turnip alpha             ;# Prints 4
carrot::turnip::beta             ;# Prints 5

rename ::carrot::potato ::spud
spud north                       ;# Prints 6,[$meta[$chars
spud south                       ;# ERROR: "south" not known
carrot potato north              ;# ERROR: No ::carrot::potato command
# Reconfigure spud; notice we get different name resolution now!
namespace ensemble configure spud -map {
   north {puts NORTH} south {puts SOUTH}
}
spud north                       ;# Prints NORTH
spud south                       ;# Prints SOUTH
namespace delete carrot
spud north                       ;# ERROR: spud command already deleted


namespace eval A {
   proc a args {puts A::a=>$args}
}
namespace eval B {
   proc b args {puts B::b=>$args}
}
# Create an ensemble in the global namespace
namespace ensemble create -command C -map {
   eg1 {::A::a foo bar}
   eg2 {::B::b 1 2 3}
   eg3 ::string
}
C eg1 spong                      ;# Prints A::a=>foo bar spong
C eg2 evil code {[exit]}         ;# Prints B::b=>1 2 3 evil code [exit]
C eg3 length qwertyuiop          ;# Returns 10


# An example demonstrating the use of -unknown to do delegation
# This uses an ensemble to add a subcommand to a frame
package require Tk
rename [frame .f] _impl.f
namespace ensemble create -command .f -unknown {delegate _impl.f} -map {
   flash {flashFrame _impl.f}
}
# General delegation framework handler
proc delegate {target ensemble subcmd args} {
   # Read the current map
   set map [namespace ensemble configure $ensemble -map]
   # Update it to include the new subcommand
   dict set map $subcmd [list $target $subcmd]
   # Install back into the ensemble
   namespace ensemble configure $ensemble -map $map
   # Result is empty string, so we reparse
   return {}
}
# Our new subcommand implementation
proc flashFrame w {
   set bg [$w cget -background]
   foreach colour {black white black white black white} {
      $w configure -background $colour
      update idletasks
      after 150
   }
   $w configure -background $bg
}
\end{verbatim}
\endgroup

\section{Consequences}

Many commands in both Tcl and Tk would benefit from leveraging this, and it would enable straight-forward implementations of things like \cite{tip65} in pure Tcl code. It would also make doing things like partial exposure of ensemble-like commands in safe interpreters much easier.

\section{Sample Implementation}

\url{http://sf.net/tracker/?func=detail&aid=786509&group_id=10894&atid=310894}

\section{Copyright}

This document has been placed in the public domain.

\section{Acknowledgements}

Thanks very much to Joe English, Don Porter and Kevin Kenny for their suggestions in the development of this TIP. Without them, it would have been a far worse suggestion. And thanks to Will Duquette for writing a piece of software (Snit) that would benefit immensely from pushing the ensemble stuff as hard as possible and then a bit. :\textasciicircum{})

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\begin{thebibliography}{TIP \#1}
\addcontentsline{toc}{section}{References}
\bibitem[TIP \#65]{tip65}
Glenn Jackman,
Don Porter,
Glenn Jackman,
\emph{Enhanced [info args]},
on-line at \url{http://purl.org/tcl/tip/65.html}
\end{thebibliography}
\end{document}
