\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#232: Creating New Math Functions for the 'expr' Command}
\date{November 26, 2004}
\author{Arjen Markus, Kevin Kenny}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#232}&\textbf{Creating New Math Functions for the 'expr' Command}\\\hline
Author&

Arjen Markus \tipmail{arjen.markus}{wldelft.nl}
\par
Kevin Kenny ‚kennykb@acm.org ƒ
\\
Created&Friday, \(\text{26}^{\text{th}}\) November 2004\\
Type&Project\\
Tcl Version&\(8.5\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.11 $\\
Keywords&math, expr, Tcl\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes to replace Tcl's math functions, ordinarily created using the \textbf{Tcl\_CreateMathFunc()} function, with ordinary Tcl commands created in a known namespace, \textbf{::tcl::mathfunc}. This change has two chief motivations: it allows programmers to define new math functions at the Tcl level, and it encapsulates the \textbf{Tcl\_Value} API so that new math functions can work on data types that are not described adequately as \textbf{Tcl\_Value} objects.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Rationale}

The two authors of this TIP, Kevin Kenny and Arjen Markus, have come at wanting the same change to Tcl from two distinct directions.

Arjen Markus has been the maintainer of several modules in Tcllib that implement ``special functions'' such as the exponential integral, the incomplete gamma function and the Bessel functions. He has wanted to have ``scripted'' math functions available to simplify his notation. In place of a complex expression like
\begingroup\small\begin{verbatim}
  set f [expr { $x*[J0 $x] + [J1 [expr {2.0*$x}]] }]
\end{verbatim}
\endgroup

which is pretty unreadable (and prone to error because of the need to brace the subexpressions), he would like a mechanism for defining new mathematical functions so that he can write:
\begingroup\small\begin{verbatim}
  set f [expr { $x*J0($x) + J1(2.0*$x) }]
\end{verbatim}
\endgroup

Donal Fellows has felt the need for such a construct, too, enough to implement a ``funcproc'' extension that provides it. [\url{http://wiki.tcl.tk/541}] 

Kevin Kenny has come at the desire for rework of the math function mechanism from a different direction. He has been investigating Tcl Core support for arbitrary-precision integers \cite{tip237}, and discovered that implementing math functions that accept them as arguments or return them as results would require an incompatible change to the \textbf{Tcl\_Value} data type that is used to communicate with the math functions. When the last such change was made (in implementing 64-bit integers \cite{tip72}), it required a source code change to several popular extensions that created new math functions \textit{min} and \textit{max}. Wishing the incompatibility introduced by \cite{tip237} to be the last of the type, he decided to eliminate \textbf{Tcl\_Value} and use the open-ended \textbf{Tcl\_Obj} data type in its place. When he derived the type signature that a math function would have if it accepted its parameters in an array of \textbf{Tcl\_Obj} pointers, he discovered that it was identical to the \textbf{Tcl\_ObjCmdProc} -{}- making him ask, ``what if math functions really \textit{were} commands?''

This TIP is the result of those two investigations, and proposes a unification of math functions with Tcl commands.

\section{Proposed Changes}

This TIP proposes that:
\begin{enumerate}
\item{}
The [expr] command shall be modified so that an expression of the form:
\begingroup\small\begin{verbatim}
   f(arg1,arg2,...)
\end{verbatim}
\endgroup

shall generate code equivalent to that generated by
\begingroup\small\begin{verbatim}
   [tcl::mathfunc::f [expr { arg1 }] [expr { arg2 }] ...]
\end{verbatim}
\endgroup

so that math functions are interpreted as Tcl commands whose arguments are parsed as subexpressions. The existing code in [expr] that checks for correct argument counts to the math functions at compile time shall be removed (the general consensus among Core maintainers is that compile-time checks of this sort are a bad idea anyway).

Note that the call to \textit{tcl::mathfunc::f} has no leading namespace delimiter. A search for the function will try to resolve it first relative to the current namespace.
\item{}
The current math functions in Tcl shall be reimplemented as ordinary Tcl commands in the \textbf{::tcl::mathfunc} namespace; a Tcl interpreter will contain commands \textbf{::tcl::mathfunc::abs}, \textbf{::tcl::mathfunc::acos}, etc. 
\item{}
The Tcl command [info functions] shall be deprecated in favor of searching for math functions using [info commands] in the appropriate namespace.
\item{}
The C functions \textbf{Tcl\_CreateMathFunc}, \textbf{Tcl\_GetMathFuncInfo}, and \textbf{Tcl\_ListMathFuncs} shall be deprecated in favor of \textbf{Tcl\_CreateObjCommand}, \textbf{Tcl\_GetCommandInfo} and [info commands]. (The last is provided only in Tcl at the present time; we do not export a C-level interface to enumerate commands in a namespace.) The functions will continue to work for extensions that use them. The \textbf{Tcl\_CreateMathFunc} command will create a Tcl command in the \textbf{::tcl::mathfunc} namespace whose command procedure checks argument count and types and dispatches to the math function handler. The \textbf{Tcl\_GetMathFuncInfo} procedure will return \textit{TCL\_OK} or \textit{TCL\_ERROR} according to whether the given math function exists in \textbf{::tcl::mathfunc} and will return parameter information for (only) those functions defined using \textbf{Tcl\_CreateMathFunc}. Functions defined as ordinary Tcl commands shall return \textit{TCL\_OK} but have a parameter count of -1. The \textbf{Tcl\_ListMathFuncs} procedure will simply enumerate all the commands in the \textbf{::tcl::mathfunc} namespace.
\item{}
Several error messages change as a side effect of changing the math function implementation. All of the new messages are more informative than the old ones (for instance, identifying which math function encountered a parameter error rather than a generic ``too few/too many arguments to math function''), with the exception of the error message for an unknown function, which is replaced with ``unknown command ::tcl::mathfunc::\tiplangle{}the\_command\tiprangle{}'' where \tiplangle{}the\_command\tiprangle{} is the name of the missing function.
\end{enumerate}

\section{Discussion}

The proposed change lifts several other restrictions in the way math functions operate:
\begin{enumerate}
\item{}
There is no longer any restriction that new math functions must accept a fixed number of arguments, or, indeed, that their arguments and results must be numeric. It will be possible to create variadic functions like:
\begingroup\small\begin{verbatim}
   proc ::tcl::mathfunc::min args {
       set m Inf
       foreach a $args {
           if { $a < $m } {
               set m $a
           }
       }
       return $m
   }
\end{verbatim}
\endgroup
\item{}
It will be possible to compile Tcl code that refers to a math function that has not yet been defined. That is:
\begingroup\small\begin{verbatim}
   proc foo { x } {
       set have_f [llength [info commands ::tcl::mathfunc::f]]
       if { $have_f } {
           return [expr { f($x) }]
       } else {
           ... fallback ...
       }
   }
\end{verbatim}
\endgroup

will be a valid procedure. (In Tcl 8.4, the procedure will fail to compile if \textit{f} is not known at compile time, something that runs contrary to the dynamic nature of Tcl.)
\item{}
Namespaces will be able to define their own math functions that are not visible outside those namespaces. If a namespace defines a function \textbf{[namespace current]::tcl::mathfunc::f}, then calls to \textbf{f} in expressions evaluated in that namespace will resolve to it in preference to \textbf{::tcl::mathfunc::f}. Not only does this rule allow two extensions both to define functions \textbf{f} without collision, but it also allows an extension to override a builtin function such as \textbf{sin}.
\end{enumerate}

Alas, all these improvements come at some cost of performance. On Kevin Kenny's machine, the command
\begingroup\small\begin{verbatim}
 expr {sin(0.325)}
\end{verbatim}
\endgroup

executes in roughly 520 nanoseconds before the change, and 870 nanoseconds afterward. Since the resolution of the function name is cached, name lookup is not the problem; rather, the issue is that invoking the function as a command needs to look for command traces; this whole mechanism costs about 300-350 ns (and has been observed to do so in other contexts). For real-life expressions, the additional cost tends to vanish quickly into statistical noise; variable access (with corresponding checks for traces) and bytecode overhead quickly comes to dominate the performance for all but the simplest expressions.

\section{Safe Interpreters}

It is not anticipated that exposing this functionality in a safe interpreter will present any new problems for safety. Any functionality that the interpreter can access by defining or overriding math functions is functionality that would have been available to it by calling the functions as commands.

\section{Impact on C-coded Extensions}

Extensions coded in C that wish to create math functions accepting parameters of type \textbf{TCL\_EITHER} may find that they do not get type coercion from parameters of new numeric types, such as extended-precision integers. The coding change to replace them with Tcl commands is fairly easy and mechanical, at a level of effort comparable to that needed for \cite{tip72}. Moreover, once it is completed, a math function will be using the known \textbf{Tcl\_CreateObjCmd} API, which has been stable since Tcl 8.0 and is unlikely to change substantially in future releases.

The \textit{tbcload} extension will need to implement a small amount of bytecode translation to preserve compatibility with bytecode compiled modules built against earlier versions of Tcl. The reason is that two bytecodes, \textbf{INST\_INVOKE\_BUILTIN1} and \textbf{INST\_INVOKE\_FUNC1} have been eliminated from \textbf{Tcl\_ExecuteByteCode} since the compiler no longer emits them. If this change for some reason should prove infeasible, we can always put the bytecodes back into \textbf{Tcl\_ExecuteByteCode}, but the authors of this TIP would prefer to avoid the code bloat.

\section{Reference Implementation}

A reference implementation is committed on the 'tcl-numerics-branch' at SourceForge and may be retrieved with
\begingroup\small\begin{verbatim}
 cvs -d:ext:USERID@cvs.sourceforge.net:/cvsroot/tcl checkout -rTIP232 tcl_numerics
\end{verbatim}
\endgroup

\section{Copyright}

Copyright (c) 2005 by Kevin B. Kenny and Adriaan Markus. All rights reserved.

This document may be distributed subject to the terms and conditions set forth in the Open Publication License, version 1.0 [\url{http://www.opencontent.org/openpub/}].

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\begin{thebibliography}{TIP \#2}
\addcontentsline{toc}{section}{References}
\bibitem[TIP \#72]{tip72}
Donal K. Fellows,
\emph{64-Bit Value Support for Tcl on 32-Bit Platforms},
on-line at \url{http://purl.org/tcl/tip/72.html}
\bibitem[TIP \#237]{tip237}
Kevin B. Kenny,
Don Porter,
\emph{Arbitrary-Precision Integers for Tcl},
on-line at \url{http://purl.org/tcl/tip/237.html}
\end{thebibliography}
\end{document}
