\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#463: Command-Driven Substitutions for regsub}
\date{February 11, 2017}
\author{Donal Fellows}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#463}&\textbf{Command-Driven Substitutions for regsub}\\\hline
Author&

Donal Fellows \tipmail{dkf}{users.sf.net}
\\
Created&Saturday, \(\text{11}^{\text{th}}\) February 2017\\
Type&Project\\
Tcl Version&\(8.7\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.6 $\\
Keywords&Tcl, regular expression\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

The \textbf{regsub} command can only do substitutions of a limited complexity. This TIP adds an option to generate substitution text using another Tcl command, allowing a more complex range of substitutions to be performed easily and safely.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Rationale and Outline Proposal}

Many scripts wish to perform subsitutions on a string where the text to be substituted can be described by a regular expression, but where the text to be substituted in cannot easily be generated by the \textbf{regsub} command. There are workarounds for this, as seen in this example (from the Wiki):
\begingroup\small\begin{verbatim}
 set text [subst [regsub -all {[a-zA-Z]} [\
     regsub -all "\[\[$\\\\\]" $text {\\&}] {[
         set c [scan & %c]
         format %c [expr {$c\&96|(($c\&31)+12)%26+1}]
     ]}]]
\end{verbatim}
\endgroup

But it is not at all trivial to write such things! Instead, we should be able to do this:
\begingroup\small\begin{verbatim}
 set text [regsub -all -command {[a-zA-Z]} $text {apply {c {
     scan $c %c c
     format %c [expr {$c&96|(($c&31)+12)%26+1}]
 }}}]
\end{verbatim}
\endgroup

It's going to be both safer (as there's no required non-obvious metadata defanging preprocessing step) and faster (as we can do this as a command call rather than a \textbf{subst} that needs separate bytecode compilation).

The parallels with Perl's ``e'' flag to its regular expression substitution operator should be obvious.

\section{Proposed Change}

My proposal is that we add a flag to the \textbf{regsub} command, \textbf{-command}, that changes the interpretation and processing of the substitution argument. When the flag is passed, instead of that argument being a string that is processed for \textbf{\&} and backslash-number sequences, it is instead interpreted as a command prefix; the various captured substrings (minimally the entire string passed in, but also any captured substrings specified in the RE) will become extra arguments added, and the result will be evaluated and the result of that evaluation will be used as the string to substitute in. If the \textbf{-all} option is not given, the substitution command will be called at most once, whereas if \textbf{-all} is given, the substitution command will be called for as many times as the regular expression matches. The indices in the original script that matched will not be available.

Non-OK results will be passed through to the surrounding script.

Substitutions too complex to be described by a simple command can be done by using a procedure or \textbf{apply}/lambda-term (as in the example above). The arguments received by the command invoked by \textbf{regsub -command} will be exactly the substrings that were matched, with no other substitutions performed on them.

\subsection{Examples}

The command:
\begingroup\small\begin{verbatim}
 regsub -all -command {\w} "ab-cd-ef-gh" {  puts  }
\end{verbatim}
\endgroup

will give \textbf{-{}-{}-} as its result and print the letters \textbf{a} to \textbf{h}, one per line in that order.

The command:
\begingroup\small\begin{verbatim}
 regsub -command {\W(\W)} "ab cd,{ef gh,} ij" {apply {{x y} {
     scan $y %c c
     format %%%02x $c
 }}}
\end{verbatim}
\endgroup

will produce this result:
\begingroup\small\begin{verbatim}
 ab cd%7bef gh,} ij
\end{verbatim}
\endgroup

\section{Implementation}

\url{http://core.tcl.tk/tcl/timeline?r=tip-463}

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
