\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#122: Use tcl\_\{non,\}wordchars Throughout Tcl/Tk}
\date{December 12, 2002}
\author{Martin Weber, Vince Darley}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#122}&\textbf{Use tcl\_\{non,\}wordchars Throughout Tcl/Tk}\\\hline
Author&

Martin Weber \tipmail{ephaeton}{gmx.net}
\par
Vince Darley \tipmail{vincentdarley}{users.sourceforge.net}
\\
Created&Thursday, \(\text{12}^{\text{th}}\) December 2002\\
Type&Project\\
Tcl Version&\(8.6\)\\
State&Rejected\\
Vote&Done\\
Version&\tipversion$Revision: 1.9 $\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP shall bring flexible management of word and non-word chars to Tcl, to be used throughout the Tcl realm in e.g. [regexp]'s \(\backslash\)w \(\backslash\)W, Tk's [textwidget], [string] wordstart/wordend etc.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Specification}

Assignment to \textit{tcl\_\{non,\}wordchars} shall influence any place in Tcl which decides whether something is a word character or not, including detection of word boundaries in e.g. regular expressions, Tk's text widget and so on.

For this there shall be no hard-coding of lists of values which are word and non-word characters, and neither shall the language rely on the language of implementation (i.e. C's \textit{is*()} functions), as this disallows dynamic changing of \textit{tcl\_\{non,\}wordchars}.

Rather shall the value(s) of \textit{tcl\_\{non,\}wordchars} be used to determine whether a given character is part of a word or not.

\section{Rationale}

Currently in Tcl there are different hard-coded ways to decide whether a certain character is a word character or a non word character. Different hard-coded ways also imply that changes on one side might not get over to the other side, so there soon are different hard-coded ways which yield different hard-coded results. As a inference of it being hard-coded, this also means that there is no way to change or fix that potentially broken behavior. Having Tcl lookup the values of those variables at runtime allows for the needed flexibility, both when dealing with nonstandard demands and nonstandard character sets.

As an example of the breakage, you can assign a regular expression to tcl\_\{,non\}wordchars, and the double click binding in the textwidget will regard that pattern when marking a ``whole word''. When you try to ask the text widget to deliver the data under a certain coordinate with the indices 'wortstart' and 'wordend', the value of tcl\_\{non,\}wordchars is not used though.

There may be a problem with the performance of the lookup, but on the other hand are C's \textit{is*()} functions also implemented via a table lookup. An installation of a caching static character table could guarantee the needed performance.

\section{Example of current word confusion}

Tk's text widget uses ``word'' in several ways:

1. selection by word (double-click + drag),

2. movement by word ('insert wordstart'),

3. regexp searching with \(\backslash\)m\(\backslash\)M wordmatching.

4. line breaks when wrapping (-wrap word)

It is not at all clear from reading Tcl or Tk's documentation what the behaviour of the above options will be. It turns out that:

1. after a convoluted call-chain, ends up calling tcl\_wordBreakAfter/Before which use tcl\_wordchars and tcl\_nonwordchars (which actually are defined differently on Windows vs Unix/MacOS!!).

2. uses 'isalnum(char)' or '\_' to define a word (hard-coded in Tk's tkTextIndex.c) (in Tk8.5a0 this has been fixed to use \textit{Tcl\_UniCharIsWordChar})

3. uses Tcl's regexp engine's definition of a word (this ought to be the same as that used in (2)).

4. Anything separated by white-space from something else, used with '-wrap word' to define line-wrapping in text widgets (and canvases).

It is quite likely that most of the above are different under some circumstances or some platforms/locales, and certainly if the user/developer wants to create a text widget with a different word definition, they basically can't in any consistent way.

\section{Implementation}

None yet.

\section{Copyright}

This document is placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
