\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#388: Extending Unicode literals past the BMP}
\date{August 10, 2011}
\author{Jan Nijtmans, Jan Nijtmans}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#388}&\textbf{Extending Unicode literals past the BMP}\\\hline
Author&

Jan Nijtmans \tipmail{jan.nijtmans}{users.sf.net}
\par
Jan Nijtmans \tipmail{jan.nijtmans}{gmail.com}
\\
Created&Wednesday, \(\text{10}^{\text{th}}\) August 2011\\
Type&Project\\
Tcl Version&\(8.6\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.7 $\\
Keywords&Tcl\\
Post History&\\
Discussions To&Tcl Core list\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes to extend Tcl's syntax in order to be able to cope with quoted forms of Unicode characters outside the Basic Multilingual Plane.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Summary}

Tcl provides backslash substitutions of the form \textbf{\(\backslash\)uhhhh} for unicode characters, but this form is not sufficient to model unicode literals past the BMP. The outcome of the discussion on Tcl-Core was to add the form \textbf{\(\backslash\)Uhhhhhhhh} (one up to 8 hexadecimal digits), but still it is ambiguous how characters \tiprangle{} 0x10ffff are to be handled. This TIP is meant to sort that out, it is not meant to specify how characters outside the BMP are handled. The reference implementation just replaces any character in the range \textbf{\(\backslash\)U010000} --- \textbf{\(\backslash\)U10ffff} with \textbf{\(\backslash\)ufffd}, but as soon as Tcl has support for characters outside the BMP this range is reserved for exactly that.

Currently, the form \textbf{\(\backslash\)U} is parsed by Tcl as a literal \textbf{U}, so --- however small --- this change results in a non-trivial potential incompatibility which therefore requires a TIP.

Considering backslash sequences, there are two other forms which are currently not consistent: \textbf{\(\backslash\)xhh} accepts an unlimited number of hex digits, unlike other modern languages, and the form \textbf{\(\backslash\)ooo}, where the first octal digit is in the range 4..7 is currently not handled consistently in Tcl. Now is an opportunity to reconsider this.

In tcl.h there is a remark regarding the possible values of TCL\_UTF\_MAX:
\begin{itemize}
\item{}
3 Currently the only supported value, defining Tcl\_UniChar as unsigned short
\item{}
6 Not supported, but reserved for a hypothetical 32-bit Unicode
\item{}
1 Not supported, possibly for a ASCII-only variant of Tcl.
\end{itemize}

This document proposes to add another value:
\begin{itemize}
\item{}
4 Not supported. The same as 3, but allowing the use of Unicode surrogate pairs to represent the range \textbf{\(\backslash\)U010000} --- \textbf{\(\backslash\)U10ffff}
\end{itemize}

\section{Rationale}

Consider the string \textbf{\(\backslash\)701}, how is that supposed to be interpreted? Tcl specifies octal sequences as 8 bits, and silently strips the 9th bit, the same as gcc does. In Tcl's regular expression engine, the 9th bit is not stripped, there it is equivalent to \textbf{\(\backslash\)u01c1}. Java --- for example --- parses it as \textbf{\(\backslash\)70} --- a valid 8-bit octal value --- followed by \textbf{1}, so it's a string of length 2.

Then the string \textbf{\(\backslash\)x1234}. Tcl specifies this as 8 bits as well, and silently strips all higher bits, so it is equivalent to \textbf{\(\backslash\)u0034}. This is the same as gcc does, but Java --- for example --- considers it as \textbf{\(\backslash\)x12} followed by \textbf{34}, so it's a string of lenght 3.

Consider the string \textbf{\(\backslash\)U00123456}, which would result in an invalid Unicode character. In the Tcl parser we don't have the possibility to flag invalid backslash sequences, in Tcl's regexp engine we have. Unicode characters higher than \textbf{\(\backslash\)U0010ffff} cannot appear in an UTF-8 stream.

In tcl.h, we find Tcl\_UniChar to be defined as unsigned int when TCL\_UTF\_MAX \tiprangle{} 3 and as unsigned short otherwise. It would be useful to reserve the value 4 and still define Tcl\_UniChar as unsigned short. That would allow the path to a full support for out-of BMP Unicode characters shorter, because Unicode Surrogate pairs can be used for that.

\section{Specification}

This document proposes:
\begin{itemize}
\item{}
Change the parsers in Tcl to handle octal sequences higher than \textbf{\(\backslash\)377} differently, splitting it in a two-digit valid octal secuence and handling the additional character separately. So \textbf{\(\backslash\)701} is handled as the valid sequence \textbf{\(\backslash\)70} followed by \textbf{1}. This is a \textbf{potential incompatibility}.
\item{}
Change the parsers in Tcl to handle the \textbf{\(\backslash\)xhh} sequence to parse just 2 digits, and not silently strip all higher hex digits any more. This is a \textbf{potential incompatibility}.
\item{}
Add the \textbf{\(\backslash\)Uhhhhhhhh} handling, similar to the \textbf{\(\backslash\)uhhhh} handling, only accepting up to 8 characters. The parser will stop parsing earlier when a code point \textbf{\(\backslash\)U00011000} or higher is reached, as shifting it 4 bits more will lead to a code point outside the Unicode range. The regexp engine already handles \textbf{\(\backslash\)Uhhhhhhhh}, but currently it always generates a character in the BMP and strips all higher bits. This is a \textbf{potential incompatibility}.
\end{itemize}

\section{Compatibility}

Tcl scripts using the form \textbf{\(\backslash\)ooo} where the first digit is in the range 4-7, will now interpred the string as \textbf{\(\backslash\)oo} followed by \textbf{o}. There is currently no test case in the Tcl test suite affected by that.

Tcl scripts using the form \textbf{\(\backslash\)xhhh}, which used to strip off the higher digits, will now end the sequence when two digits are handled.

Tcl scripts using \textbf{\(\backslash\)U} as a literal \textbf{U} will no longer work when it is followed with at least one hexadecimal digit. There is currently no test case in the Tcl test suite affected by that.

\section{Alternatives}

Unicode Noncharacters or Surrogages are not supposed to be in an UTF-8 stream, the Unicode standard recommends to replace those with \textbf{\(\backslash\)ufffd}. This TIP does not propose that, because they may be used internally e.g. as part of a character range in a regular expression. A better place to do that is in the Utf8-To-Utf8 conversion, but that's outside the scope of this TIP.

\section{Reference Implementation}

A reference implementation is available at \url{http://core.tcl.tk/tcl} in branch tip-388-impl

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
