\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#389: Full support for Unicode 8.0 and later}
\date{August 23, 2011}
\author{Jan Nijtmans, Jan Nijtmans}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#389}&\textbf{Full support for Unicode 8.0 and later}\\\hline
Author&

Jan Nijtmans \tipmail{jan.nijtmans}{users.sf.net}
\par
Jan Nijtmans \tipmail{jan.nijtmans}{gmail.com}
\\
Created&Tuesday, \(\text{23}^{\text{rd}}\) August 2011\\
Type&Project\\
Tcl Version&\(8.7\)\\
State&Draft\\
Vote&Pending\\
Version&\tipversion$Revision: 1.4 $\\
Keywords&Tcl\\
Post History&\\
Discussions To&Tcl Core list\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes to add full support for all characters in Unicode 8.0+, inclusive the characters \tiprangle{}= \textbf{U+010000}.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Summary}

In order to extend the range of the characters to more than 16 bits, the type \textit{Tcl\_UniChar} is not big enough any more to hold all possible characters. Changing the type of \textit{Tcl\_UniChar} to a 32-bit quantity is not an option, as it will result in a binary API incompatibility.

The solution proposed in this TIP is to keep \textit{Tcl\_UniChar} a 16-bit quantity, but to increase the value of TCL\_UTF\_MAX to 4 (from 3). Any conversions from UTF-8 to \textit{Tcl\_UniChar} will convert any valid 4-byte UTF-8 sequence to a sequence of two Surrogate characters. All conversions from UTF-16 to UTF-8 will make sure that any High Surrogate immediately followed by a Low Surrogate will result in a single 4-byte UTF-8 character.

This can be done in a binary compatible way: No source code of existing extensions need to be modified. As long as no characters \tiprangle{}= \textbf{U+010000} or Surrogates are used, all functions provided by the Tcl library will function as before. There are few functions which currently return a value of type \textit{Tcl\_UniChar}, those will be modified to return an int in stead.

\section{Rationale}

As Unicode 8.0, and future Unicode versions, will supply more and more characters outside the 16-bit range, it would be useful if Tcl supports that as well.

\section{Specification}

This document proposes:
\begin{itemize}
\item{}
Change the functions \textit{Tcl\_UniCharToUtf} and \textit{UnicodeToUtfProc} such that when they are fed with a valid High Surrogate followed by a Low Surrogate, the result will be a single 4-byte UTF-8 character.
\item{}
Change the functions \textit{Tcl\_UtfToUniChar} and \textit{UtfToUnicodeProc} such that when they are fed with a valid 4-byte UTF-8 character, the first call will return a High Surrogate character, and the next call will return a Low Surrogate character.
\item{}
The following functions, which currently return a Tcl\_UniChar, will be changed to return an int instead:
\begin{itemize}
\item{}
\textit{Tcl\_UniCharAtIndex}
\item{}
\textit{Tcl\_UniCharToLower}
\item{}
\textit{Tcl\_UniCharToTitle}
\item{}
\textit{Tcl\_UniCharToUpper}
\item{}
\textit{Tcl\_GetUniChar}
\end{itemize}
\item{}
Extend tclUniData.c to include all Unicode 8.0 characters up to \textbf{U+02FA20}. A special case will be made for the functions \textit{Tcl\_UniCharIsGraph} and \textit{Tcl\_UniCharIsPrint} for the characters in the range \textbf{U+0E0100} --- \textbf{U+0E01EF}, otherwise it would almost double the Unicode table size.
\end{itemize}

\section{Compatibility}

As long as no Surrogates or characters \tiprangle{}= \textbf{U+010000} are used, all functions behave exactly the same as before. The only way that \textit{Tcl\_UniCharToUtf} can produce a 4-byte output is when Surrogates or characters \tiprangle{}= \textbf{U+010000} are used.

Extension that want to be compatible with any Tcl version, can include tcl.h as follows:
\begingroup\small\begin{verbatim}
#define TCL_UTF_MAX 4
#include <tcl.h>
\end{verbatim}
\endgroup

or they can call the C compiler with the additional argument -DTCL\_UTF\_MAX=4, in order to be sure that UTF-8 representations of length 4 can be handled. This way, the extension can be used with any Tcl version, whether it supports Surrogates or not.

Apart from this, it is advisable to initialize the variable where the chPtr argument from \textit{Tcl\_UtfToUniChar} points to, as this location is used to remember whether the High Surrogate is already produced or not. Not doing so when the first character of a string is a character \tiprangle{} \textbf{U+010000} might result in a Low Surrogate character only. This danger, however unlikely, only exists for the first character in a string, and it only occurs when the (random) value is exactly equal to the expected High Surrogate.

\section{Reference Implementation}

A reference implementation is available at \url{http://core.tcl.tk/tcl} in branch tip-389-impl

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
