\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#258: Enhanced Interface for Encodings}
\date{October 01, 2005}
\author{Don Porter}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#258}&\textbf{Enhanced Interface for Encodings}\\\hline
Author&

Don Porter \tipmail{dgp}{users.sf.net}
\\
Created&Saturday, \(\text{1}^{\text{st}}\) October 2005\\
Type&Project\\
Tcl Version&\(8.5\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.4 $\\
Keywords&encoding\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes public C routines and a new \textbf{encoding dirs} subcommand to improve the interfaces to Tcl's encodings.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Background}

Several internal improvements have been made to the internals of how Tcl encodings are initialized, found, stored, and refcounted during Tcl 8.5 development. This TIP is primarily about making these improvements available via public interfaces.

The lifetime of \textbf{Tcl\_Encoding} values has been identified as a problem (Bug 1077262), where premature freeing means repeated re-loading of encoding data. Since each encoding data load involves interaction with the filesystem, this can be an expensive mistake.

The Tcl documentation has long claimed that by setting the value of a global variable \textbf{::tcl\_libPath} a script could influence the search path of directories where encoding data files are sought. That documentation has never been correct (Bug 463190).

Tclkit suffers from an initialization dilemma. It stores encoding data files in a virtual filesystem. In particular the system encoding is often based on a data file in the virtual filesystem. The Tclkit virtual filesystem is (largely) script-implemented and cannot exist until a \textbf{Tcl\_Interp} has been created. However, Tcl wants to determine the correct value for the system encoding very early in its initialization, before any \textbf{Tcl\_Interp} gets created. The consequence is that Tclkit fails to successfully set the system encoding in Tcl's early initialization, and Tclkit has had to jump through hoops to get Tcl to repeat those early initialization steps after the virtual filesystem is in place.

\section{Proposed changes}

Add the following routines to Tcl's public interface:

\begin{quote}
int \textbf{Tcl\_GetEncodingFromObj}(Tcl\_Interp *\textit{interp}, Tcl\_Obj *\textit{objPtr}, Tcl\_Encoding *\textit{encodingPtr})
\end{quote}

Writes to *\textit{encodingPtr} the \textbf{Tcl\_Encoding} value that corresponds to the value of \textit{objPtr}, and returns \textbf{TCL\_OK}. The \textbf{Tcl\_Encoding} value is also cached as the internal rep of \textit{objPtr} so that the lifetime of the \textbf{Tcl\_Encoding} data in the process will be at least the lifetime of that internal rep of \textit{objPtr}. The caller is expected to call \textbf{Tcl\_FreeEncoding} on *\textit{encodingPtr} when it no longer needs it. If no corresponding \textbf{Tcl\_Encoding} value for the value of \textit{objPtr} can be determined, \textbf{TCL\_ERROR} is returned, and an error message is stored in the result of \textit{interp}.

\begin{quote}
Tcl\_Obj *\textbf{Tcl\_GetEncodingSearchPath}()
\end{quote}

Returns a list of directory pathnames that Tcl's encoding subsystem will search for encoding data files when an encoding is requested that's not already loaded in the process. This will be the value stored by the last successful call to \textbf{Tcl\_SetEncodingSearchPath}. If no calls to \textbf{Tcl\_SetEncodingSearchPath} have occurred, Tcl will compute an initial value based on the environment. There is one encoding search path for the entire process, shared by all threads in the process.

\begin{quote}
int \textbf{Tcl\_SetEncodingSearchPath}(Tcl\_Obj *\textit{searchPath})
\end{quote}

Stores \textit{searchPath} as the list of directory pathnames for Tcl's encoding subsystem to search for encoding data files, and returns \textbf{TCL\_OK}. Returns \textbf{TCL\_ERROR} only if \textit{searchPath} is not a valid Tcl list. There is no checking for validity of the directory pathnames, so for example, one can place a directory on the encoding search path before mounting the \textbf{Tcl\_Filesystem} that contains that directory. When searching for encoding data files, Tcl's encoding subsystem ignores any non-existent directories in the search path as well.

\begin{quote}
CONST char *\textbf{Tcl\_GetEncodingNameFromEnvironment}(Tcl\_DString *\textit{bufPtr})
\end{quote}

This routine exposes Tcl's determination about what the system encoding should be, based on system calls and examination of the environment suitable for the platform. It accepts \textit{bufPtr}, a pointer to an uninitialized or freed \textbf{Tcl\_DString} and writes to it the string value of the appropriate system encoding dictated by the environment. The \textbf{Tcl\_DStringValue} is returned.

In a properly initialized Tcl, the string value returned by \textbf{Tcl\_GetEncodingNameFromEnvironment} ought to be the same as that returned by \textbf{Tcl\_GetEncodingName}(\textbf{NULL}); that is, the system encoding dictated by the environment ought to be the encoding Tcl will return as the result of \textbf{encoding system}.

If these two results do not match, it indicates that at the time Tcl was initialized, the proper sytem encoding was not available. Perhaps the necessary data file was not on the encoding search path at that time. With this new routine, the check for this match can be performed, and if the match does not exist, a call to \textbf{Tcl\_SetSystemEncoding} can try again to get Tcl's system encoding to agree with what the environment dictates.

Add a new subcommand, \textbf{encoding dirs} with syntax:

\begin{quote}
\textbf{encoding dirs} ?\textit{searchPath}?
\end{quote}

This subcommand is the script-level interface to the \textbf{Tcl\_GetEncodingSearchPath} and \textbf{Tcl\_SetEncodingSearchPath} routines. When called without an argument, the current list of directory pathnames to be searched for encoding files is returned. When called with \textit{searchPath} argument, the value \textit{searchPath} is set as the new list of directory pathnames to be searched.

The documentation for existing routines \textbf{Tcl\_GetDefaultEncodingDir} and \textbf{Tcl\_SetDefaultEncodingDir} will be updated to discourage their use and to encourage the use of \textbf{Tcl\_GetEncodingSearchPath} and \textbf{Tcl\_SetEncodingSearchPath} instead.

\section{Compatibility}

This proposal includes only new features. It is believed that existing scripts and C code that operate without errors will continue to do so.

The \textbf{encoding dirs} command has been available with the name \textbf{::tcl::unsupported::EncodingDirs} since the Tcl 8.5a3 release. It is proposed to remove this unsupported command completely, as it has only existed in alpha releases. Anyone using it should be able to migrate to \textbf{encoding dirs} without difficulty.

\section{Reference Implementation}

The actual code is already complete as internals corresponding to the proposed public. Implementation is just an exercise in renaming, placing in stub tables, documentation, etc.

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
