\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#137: Specifying Script Encodings for [source] and tclsh}
\date{May 29, 2003}
\author{Anton Kovalenko}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#137}&\textbf{Specifying Script Encodings for [source] and tclsh}\\\hline
Author&

Anton Kovalenko \tipmail{a_kovalenko}{fromru.com}
\\
Created&Thursday, \(\text{29}^{\text{th}}\) May 2003\\
Type&Project\\
Tcl Version&\(8.5\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.6 $\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes a way to specify encoding used to read a script with \textit{source} command and (tclsh, wish) shell.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Rationale}

Now all Tcl scripts are expected to be in the system encoding. An author of a script or a package can't specify concrete encoding to source the script or the files of package. Common practise is to assume that encoding to be superset of ASCII, and not to use non-ASCII characters in the scripts that are targeting the wide audience of potential users.

There's a way to specify UNICODE literals with \(\backslash\)uXXXX sequences. But it's not an universal and convenient way --- this sequences aren't substituted in \{\}-quoted strings, and they can't be edited in WYSIWYG editors without special (and hence uncommon) support.

This TIP proposes to add the \textit{-encoding} option support to \textit{source} command and to tclsh and wish. Thus, package authors will be able to specify encoding of the package files in \textit{pkgIndex.tcl}; and script authors will be able to specify the script encoding when calling tclsh (either in the first line beginning with \textit{\#!} or in the line where the \textit{exec tclsh ``\$0'' ``\$@''} is located).

This TIP also proposes to use utf-8 for all the standard system scripts, for \textit{pkgIndex.tcl} and \textit{tclIndex} files. Now they all are supposed to be in the system encoding (message catalogs do not need this support, as they are already always loaded using the UTF-8 encoding scheme), and it could event prevent Tcl itself from running when system encoding is not a compatible superset of ASCII.

\section{Specification}

Tclsh will allow the encoding to be specified on the command-line in two forms: the first form is \textit{-encoding name} as two separate arguments, and the second is \textit{-e:name} (a single argument.) The second form is intended for when the script begins with \textit{\#!} and is because Unix kernels pass extra parameters from the \textit{\#!} line as a single argument. This very short notation (-e:) is chosen because some Unices limit the \textit{\#!} line to the length of 32.

To implement all these options, this TIP proposes a new C-level public API function Tcl\_FSEvalFileEx, which is similar to Tcl\_FSEvalFile, but takes one extra argument that must be an encoding name or NULL (to use system encoding).
\begingroup\small\begin{verbatim}
 int Tcl_FSEvalFileEx(Tcl_Interp *interp, Tcl_Obj *pathPtr,
                      CONST char *encodingName);
\end{verbatim}
\endgroup

Common use of this new options will be like this:
\begin{enumerate}
\item{}
In a script:
\begingroup\small\begin{verbatim}
 !#/usr/bin/tclsh -e:utf-8
 do something...
\end{verbatim}
\endgroup

or it could be:
\begingroup\small\begin{verbatim}
 #!/bin/sh
 # I wish to use tclsh \
 exec tclsh -encoding utf-8 "$0" "$@"
\end{verbatim}
\endgroup
\setcounter{enumi}{1}
\item{}
In a \textit{pkgIndex.tcl}:
\begingroup\small\begin{verbatim}
 package ifneeded pkg 1.02 [list source -encoding utf-8 \
      [file join $dir pkg.tcl]]
\end{verbatim}
\endgroup
\end{enumerate}

\section{Implementation}

The partial implementation of this TIP can be found at \url{http://sourceforge.net/projects/tcl} as Patch \#742683. Some fixes to source system scripts and pkgIndex'es in utf-8 are not yet there. But with the existing implementation of ``-encoding'', it would be really easy to implement.

\section{Copyright}

This document is placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\end{document}
