\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#412: Dynamic Locale Changing for msgcat with On-Demand File Load}
\date{March 27, 2012}
\author{Harald Oehlmann, Harald Oehlmann}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#412}&\textbf{Dynamic Locale Changing for msgcat with On-Demand File Load}\\\hline
Author&

Harald Oehlmann \tipmail{harald.oehlmann}{elmicron.de}
\par
Harald Oehlmann \tipmail{oehhar}{users.sourceforge.net}
\\
Created&Tuesday, \(\text{27}^{\text{th}}\) March 2012\\
Type&Project\\
Tcl Version&\(8.6\)\\
State&Final\\
Vote&Done\\
Version&\tipversion$Revision: 1.9 $\\
Keywords&Tcl, localization, msgcat\\
Obsoletes&399\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP adds dynamic locale switching capabilities to the \textbf{msgcat} package.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Rationale}

\subsection{Dynamic Locale Switching}

Within a multi-language application like a web-server, one may change the locale quite frequently, for example if users with different locales are requesting pages. Unfortunately, this does not fit well with the model adopted by the msgcat package, which assumes that all code follows this sequence:
\begin{enumerate}
\item{}
 Set locale list: \textbf{mclocale} \textit{locale}
\setcounter{enumi}{1}
\item{}
 Load language files with other package load: \textbf{mcload} \textit{msg-folder}
\setcounter{enumi}{2}
\item{}
 Translate strings: \textbf{mc} \textit{key args...}
\end{enumerate}

Note that if the locale should be changed after other packages are loaded, one must restart at step 2. This requires reloading all packages which is mostly not practical.

The aim of this TIP is to extend the package by dynamic locale change capabilities.

msgcat will reload any missing message catalog files of all currently loaded packages on a locale change. In addition, any package may register to get informed to a locale change. Other packages may do changes to reflect the locale change like rebuilding the GUI.

This TIP compares to \cite{tip399} that the package is able to load message catalog files on demand, e.g. specially on a locale change.

\subsection{package locale}

If the clock command gets called with the argument ``-locale'', the locale is changed using msgcat::mclocale. After processing, the initial value is restored. The package keeps track, which locales where already used and calls msgcat::mcload for any new locale. The locale is restored after processing.

This is an implementation of dynamic locales but conflicts with the new features described above. Other packages may be informed to change the locale and may trigger expensive operations like a rebuild of the GUI.

In consequence, each package may define a package locale which is independent of the default locale.

\section{Overview of the proposed solution}

Proposed changes in brief:

\subsection{Dynamically load message catalog files}

if the locale is changed by mclocale locale, the message file load process is executed for every present package.

\subsection{Package locale}

A package may install a package local locale which is independent to the global locale.

\subsection{Locale change callback}

A callback may be registered to get informed about the change of locale. A use case is to refresh a GUI if the locale changed.

\subsection{Non message file operation}

A program may use message files to issue mcset commands or may issue them by other means, if the message catalogs are, for example, stored in a data base.

Each package may register a callback to get informed that a certain locale should be loaded and may issue the corresponding mcset commands.

\subsection{Package mcunknown}

A package may have a certain way to provide translations for message keys not included in the message catalog. Thus, it may register an own package message unknown callback to provide a translation.

\section{Specification}

\subsection{Package Equals Client Namespace}

A client package is a package which uses msgcat. A unique namespace is required for each client package. Within msgcat, namespace and package is always connected.

Up to now, the msgcat package used this namespace as an identifier to store the catalog data of a certain package. This is now extended to additional properties which are stored for a package.

\subsection{Package locale}

A package locale may be used by a package instead the default locale set by msgcat::mclocale. A package may choose to use a package locale or the default locale.

\subsection{Default and Package State}

Some state values (like the locale) are available as default (global) values. In addition, each package may choose to use a package locale state.

The used naming is:

 default
\begin{description}
\item[state]
valid for all packages which do not set a package state.
\end{description}

 package
\begin{description}
\item[state]
only valid for one package if it has set a package state.
\end{description}

The following state values are present as default state and may be set individually per package:

 The locale like ``de\_ch''. The preferences property is a list of locales in their preference order and is automatically computed from locale. Example locale = ``de\_ch'' -\tiprangle{} preferences = ``de\_ch de \{\}''. The loadedlocales state value is the list of currently loaded locales.

\subsection{Default State}

The following standard methods exist to get or set the default state:

\subsubsection{msgcat::mclocale}

The default locale. It may be read using msgcat::mclocale.

It may be set using msgcat::mclocale locale. This command is extended, that the message catalogs of all missing locales for all packages not having set a package state are loaded.

\subsubsection{msgcat::mcpreferences}

Get the default preferences (derived from the default locale).

\subsubsection{msgcat::mcloadedlocales}

The following new command may be used to deal with the default state:

\begin{quote}
\textbf{msgcat::mcloadedlocales} \textit{subcommand} \textit{?locale?}
\end{quote}

The parameter locale is mandatory for the subcommand present.

The following subcommands are available:

\subsubsection{Subcommand "get"}

Get the list of current loaded locales

\subsubsection{Subcommand "present"}

Returns true, if the given locale is loaded

\subsubsection{Subcommand "clear"}

The list of currently loaded locales is set to mcpreferences and all message catalog keys of packages without a package locale set and with locales not in mcpreferences are unset.

\subsection{Package Configuration}

The package configuration of the calling package may be changed using the following new command:

\begin{quote}
\textbf{msgcat::mcpackagelocale} \textit{subcommand} ?\textit{locale}?
\end{quote}

The parameter locale is mandatory for the subcommands set and present.

Available subcommands are:

\subsubsection{Subcommand "set"}

Set or change the package locale.

The global state values are copied, if there were no package locale set before.

The package locale is changed to the optional given new package locale.

\subsubsection{Subcommand "get"}

Return the package locale or the default locale, if no package locale set.

\subsubsection{Subcommand "preferences"}

Return the package preferences or the default preferences, if no package locale set.

\subsubsection{Subcommand "loaded"}

The list of locales loaded for this package is returned.

\subsubsection{Subcommand "isset"}

Returns true, if a package locale is set.

\subsubsection{Subcommand "unset"}

Unset the package locale and use the default state for the package. Load all message catalog files of the package for locales, which were not present in the package loadedlocales list and are present in the default list.

\subsubsection{Subcommand "present"}

Returns true, if the given locale is loaded

\subsubsection{Subcommand "clear"}

Set the current loaded locales list of the package to preferences and unset all message catalog keys of the package with locales not included in the package preferences.

\subsection{Package Configuration Options}

Each package may have a set of configuration options set to invoke certain actions. They may be retrieved or changed with the following new command:

\begin{quote}
\textbf{msgcat::mcpackageconfig} \textit{subcommand option} ?\textit{value}?
\end{quote}

Available subcommands are:
\begin{description}
\item[get]
Get the current value of the option or an error if not set.
\item[isset]
Returns true if option is set.
\item[set]
Set the given value to the option. May have additional consequences and return values as described in the option section.
\item[unset]
Unset the option.
\end{description}

Available options are:

\subsubsection{Package Option "mcfolder"}

This is the message folder of the package. This option is set by mcload and by the subcommand set. Both are identical and both return the number of loaded message catalog files.

Setting or changing this value will load all locales contained in the preferences valid for the package. This implies also to invoke any set loadcmd (see below).

Unsetting this value will disable message file load for the package.

If the locale valid for this package changes, this value is used to eventually load message catalog files.

Message catalog files are always sourced in the namespace of the package registering the value.

\subsubsection{Package Option "loadcmd"}

This callback is invoked before a set of message catalog files are loaded for the package which has this property set.

This callback may be used to do any preparation work for message file load or to get the message data from another source like a data base. In this case, no message files are used (mcfolder is unset).

See chapter callback invocation below. The parameter list appended to this callback is the list of locales to load.

If this callback is changed, it is called with the preferences valid for the package.

\subsubsection{Package Option "changecmd"}

This callback is invoked when a default local change was performed. Its purpose is to allow a package to update any dependency on the default locale like showing the GUI in another language.

Tk may be extended to register to this callback and to invoke a virtual event.

See the callback invocation section below. The parameter list appended to this callback is mcpreferences. All registered packages are invoked in no particular order.

\subsubsection{Package Option "unknowncmd"}

Use a package locale mcunknown procedure instead of the standard version supplied by the msgcat package (msgcat::mcunknown).

The called procedure must return the formatted message which will finally be returned by msgcat::mc.

A generic unknown handler is used if set to the empty string. This consists in returning the key if no arguments are given. With given arguments, format is used to process the arguments.

See chapter callback invocation below. The appended arguments are identical to mcunknown.

\subsubsection{Callback Invocation}

Callbacks are invoked under the following conditions:

 the callback command is set, the command is not the empty string, the registration namespace exists.

Any error within the callback stops the operation which invoked the callback. This might be surprising, as the error might be in another package.

\subsection{Test if Message Key is Set}

Message catalog keys may be expensive to calculate and thus may be set on demand.

The following new procedure returns false, if mc would call mcunknown for a key:

\begin{quote}
\textbf{msgcat::mcexists} \textit{src}
\end{quote}

There are two options, to limit the key search to just the current namespace (don't search in parent namespaces) and just the current locale (don't search the preferences but the first item):

\begin{quote}
\textbf{msgcat::mcexists} ?\textbf{-exactnamespace}? ?\textit{-exactlocale}? \textit{src}
\end{quote}

\subsection{forget package}

A package may clear all its keys and state using the new command:

\begin{quote}
\textbf{msgcat::mcforgetpackage}
\end{quote}

\subsection{Locale and Preferences Format}

Locales set by mcset may eventually not correspond to the current preferences, as the preferences are treated as follows:

 put to lower case, remove any multiple ``\_'' and any ``\_'' at the beginning or at the end of the

 locale.

It is proposed, that:

 the locale and the first preferences element is always identical to the

 lowercase passed locale,

 any multiple ``\_'' are seen as one separator.

Example: preferences of locale ``sy\_\_cyrl\_win''
\begin{description}
\item[current preferences]
``sy\_cyrl\_win sy\_cyrl sy'' proposed preferences: ``sy\_\_cyrl\_win sy\_\_cyrl sy''.
\end{description}

Alternatively, all locales may normalized using the upper algorithm, which felt heavy in computation with little gain.

\section{Example Usage}

\subsection{Example from TIP \#399}

Imagine an application which supports the current user language and French, German and English. An external package tp is used. The package uses msgcat and installs itself during the package require tp call:
\begingroup\small\begin{verbatim}
package require msgcat
msgcat::mcload [file join [file dirname [info script]] msgs]
\end{verbatim}
\endgroup

An implementation of the application with the current msgcat 1.5.0 would require the following initialization sequence:
\begingroup\small\begin{verbatim}
package require msgcat
package require np
\end{verbatim}
\endgroup

and the following code to change the locale to French:
\begingroup\small\begin{verbatim}
package forget np
msgcat::mclocale fr
package require np
\end{verbatim}
\endgroup

Using the extension of this TIP, one may load as usual:
\begingroup\small\begin{verbatim}
package require msgcat
package require np
\end{verbatim}
\endgroup

and to change to french locale:
\begingroup\small\begin{verbatim}
msgcat::mclocale fr
\end{verbatim}
\endgroup

The first time, a locale is required, all corresponding message files of all packages which use msgcat get loaded. This might be a heavy operation.

If a locale is reactivated (and the message catalog data was not cleared), it is a quick operation.

Without this TIP, it is computational expensive (if possible, as many packages are not reloadable or a reload may disturb current processing, e.g., by forcing the closing of sockets, etc.).

\subsection{Change with No Need to Come Back}

If it is certain that a locale is changed and the then obsolete data is of no use, one may clear unused message catalog items:
\begingroup\small\begin{verbatim}
msgcat::mclocale fr
msgcat::mcloadedlocale clear
\end{verbatim}
\endgroup

\subsection{Use a Callback to be Notified About a Locale Change}

Packages which display a GUI may update their widgets when the locale changes. To register to a callback, use:
\begingroup\small\begin{verbatim}
namespace eval gui {
 msgcat::mcpackageconfig changecmd updateGUI

 proc updateGui args {
 puts "New locale is '[lindex $args 0]'."
 }
}

% msgcat::mclocale fr
fr
% New locale is 'fr'.
\end{verbatim}
\endgroup

\subsection{To Use Another Locale Source than Message Catalog Files}

If locales (or additional locales) are contained in another source like a data base, a package may use the load callback and not mcload:
\begingroup\small\begin{verbatim}
namespace eval db {
 msgcat::mcpackageconfig loadcmd loadMessages
 msgcat::mcconfig loadedpackages\
 [concat [msgcat::mcconfig loadedpackages] namespace current]

 proc loadMessages args {
 foreach locale $args {
 if {[LocaleInDB $locale]} {
 msgcat::mcmset $locale [GetLocaleList $locale]
 }
 }
 }
}
\end{verbatim}
\endgroup

\subsection{Use a package locale}

The reference implementation also contains a changed clock command which uses a package locale. Here are some sketches from the implementation.

First, a package locale is initialized and the generic unknown function is activated:
\begingroup\small\begin{verbatim}
msgcat::mcpackagelocale set
msgcat::mcpackageconfig unknowncmd ""
\end{verbatim}
\endgroup

If the user requires the week day in a certain locale, it is changed:
\begingroup\small\begin{verbatim}
clock format clock seconds -format %A -locale fr
\end{verbatim}
\endgroup

and the code:
\begingroup\small\begin{verbatim}
msgcat::mcpackagelocale set $locale
return [lindex [msgcat::mc DAYS_OF_WEEK_FULL] $day]
### Returns "mercredi"
\end{verbatim}
\endgroup

Some message-catalog items are heavy in computation and thus are dynamically cached using:
\begingroup\small\begin{verbatim}
proc ::tcl::clock::LocalizeFormat { locale format } {
 set key FORMAT_$format
 if { [::msgcat::mcexists -exactlocale -exactnamespace $key] } {
 return [mc $key]
 }
 #...expensive computation of format clipped...
 mcset $locale $key $format
 return $format
}
\end{verbatim}
\endgroup

\section{Reference Implementation}

See Tcl fossil tag msgcat\_dyn\_locale \cite{tip1}.

\section{Compatibility}

Imagined incompatibilities:

 If packages call mcload multiple times with different folders, the

 data was currently appended. This is still the case, but only the last folder is used for any reload. The property \textbf{mcfolder} may be transformed to a list to cover this case.

 The return value of mcload (file count) may be much higher as there

 may be loaded much more files. I suppose, this value is only used by the test suite to verify functionality and is not for big general use.

 Message files may not be aware, that they may be loaded at any moment and

 not only after their own \textbf{mcload}. I suppose, this is the biggest issue but I think, there is no alternative.

 Message files do not get reloaded any more, if a second mcload is

 issued with the same path argument.

 Package which temporary change the default locale trigger any callback

 and may lead to user visible side effects.

\section{Issues}

Known issues:

 Packages might not be aware of a locale change and may buffer

 translations outside of \textbf{msgcat}. Packages should not buffer msgcat messages if they are used in a dynamic locale application (like tklib tooltip does for example).

 The clock command currently has a small dynamic patch for msgcat

 implemented. This must be removed in favor to new msgcat features due to the temporarily change of the default locale.

\section{Extensions}

 Expose the function to calculate the preference list from a given locale. Load a message catalog file for a given locale without changing the

 default/package locale.

 Methods isloaded to check if a locale is currently loaded. Access message catalog with specified namespace, locale and search

 behavior.

\section{Alternatives}

The alternative is the former \cite{tip399}, but that is problematic because the list of locales must be known before any package load. The additional complexity of this TIP is a justifiable trade-off against the greatly improved flexibility in the loading and locale selection order.

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\begin{thebibliography}{TIP \#2}
\addcontentsline{toc}{section}{References}
\bibitem[TIP \#1]{tip1}
TIP Editor,
\emph{TIP Index},
on-line at \url{http://purl.org/tcl/tip/1.html}
\bibitem[TIP \#399]{tip399}
Harald Oehlmann,
\emph{Dynamic Locale Changing for msgcat},
on-line at \url{http://purl.org/tcl/tip/399.html}
\end{thebibliography}
\end{document}
