\documentclass[]{article}
\usepackage{amsmath,graphicx,supertabular,hyperref,tabularx,ifthen}
\title{TIP \#453: Tcl Based Automation for tcl/pkgs}
\date{September 13, 2016}
\author{Sean Woods}
\urlstyle{sf}
\setlength{\parskip}{1ex}
\setlength{\parindent}{0pt}
\def\tipversion$#1${\texttt{\$#1\$}}
\def\tiplangle#1{\ensuremath{<}}
\def\tiprangle#1{\ensuremath{>}}
\def\tipbar#1{\ensuremath{|}}
\def\tipmail#1#2{\(\langle\){\small\expandafter\url{#1@#2}}\(\rangle\)}
\ifx\pdfoutput\undefined
\newcommand{\tipimage}[2]{\typeout{Make sure you download #1.eps}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.eps}}{\includegraphics[width=0.8\textwidth]{#1.eps}}}
\newcommand{\tipxref}[1]{}
\newcommand{\tipxrefend}{}
\else
\newcommand{\tipimage}[2]{\typeout{Make sure you create #1.pdf}\ifthenelse{\lengthtest{0.8\textwidth>#2}\and\lengthtest{0pt<#2}}{\includegraphics{#1.pdf}}{\includegraphics[width=0.8\textwidth]{#1.pdf}}}
\pdfcatalog{/PageMode /UseOutlines}
\newcommand{\tipxref}[1]{\pdfannotlink attr {/C [0.5 0.5 1.0] /Border [0 0 1]} goto name {#1}}
\newcommand{\tipxrefend}{\pdfendlink}
\fi
\newenvironment{tipabstract}{\begin{abstract}}{\end{abstract}}
\begin{document}\maketitle
\begin{center}\begin{tabularx}{\linewidth}{|r@{: }X|}\hline
\textbf{TIP \#453}&\textbf{Tcl Based Automation for tcl/pkgs}\\\hline
Author&

Sean Woods \tipmail{yoda}{etoyoc.com}
\\
Created&Tuesday, \(\text{13}^{\text{th}}\) September 2016\\
Type&Project\\
Tcl Version&\(8.7\)\\
State&Draft\\
Vote&Pending\\
Version&\tipversion$Revision: 1.2 $\\
Keywords&Build tooling\\
Post History&\\
\hline\end{tabularx}\end{center}
\thispagestyle{empty}\pagestyle{empty}
\begin{tipabstract}

This TIP proposes replacing the \textbf{make package} process currently employed by the core with a Tcl-based build automation tool.
\end{tipabstract}
\tableofcontents\setcounter{page}{0}\clearpage\pagestyle{plain}

\section{Background}

\cite{tip376} provides for the distribution of third party packages in the Tcl/Tk core distributions. To support that TIP currently requires three separate build automations: a Makefile based automation in Unix, and both an nmake and Makefile based automation for Windows. These automation systems can get out of sync and they assume that their job is to build dynamic libraries for local installation.

\section{The Pitch}

By the time \textbf{make packages} has run, the local Tcl interpreter has been built already. Rather than rely on delicate hacks and makefile tricks, core distributed packages could be built and installed via exec commands inside a Tcl script. In addition, this same automation could handle functions like injecting a core distributed package into a virtual file system, as well as bundling the Tcl/Tk library file system for \cite{tip430}.

\section{The Implementation}

For the \textit{practcl} branch of tclconfig, I put together a 4000 line self-contained package and kit building library of tools. This library is TclOO based, and provides a rudimentary (but functional) system for templating C code in Tcl, as well as a build system that is capable of nesting sub-projects. It also steals the useful bits from the \textbf{fileutil} module of tcllib, providing implementations for concatenating files, performing file searches, and building a global package index from a soup of modules. The library also has a wrapper to download external sources from fossil. It also contains procs that can compile a static library, dynamic library, or self contained shell directly from exec calls.

I would propose that this tool system (or a new creation by the community in a similar spirit) be included in the library/ section of the tcl core. The provisional name for this tool set would be \textbf{practcl}. A version of this tool could also be provided in tcllib to allow 8.5 and 8.6 based cores to continue to build extensions.

In the new scheme, \textbf{make packages} (in all its forms) would be replaced with a call to ``\textit{\$(TCLSH) \$\{srcdir\}/pkgs/make.tcl build}''. \textbf{make packages-install} would be replaced by a call to ``\textit{\$(TCLSH) \$\{srcdir\}/pkgs/make.tcl install}''. For advanced users, these toplevel commands \textbf{build} and \textbf{install} will accept additional arguments. For instance, to install the core distributed packages into the VFS of a kit: ``\textit{\$(TCLSH) \$\{srcdir\}/make.tcl install -destdir \$\{MyVFS\}/lib}''.

\section{pkgs/make.tcl}

\textbf{make.tcl} would be maintained as part of the core, and provide the top-level control system to build, install, or repackage the core distributed extensions. That script will also provide mechanisms to populate the pkgs file system for developers who build the tcl core from fossil checkouts.

Commands:
\begin{itemize}
\item{}
\textbf{basekit}

Compile a ZipFs style basekit suitable for the \textbf{wrap} command.
\item{}
\textbf{build} ?\textbf{all}? ?\textit{package}? ?\textit{package...}?

Compile the source code for core distributed packages into binary products (as applicable.) If \textbf{all} is given, an attempt is made to compile all packages under \textit{pkgs/}. Any other argument is interpreted to be the name of an individual package to be compiled.
\item{}
\textbf{install} ?\textbf{-destdir} \textit{destinationpath}?

Install all core distributed packages locally. If \textbf{-destdir} is given, install the packages relative to \textit{destinationpath} in the same way that ``\textbf{make DESTDIR=} \textit{destdir}'' would. If \textbf{-destdir} is not given, or is an empty string, perform an install relative to the \textbf{exec\_prefix} in \textbf{tclConfig.sh}
\item{}
\textbf{wrap} \textit{exename} \textit{vfspath} ?\textit{dir...}?

Generate a self contained executable constructed from the virtual file system amalgamated from \textit{vfspath} and any other directories given as arguments. This VFS will automatically be populated with the \textbf{library/} file system from Tcl.
\item{}
\textbf{distribution}
\end{itemize}

\tiprangle{} Download packages listed in the ``packages.tcl'' file, unpack their source code in the /pkgs folder, and perform any steps required to prepare those extensions for inclusion in a core snapshot for distribution.
\begin{itemize}
\item{}
\textbf{developer}
\end{itemize}

\tiprangle{} Download packages listed in the ``packages.tcl'' file, unpack their source code in the parent folder to the one the core has been unpacked from, and perform any steps required to prepare those extensions to be compiled locally as part of a developer build. This is intended for developers who work from fossil checkouts of the Tcl core.
\begin{itemize}
\item{}
\textbf{package-list}
\end{itemize}

\tiprangle{} Stream to stdout a list of all of the packages in the packages.tcl file, in a flat machine readable format.

\section{Tk}

This same mechanism will be adapted for Tk. Tk will be also provide a \textbf{pkgs/} directory. Its base kits will be based on a modified \textit{wish} instead of a modified \textit{tclsh}.

\section{Normal Operation}

During the build/install/etc phase each directory will be scanned for either a ``configure'' file or a ``prac.tcl'' file. Standard TEA extensions will be detected by the presence of a ``configure'' file. The prac.tcl file is a hint to the build system that the package needs either additional instructions and guidance. The contents of the file are interpreted by the object which is implementing the extension's ambassador to the build system.

If one were to decide to bundle tcllib, and wished to exercise its SAK based installer the prac.tcl file would read:
\begingroup\small\begin{verbatim}
# Implement the install routine for tcllib
#
oo::objdefine [self] {
  method install DEST {
    set pkg [my define get pkg_name [my define get name]]
    my unpack
    set prefix [string trimleft [my <project> define get prefix] /] 
    set srcdir [my define get srcdir]
    ::practcl::dotclexec [file join $srcdir installer.tcl] \
      -pkg-path [file join $DEST $prefix lib $pkg] \ -no-examples -no-html -no-nroff \
      -no-wait -no-gui -no-apps
  }
}
\end{verbatim}
\endgroup

\section{Maintaining the Package List}

Each \textbf{pkgs/} file system for both Tcl and Tk will also contain a file \textbf{packages.tcl}. This file will be human and machine readable. It contains a description of every core distributed package, where the sources can be found, as well as which fossil tags can be utilized as either development or release with this particular version of the core.

\textbf{packages.txt} contains a series of keywords populating a data structure.

A simple example would by the tclconfig templates from TEA:
\begingroup\small\begin{verbatim}
EXTENSION tclconfig {
   tag  trunk
   fossil_url http://core.tcl.tk/tclconfig
}
\end{verbatim}
\endgroup

The EXTENSION keyword is intended to take the following arguments:

\begin{quote}
\textit{name} \textit{key/value-configuration-dict}
\end{quote}

\subsection{Reserved keys}

\subsubsection{tag Source code management tag for the release bundled with this edition of the core}

\subsubsection{fossil\_url If the extension is managed via fossil, a url that can be fed to "fossil clone". If no tag is specified "trunk" is assumed.}

\subsubsection{git\_url If the extension is managed via git, a url that can be fed to "git clone". If no tag is specified "HEAD" is assumed}

\subsubsection{file\_url If the extension is only available via source snapshot, a url where the file can be downloaded. Supported formats are .tar.gz and .zip.}

The list is kept separate from the actual \textbf{make.tcl} so that users can simply steal the list for making ``batteries included'' distributions. It also allows the package list to remain distinct for each branch of the core.

\section{Copyright}

This document has been placed in the public domain.

\section{Colophon}

\textit{TIP AutoGenerator --- written by Donal K. Fellows}
\begin{thebibliography}{TIP \#2}
\addcontentsline{toc}{section}{References}
\bibitem[TIP \#376]{tip376}
Don Porter,
D. Richard Hipp,
Kevin Kenny,
\emph{Bundle sqlite3 and tdbc::sqlite3 Packages},
on-line at \url{http://purl.org/tcl/tip/376.html}
\bibitem[TIP \#430]{tip430}
Sean Woods,
Donal Fellows,
Poor Yorick,
Harald Oehlmann,
\emph{Add basic ZIP archive support to Tcl},
on-line at \url{http://purl.org/tcl/tip/430.html}
\end{thebibliography}
\end{document}
