Figure 1 - Role of Tkhtml
Tkhtml is a Tk widget written in C that displays documents formatted according to the HTML and CSS standards. This paper reports on the recent work on Tkhtml 3.0 sponsored by Eolas Technologies. The motivations for creating a standards compliant HTML/CSS widget for Tcl/Tk are examined, expected uses of the widget identified and the scope of functionality provided defined. Also discussed is the web browser hv3, developed as part of the Tkhtml project to test the HTML widget. Future extensions and enhancements to Tkhtml are contemplated.
Adding a widget to display documents formatted using the Hyper-Text Markup Language (HTML) [1] to Tk is not a new idea. There are currently at least three widely used widgets for just this purpose, Tkhtml 2 [2], htmllib [3] and the scrolledhtml widget included in the Iwidgets package [4]. The most common use is to embed hyper-linked documentation in an application ("help" documentation), although there exists at least one fully-functional web browser application built on top of Tkhtml 2, browsex [5]. Both htmllib and scrolledhtml are written in pure Tcl, making them particularly easy to deploy as part of Tcl applications. Tkhtml 2 is written in the C language, and has been used by applications written in Tcl, Ruby, Python and possibly others.
All existing HTML renderers for Tcl/Tk have one thing in common: A significant subset (possibly even a majority) of documents found on todays Internet are not rendered as the author intended. This is not due to major technical flaws in existing packages, but simply because the web has moved on from pure HTML. In 2006, most web documents use a combination of HTML and Cascading Style Sheets (CSS) [6] to define the presentation of documents. The requirement to support CSS is the reason Tkhtml 3 was created instead of channeling effort into improving Tkhtml 2 or another existing package. In many ways, it now seems that "TkCSS" would have been a more descriptive name than "Tkhtml".
Figure 1 roughly depicts the functionality provided by the Tkhtml 3 package. The application configures a Tkhtml widget by specifying:
The widget displays a scrollable viewport that may be used to view the rendered document. Various introspection interfaces to query the document tree, CSS property values and document layout are also available.
The role of HTML has changed slightly since it's inception. Last century, HTML was used to define both the content and presentation of documents for display. More recently, the task of specifying presentation has been abrogated to CSS and HTML constructs solely concerned with presentation have been deprecated by the World Wide Web Consortium (WC3) [7]. Content and structure are still specified using HTML. Of course, many documents do not conform to these guidelines and so any HTML widget is required to support both CSS and the older, deprecated HTML presentational attributes.
As of version 2.1, CSS consists of 108 separate properties, around 90 of which are applicable to a visual display agent like Tkhtml (other properties concern aural or fixed-page document renderings). Based on rules specified in various CSS documents (including the default stylesheet, see below), each document element is assigned a value for each applicable property. Rules may specify that certain property values be assigned to elements based on their type, attributes or their position in the document tree relative to other elements, The values of the assigned properties determine the presentation of the document element. For example:
Internally in Tkhtml, document layout is determined entirely by CSS properties. The section entitled "The Default Stylesheet" below clarifies the approach to handling legacy document that use deprecated HTML presentation attributes.
Despite lack of recognition as such, Tcl/Tk remains one of the most portable and complete application platforms available today. Complex graphical applications move seamlessly between embedded devices and many different personal computer configurations. However, it is not currently possible to create a web browser application. The primary purpose of Tkhtml is to facilitate the development of web browser applications using Tcl/Tk.
The above paragraph brazenly states that it is not possible to create a web browser application using Tcl/Tk, which is obviously not a true statement, if only because the introduction of this paper refers to a "fully-fledged web browser", browsex. A casual inspection of browsex reveals a stable cross-platform application, with similar functionality to all but the latest generation of popular web browsers [20,21] Performance is an order of magnitude better than that of popular browsers, both in terms of initialization time and resource consumption. Despite this, browsex has been all but abandoned (no updates since January 2003). The reason for this is simple: because it uses an older generation of HTML widget, a large number of web pages are rendered incorrectly. Tkhtml aims to address this by supporting modern versions of the relevant standards. Therefore:
The primary purpose of Tkhtml is to provide an HTML/CSS rendering engine of sufficient accuracy and flexibility to facilitate the development of high quality, standards compliant web browser applications for the Tcl/Tk platform.
Although targeted at web browser applications, other uses are possible. It is anticipated that applications may use Tkhtml for the following broad categories of purpose. All categories of application are taken into consideration when design or API decisions are taken, although in practice it is fair to say that the other application pattern appear to demand a subset of the functionality of the web browser.
This section contains a high level conceptual description of the API provided by Tkhtml and in so doing communicates the scope and flexibility of the provided functionality. The goal of this section is not to provide a programming tutorial, but to clarify the role of Tkhtml in the Tcl/Tk software ecosystem.
Along with the application patterns in the preceding sections, the API was designed with the following general principles in mind:
# Create a shrink-wrapped HTML "label" set html [html $pathName -shrink true] pack $html # Create a document frame (default size 800x600) set html [html $pathName] pack $html -fill both -expand true
Using a Tkhtml widget is similar to using any other Tk widget. An instance must be created and handed off to a geometry manager for display. Usually, the widgets requested height and width are set by the -width or -height options in the same way as they are for built-in widgets. The exception is if the boolean -shrink option is set to true. In this case the widgets requested width and height are the size of the currently rendered document. The current widget width, or value of the -width option if the widget is unmapped, is used as the desired width when calculating the layout in this mode.
The global [html] command is currently used to create new widget instances. This may be changed or moved into a namespace just before Tkhtml moves to beta stage.
Note:The example code in figure 3 assumes that the Tcl variable $pathName contains a suitable window name for a new widget. Subsequent example code (figures X, X and X) assume that the Tcl variable $html is the name of an existing HTML widget.
Before anything can be displayed, the widget must parse a single HTML document and zero or more stylesheets. HTML documents may be incrementally parsed and displayed, but each stylesheet document must be parsed atomically. All documents must be provided to Tkhtml as UTF-8 encoded text.
# Parse two HTML document fragments.
$html parse { <html> <body> <p> An abruptly }
$html parse -final { terminated HTML document }
# Parse a CSS stylesheet
$html style -id author-0001 { p {font-size:80%} }
# Discard all parsed documents and stylesheets
$html reset
The example code in figure 4 demonstrates the interfaces used to pass HTML or CSS documents to an HTML widget for parsing. HTML documents may be incrementally parsed by making multiple calls to the [parse] sub-command. However, the final call to the [parse] sub-command should be qualified by the -final option so that the widget knows that the document has been completely parsed. This is so that any open elements may be considered to be implicitly closed and any configure actions (see "Element Handlers" below) taken. The -final option is required even if the caller is sure that the specified document is well-formed. This is because of the esoteric way HTML parsers handle the <html>, <body> and <head> elements: these elements may not be considered completely parsed until the end of the document as defined by the source (i.e. HTTP server) is reached. Once a call to the [parse] subcommand has been made with the -final switch specified, [parse] may not be called again until the [reset] sub-command is invoked. The [reset] sub-command resets the widget to it's initial state, ready to load a new document.
In contrast to HTML documents, CSS documents may not be incrementally parsed, although multiple CSS documents may be loaded concurrently. CSS documents are passed to the widget for parsing using the [style] command. As shown in figure 4, The caller may optionally specify an "id" for the stylesheet by qualifying the [style] command with the -id option. The stylesheet id determines the priority of each stylesheet for use in the CSS "Cascade" algorithm. Essentially, each stylesheet id must begin with the string "author" or "user", indicating whether the stylesheet was provided by the document author or the application end-user. Stylesheets provided by the application user are given a higher priority. If two or more stylesheets are provided by the document author or application user, then the stylesheet with the lexicographically lesser stylesheet id is considered to be of higher priority. A fuller explanation and an example illustrating a method for completely implementing the CSS cascade is provided in the Tkhtml man page [10].
Each time a new stylesheet document or HTML document fragment is passed to the widget the display is updated automatically next time the event loop is entered.
P {display: block; margin: 1em 0;}
IMG[align="left"] { float:left }
IMG[align="right"] { float:right }
As well as stylesheets explicitly provided by the user, document layout is governed by the "default stylesheet" document. At any one time there is a single default stylesheet, configured by setting the value of the -defaultstyle option. The default stylesheet defines the display properties that are built-in to the document language. For example, in HTML a <p> element is expected to generate a block level box with vertical margin spacing. The default stylesheet (see figure 5) contains a CSS declaration to make this so.
As shown in figure 5, the default stylesheet also contains rules for assigning CSS properties based on HTML attributes. The final two rules shown in figure 5 demonstrate mapping between the "align" attribute specified on <img> elements to the CSS 'float' property. As a result, such <img> elements generate floating boxes, as specified by HTML 4.01. For cases where it is not possible to set appropriate CSS properties using standard CSS grammar (or rather, where it would be extremely cumbersome to do so), the default stylesheet may include Tcl code to calculate property values based on HTML attributes and other considerations. This facility is only available to the default stylesheet, not stylesheets parsed using the widget [style] sub-command. This is for security reasons. If it were not the case, then malicious scripts could be embedded in documents.
Tkhtml includes two built-in default stylesheet documents which the application must manually choose between. One implements HTML "standards mode", the other "quirks mode". Tkhtml does not automatically switch between these based on DOCTYPE tags, but is possible to implement this at the application level. Hv3 (see below) implements Gecko compatible "DOCTYPE sniffing" and selects one of the two default stylesheet documents based on the results.
As of CSS 2.1, almost all presentation rules of HTML may be implemented using CSS. Where this is not the case (primary examples are the "rowspan" and "colspan" attributes of table cells) Tkhtml provides minor extensions to CSS to work around it. It seems likely that these proprietary extensions will be removed and replaced by standard idioms as CSS 3 matures.
In theory, this means that it should be possible to use Tkhtml to display arbitrary XML documents, as long as a default stylesheet is supplied to define the built-in rules of the XML based document language. However this is not currently possible for the following two reasons, both related to the HTML parser inherited from Tkhtml version 2:
It is hoped that these problems, along with minor bugs caused by parsing XHTML using the HTML parser, will be addressed in the near future.
<html>
<head></head>
<body>
<h1>A heading</h1>
<p>A paragraph</p>
</body>
</html>
Once an HTML document or part thereof has been parsed, a document tree is generated internally. This tree has one node for each element and one node for each contiguous block of text and/or whitespace. All whitespace present in the original document is also present in the document tree. Figure 6 depicts an HTML document and the corresponding document tree generated by Tkhtml.
Tcl scripts access the document tree using node handles (each handle is a Tcl command). Node-handles may be obtained for element, text and whitespace nodes (text and whitespace nodes are really the same thing). Each node-handle provides access to the element type and HTML attributes for an element node, or to the content text of a text or whitespace node. Node-handles also provide an interface to obtain the handles of a nodes parent or children, allowing document tree browsing.
Although this has yet to be implemented at time of writing, eventually node-handles will provide an interface that may be used to programmatically modify the structure of the document tree. Web browser applications require such an interface to implement the Document Object Model (DOM) [11], a standardized interface used by dynamic web pages and AJAX [12] applications.
# Query for the document root:
set node [$html node]
# Loop through all <a&rt; elements
# with the "href" attribute set:
foreach node [$html search {a[href]}] {
...
}
# Loop through all nodes that generate
# content currently displayed at the
# coordinates (100,120) of the viewport.
foreach node [$html node 100 120] {
...
}
Once a single node-handle is obtained, it may be used to browse the tree. For example, an application can perform a linear search of the document by first obtaining the root node and implementing a depth-first traversal algorithm.
Supporting the third query mode enumerated above, query by viewport coordinates, allows Tkhtml to elegantly defer much functionality that one might naturally associate with an HTML widget to the application level. For example, Tkhtml itself contains no code to implement document hyper-links. Instead, applications (or mega-widget packages) are expected to take the following approach:
[bind] command),
associate a callback script with a button-click on the Tkhtml widget.
It could be argued that since the majority of applications that use an HTML widget require hyper-link functionality, the procedure described above should be implemented as part of the widget code. However the approach describe above is preferred because it allows the application maximum flexibility in determining exactly which elements are hyper-links, what exactly constitutes activating a hyper-link and the nature of the action taken in response. It is hoped that at some stage a pure Tcl mega-widget based on Tkhtml will emerge that provides pre-packaged versions of this and other commonly required functionality.
Element handlers provide an alternative to querying for node handles for scripts that perform special processing based on specific types of document elements. Element handlers are Tcl scripts registered with the widget that are invoked when a particular type of element (i.e. <h1> <p> etc.) is parsed. Three different types of element handlers may be registered. All are invoked by Tkhtml from within calls to the [parse] sub-command. Each element type may have a maximum of one of each of the following (although not all combinations make sense):
<html>
<head>
<style>
h1,h2,h3 { color: green; }
p { color: red; }
</style>
</head>
<body>
<table>
<form action="query.php">
<tr><td> Search:
<td> <input name="value" type=text>
</table>
<input name="Go" type=submit>
</form>
</body>
</html>
Figure 8 contains a malformed but nevertheless fairly typical HTML document. An application parsing the document could register a script handler to intercept the contents of the <style> element and pass them to the widget [style] sub-command. A node handler callback could be registered to detect and perform special processing for the form controls declared by <input> elements. A parse handler could be used to handle the <form> and </form> tags themselves. Although in any reasonable document tree constructed from the example document the second <input> node is not descended from any <form> node, by convention it should be associated with the open <form> tag. Although they may also serve other purposes, parse handlers are primarily to allow applications to make this kind of association.
Tkhtml may include images in rendered documents under three circumstances:
IMG[src] { -tkhtml-replacement-image: attr(src) }
IMG { -tkhtml-replacement-image: "" }
In the first two cases, the relevant CSS property is set to the URI of the required image. For the third case, Tkhtml defines a custom CSS property, '-tkhtml-replacement-image' which is set to the URI of the elements replacement image, if any. For HTML documents, the default stylesheet (see above) includes a rule that maps the contents of the "src" attribute to the '-tkhtml-replacement-image' property for <img> elements. This rule is shown in figure 9.
If image support is required, the application must configure the widget -imagecmd option with a Tcl script that the widget invokes each time an image is required. The script is passed the URI value for the required image (i.e the computed value of the 'background-image', 'list-style-image' or '-tkhtml-replacement-image' property) and must return a Tk image to be used for display. If required, the HTML widget may create scaled copies of the image internally, but will never modify the original. The application is free to modify the size or contents of the Tk image at any later time, and the HTML widget display is updated accordingly. This can be used to implement asynchronous or incremental retrieval of image files, or to implement animated images. The URI passed to the -imagecmd script may be either relative or absolute.
If the -imagecmd script is passed a relative URI, then it should be resolved relative to the HTML document. This can lead to ambiguity if a stylesheet obtained from a different location from the HTML document includes a relative URI (in this case the URI should be resolved relative to the location of the stylesheet, not the document). To resolve this, when a script passes a stylesheet to the widget to be parsed using the [style] sub-command, it has the option of specifying a resolver script. If one is specified, then all relative URIs encountered in the stylesheet are passed to the resolver script to be converted to absolute URIs before the -imagecmd script is invoked.
Tkhtml allows applications to specify a Tk window to "replace" a nominated document node. When a node is replaced (by a replacement object), the content of the document node and any descendant nodes is not displayed. Instead, the specified Tk window is mapped into the widget display, using the CSS rules for "replaced elements". HTML form controls and Plugins are the two most common examples of replaced elements required by a web browser application.
The full replacement object interface allows the application to specify zero or more of the following for each document node:
CSS defines three dynamic pseudo-classes that may be specified as part of a CSS rule so that the specified properties are only applied when the dynamic condition is true. The three conditions are "hover", "active" and "focus", but CSS does not provide any guidelines as to when each condition can be considered to be true. Most CSS applications consider the "hover" condition to be true for a document node whenever the mouse cursor is positioned over content generated by the node. This can be used to implement so-called rollover effects (for example underlining hyper-link text when the mouse cursor is on top of it).
bind $html <Motion> "hover $html %x %y"
proc hover {html x y} {
# Clear the "hover" flag on all nodes on
# which it is currently set.
foreach node [$html search :hover] {
$node dynamic clear hover
}
# Set the hover flag on all nodes that
# generate content at the specified
# coordinates, and all ancestors of
# said nodes.
foreach N [$html node $x $y] {
for {} {$N != ""} {set N [$N parent]} {
$N dynamic set hover
}
}
}
Because they are not defined by CSS, Tkhtml does not attempt to assign interpretations to the dynamic pseudo-selectors. Instead, an interface is provided so that applications can set and clear dynamic conditions on a per node basis using the node-handle interface. For example, to support dynamic :hover selectors, application code may perform the following steps:
Figure 10 illustrates Tcl code to implement this procedure. Tkhtml automatically updates the display when dynamic conditions are modified.
Tkhtml treats the :link and :visited pseudo-selectors in the same way as :hover, :active and :focus.
Tkhtml also supports interfaces that an application may use to query and interact with the text displayed by the widget. Essentially three functions are provided:
# Search for the first instance of
# $string in the document currently
# displayed by Tkhtml widget $html.
# If an instance of $string is found,
# cause it to be displayed in white on
# a black background
proc highlight {html string} {
set plain [$html text text]
set offset1 [string first $string $plain]
if {$offset1 >= 0} {
set offset2 $offset1
incr offset2 [string length $string]
set idx1 [$html text index $offset1]
set idx2 [$html text index $offset2]
$html tag add highlight $idx1 $idx2
$html tag configure -bg black -fg white
}
}
In a web browser application, these interfaces may be used to implement two common functions: selection of text with the pointer device and the "Find in Page..." function used to search for text within the current document. For example, to search for and then highlight instances of a supplied string, an application may follow the following procedure:
Figure 11 contains Tcl code to accomplish the above procedure.
Html Viewer 3 (Hv3) is a simple web browser based on Tkhtml 3.0. Hv3 is developed in pure Tcl and is part of the same source tree as Tkhtml. Especially considering that the goal of the Tkhtml project is not to develop a web-browser, but simply an HTML widget, significant time has been spent on Hv3. This is because Hv3 is required to test Tkhtml. Without a functional web browser, it would be extremely difficult to test the HTML widget, both in terms of correctness and performance. As time passes, it is hoped that Hv3 can attract a user-base that will act as informal testers and generate bug reports for Tkhtml. This, rather than attempting to develop comprehensive regression tests in-house, is the strategy successfully used by other open-source rendering components [13,14].
As a general purpose web-browser, Hv3 has two major shortcomings preventing widespread adoption: it does not support javascript (or any other scripting language), and nor does it support web-browser Plugins such as Macromedia Flash. However, it does support tabbed browsing, integrated web-search, find-as-you-type, auto-completion in the location entry field and several other modern features. Hv3 supports enough of the HTML forms and HTTP cookies mechanisms to use web-applications like GMail and Yahoo Mail.
Hv3 requires only Tkhtml and Tcl/Tk version 8.4 or greater to run, but can optionally use the following extensions to enhance user experience:
The modified HTTP proxy, hv3_polipo, is by far the most important of the three optional extensions. Using hv3_polipo provides the following benefits:
Current released versions of Tkhtml are marked as "alpha", indicating that features or APIs may still be removed, modified or added. Tkhtml will move to "beta" stage after an API freeze is deemed possible. The two biggest obstacles to this are described below: changes to facilitate support of the DOM and changes to support CSS compatible management of mapped windows. Other APIs may also be extended or refined ,but at this stage seem likely to remain as they currently are.
Tkhtml 3.0 is not yet a complete implementation of CSS 2.1. However, it correctly renders a majority of documents found on the WWW. Correctly, in this case, is defined as identically to other modern web browsers [20,21] with javascript support disabled. Where problems do exist, they are usually due to bugs in the implementation or specification interpretation, not unsupported CSS features. The correctness of the document layout produced by Tkhtml is not currently considered a barrier to moving to "beta" release stage.
Hv3 (see above) provides evidence of the suitability of Tkhtml for developing "scriptless" browsers (browsers that do not support javascript or any other scripting language). Although there is room for improvement, widget performance in terms of speed and memory consumption is competitive with those used by major web browsers. There is no reason to believe that this should not be significantly improved after Tkhtml is moved to beta stage and optimization attempted.
The biggest obstacle to moving Tkhtml into a "beta" stage is that integration with an implementation of the DOM has not yet been properly considered or prototyped. Because some DOM APIs query and manipulate data structures that Tkhtml currently considers internal, Tkhtml is required to provide APIs that provide equivalent flexibility and functionality.
The relevant DOM APIs (DOM level-2 Modules "Core" and "Style") provide web applications with an interface to manipulate and modify the document tree and to explicitly override the computed style of document nodes with externally supplied property values. New document tree nodes may be inserted, and existing nodes may modified or deleted. The HTML widget is required to update the viewport following a modification to the underlying data model. Sometimes this involves recalculating layout, sometimes not. Since a modern web browser is not complete without a standards compliant DOM implementation, Tkhtml must support this efficiently to be considered fit for it's primary purpose.
At time of writing, the performance of the current design under these circumstances is an unknown quantity. This represents the most significant identified risk to the project going forward (that it will take too long to redraw the display after the document tree is modified by a DOM application). Some positive indications can be gleaned from the performance of the features described under "Dynamic CSS Effects" above, but no definite statements made.
The definitions of the APIs described under "Element Handlers" above also require clarification to account for the possibility that the document tree might be constructed and elements of interest added in ways other than by parsing an HTML document.
As described above in the "Replaced Objects" section, Tkhtml allows applications to specify Tk windows to be displayed in place of a specified document node. Difficulties arise in the following situations:
The first problem enumerated enough can be solved at the application level by requiring the calling script to specify a pixel offset to be used for baseline alignment. This is a little unsatisfactory, but possible.
The second problem is more difficult to solve. The current solution produces strange results in some circumstances, but given that few web pages expose the problem, may be acceptable in the short term. Alternatively, it may be that applications will be given the option to supply special widgets that support a clipping API as mapped windows to the Tkhtml widget.
| [1] | HTML 4.01 Specification, http://www.w3.org/TR/html4/ |
| [2] | Tkhtml version 2, http://www.hwaci.com/sw/tkhtml/index.html |
| [3] | htmllib, http://noucorp.com/tcl/utilities/htmllib/ |
| [4] | scrolledhtml - Create and manipulate a scrolled text widget with the capability of displaying HTML documents. http://www.tcl.tk/man/iwidgets3.0/scrolledhtml.n.html |
| [5] | BrowseX Home Page, http://www.browsex.com |
| [6] | Cascading Style Sheets, level 2 revision 1, http://www.w3.org/TR/CSS21/ |
| [7] | World Wide Web Consortium, http://www.w3.org/ |
| [8] | The Second Acid Test, http://www.webstandards.org/files/acid2/test.html |
| [9] | Html Viewer 3 - Tkhtml test application, http://tkhtml.tcl.tk/hv3.html |
| [10] | tkhtml(n), http://tkhtml.tcl.tk/tkhtml.html |
| [11] | Document Object Model (DOM) Specifications, http://www.w3.org/DOM/DOMTR |
| [12] | Ajax: A New Approach to Web Applications, http://www.adaptivepath.com/publications/essays/archives/000385.php |
| [13] | Mozilla Layout Engine, http://www.mozilla.org/newlayout/ |
| [14] | The WebKit Open Source Project http://webkit.org/ |
| [15] | Tls SourceForge Project, http://tls.sourceforge.net/ |
| [16] | SourceForge.net: tkImg, http://sourceforge.net/projects/tkimg/ |
| [17] | Portable Network Graphics (PNG) Specification (Second Edition), http://www.w3.org/TR/PNG/ |
| [18] | Pixane, http://www.evolane.com/software/pixane/index.html |
| [19] | Polipo - a caching web proxy, http://www.pps.jussieu.fr/~jch/software/polipo/ |
| [20] | Firefox - Rediscover the Web, http://www.mozilla.com/firefox/ |
| [21] | Opera browser: Homepage, http://www.opera.com/index.dml |