This article presents a case study of the use of meta-programming in Java compatibility testing. It shows how parts of the source code can be shared between different products and modified to generate programs targeting specific functions and describes the approach Sun Microsystems has used for building Technology Compatibility Kits (TCK) for more than five years.

In modern Java TCKs, testing components for multiple products are stored in a single repository not as pure Java language, but as XML files where code is annotated with attributes called metadata. A set of tools is used to select only tests that are applicable for a particular product and transform them into final Java programs. These post-processing tools written in Java provide a simple and efficient way to generate testing components that match specific product needs.

Here we'll illustrate Sun's approach by providing examples of test storage formats and the processing scheme used by the conversion tools. We'll also draw a comparison with Sun's former approach, which used Perl for test generation, and explain why Sun moved to the XML-based methodology.

TCKs
The international community develops and evolves Java technology specifications using the Java Community Process (JCP). The JCP produces high-quality specifications in Internet time, using an inclusive consensus-building approach that produces a specification; a reference implementation that demonstrates that the specification can be implemented; and a technology compatibility kit, or TCK, a suite of tests, tools, and documentation used to test implementations for compliance with the specification.

There are as many TCKs as there are Java technologies, and these technologies might be deeply intertwined despite their differences. For example, both the Java Platform Standard Edition (SE) and the Java Platform Micro Edition (ME) include the Java Virtual Machine (JVM) and the java.lang package, so most tests written for the Java SE VM can be reused to test the Java ME VM.

Each separate TCK has its own requirements for tests such as performance targets or maximum memory size. As a rule, the kernel of the test is the same across TCKs, but the test might take different forms. For example, for Java SE several tests can be combined in one class; while for Java ME, due to a restriction on class file size, each test case should be put in its own class.

Sun Microsystems has a long history of successful TCK development. During the past 10 years, about 100 TCKs with millions of tests were shipped. This success would have been impossible without effective reuse or sharing of tests between TCKs. At the heart of the approach applied at Sun for TCK development is meta-programming, which implies that sources are generated from metadata.

This article describes how this approach is implemented. Even though the area of conformance testing is narrow, the underlying principles can be generalized:

• All tests are stored as templates, not as tests to be included in the product.
• Each test is represented as code with attributes (meta information).
• Templates are stored in a single repository shared across different TCKs.
• Tests applicable for a particular technology are selected by attributes at the TCK build stage based on selection criteria set by the TCK.
• All TCKs use the same tools for extracting test sources from templates, but tools can be customized according to particular TCK needs.
• All TCKs abide by the general policy regulating the rules for writing tests.

• Test Monitor: Tool for test configuration and execution
• Tests: Programs returning either a passed or failed status
• Libraries: Code shared by tests
• Product Documentation

  Each test consists of:
  

• Test sources: Java source code and data files
• Class files: Compiled sources
• Test descriptions: HTML files that consist of Test Monitor instructions showing how to execute tests, links to test sources and data files, and human-readable test documentation

Some tests require arguments be passed or some action performed prior to execution. This information is specified in the Test Description and stored in an HTML file. The Test Monitor reads the Test Description and runs the test according to the instructions. Besides execution instructions, the HTML file contains test documentation, which lets one understand what the test checks for and how without reading the sources.

Here is a sample fragment of an HTML file:

Test Specifications and Descriptions for Applet
       * public Applet()
       * public void start() List of tested methods
       * public void stop()
public Applet() (See Table 1 and Table 2)