Test Development
Tests aren't only the largest part of any TCK, but the main component. Their number and quality define how well the specification is covered and, as a result, the likelihood of incompatibilities in the implementation. Some TCKs are very large. For example, the TCK for Java SE 6.0 contains more than 100,000 tests. Effective management of such big products is impossible without applying special techniques that let one to do mass updates or easily add or remove tests without breaking other functionality.

To develop a high-quality TCK you have to produce a huge number of tests. There are three ways of creating new tests: developing new ones, reusing existing tests, or automatically generating tests from the specification. The last option is the best, but unfortunately it exists only in theory. Reusing tests is cheaper than developing new ones so the development of most TCKs starts with a search for existing tests that can be used. The more tests you can reuse, the less you need to develop. But when dealing with thousands of files, the task becomes daunting.

Java technologies tend to evolve and most of them have several releases. Each release requires a separate TCK that might include tests for new features as well as bug fixes and tests that improve the coverage in existing areas. So it's important to be able to include tests easily and incorporate newly developed code and bug fixes. Meta-programming can help.

Meta-programming
Wikipedia defines meta-programming as:

The writing of programs that write or manipulate other programs (or themselves) as their data or that do part of the work that is otherwise done at compile time during runtime. http://en.wikipedia.org/wiki/Metaprogramming

Representing the test as data for further processing enables the program to achieve the expected results by modifying control programs of several kilobytes, not by adopting thousands of tests. When all existing and applicable tests are found, tests are decorated according to TCK requirements and HTML files don't contain orphans.

Additionally, not keeping data that can be calculated guarantees consistency and enables the program to write less code. It also enables the generation of similar tests from one template.

Honestly, it's more convenient for engineers to develop code in pure Java than in any other format. While applying this approach will increase development time minimally, it will save resources for further reuse of tests by other TCKs.

Sun represents a TCK test as its code and related attributes (meta-information). In the past Sun used Perl as a language for test data definitions. Developers wrote simple Perl programs consisting mostly of variable settings and invocations of library methods:

require $ENV{TESTGEN}; # import test generation library
$package = "applet";      # meta information describing
$class = "Applet";      # api under test
$method = "public Applet()"; #
$code = '{
a = new Applet();
if (a.isVisible() == true && a.isEnabled() == true
&& a.isValid() == false) { // Check result
return Status.passed("OKAY");
} else {
return Status.failed("Incorrect ' . $class . ' object created");
}
}';
&gen; # invocation of test generation method

All generation functionality was implemented in the Perl library. It was possible to customize generation by defining the TESTGEN environment variable pointing to the alternative library. Behind the simplicity are a number of serious restrictions: the necessity of manually synchronizing generation libraries between different TCKs, the difficulty in reading test attributes, and unchecked variable names.

The XML language fits the purpose of specifying meta-info better. The XML language lets us keep data in a well-defined format unified across all TCKs. In its turn the unified format results in a set of shared tools that implement processing functionality common for all TCKs (see Listing 2). Tools are customizable, so all TCKs use the same tools and just provide their own specific plug-ins.

Figure 1 demonstrates how tests are processed during the build.

Tests for all TCKs are stored in a common Test Repository so newly developed tests as well as bug fixes made by one TCK are available for all TCKs. A set of tools is shared between all TCKs: the Filtering Tool and the Generation Tool. TCK provides test selection criteria to the Filtering Tool, which selects only those tests applicable to the technology. Selected tests are passed to the Generation Tool, which generates test sources as they need to be presented in TCK. The Generation Tool is implemented to extract tests from templates in a way suitable for most TCKs, but it can be customized when needed through test generation customization.

In this approach, TCK needs to define only its own test selection criteria and test generation customization. Test sources and tools can be taken from a general repository.

Implementation
This section covers the technical implementation of shared tools.
• Internal Test Representation: A DTD defines about 50 elements that can be used in XML test descriptions. For each element a corresponding Java class was created. The specially developed parser reads XML test descriptions and creates a Java object representing the test. An XML emitter can write Java objects back to XML. This technique is known as marshaling and unmarshaling. It lets tools use the Internal Representation API and doesn't involve XML.
• Filtering Tool: The purpose of the Filtering tool is to select tests that apply to a specific technology. Based on TCK properties, the Filtering tool creates an instance of AttributeFilter:

public interface AttributeFilter {
    public boolean accept(TestGroup tg);
    public boolean accept(TestGroup tg, TestCase tc);
}