The Ant script in Listing 3 demonstrates a similar use of global properties to ensure consistency and standardization.

Finally, isolate the build scripts to just that: builds. Too often, "build" scripts include substantial pre- and post-build logic unrelated to the build. Pre- and post-build logic can be extremely complex, especially as an application matures and development is being performed on multiple versions simultaneously. The Ant script in Listing 4 demonstrates a build script with a very basic and generic deployment portion.

Rather than writing pre- and post-build logic within a build script (where the functionality is often limited by the scripting language or tool), place the non-build logic in external scripts. The external scripts should be written in a scalable, lightweight, and cross-platform language such as PERL or PYTHON. Tightly focused build scripts can then have built-in hooks to the external build utility. Listing 5 takes the overly complex build script of the prior example and replaces it with a call to an external script.

By partitioning the build scripts in this way, developers (or build masters) who encounter build problems can drill down to the root cause very rapidly. In addition, as development grows in complexity and new languages or target operating systems are added, the in-house build utility can scale more effectively. For example, consider a C and C++ development shop that uses an entirely Make-based build system with all pre- and post-build logic written in the Make scripts. When the development shop decides to add a Java component to their application, they are faced with writing an Ant component (equivalent to their existing Make scripts) that manages all Java-related pre-build, build, and post-build logic. However, if the development shop has a build utility, written in PERL, that executes Make scripts limited to build execution, they only have to write Ant scripts that handle the Java builds, and can use the existing PERL framework as a basis for all of the non-build functionality.

Dealing with Growth
As development environments grow, new distributed-management requirements are introduced and problems grow more complex. Good build management approaches will, for the most part, scale effectively, but there are several new problems that are often added to the mix.

For instance, the more complex the environment, the more likely it is that there will be longer delays between when builds break and when those breaks are identified. Developers might execute builds on their own machine and perform successful unit testing, only for the integration build to break when the developer's code is introduced. The solution is to perform a regular "nightly" build once build scripts are consistent, predictable, and portable. This ensures that each developer's latest revisions will be tested in an integration build so problems are immediately identified. Another solution is to place all enterprise-wide common code on a shared network drive (often called a reference directory). This ensures that all development teams are building against the latest approved versions of enterprise-wide common code. If a change in common code corrupts a developer's work, it will be discovered immediately.

Another common problem in complex environments is build scripting inconsistency resulting from development in multiple languages. Build administrators can either force a single scripting language or maintain different build scripts for different teams (Make, for example, works for C and C++, but is not particularly well suited for Java). The best approach is to maintain different scripts with isolated build functionality using a consistent, cross-platform, lightweight scripting language for all non-build functionality (e.g., retrieving code from a CM tool, moving files around, deploying binaries, etc.). Separating build functionality from all non-build functionality limits variances. There is no reason to be using Make or Ant scripts to copy files around or make a logical decision during batch processing.

The third most common problem in complex environments is the lack of an effective audit trail. Log all build script templates and "non-build" script components and make sure audit trails track source code to the executable. For each action that touches source code (check-out, move, compile, etc.), embed a logging message into the script templates. This is facilitated by adding a basic bill of materials report to the in-house build solution, including:

• Name of the final target being built
• Build machine environment information
• Compiler version information
• Version information derived from the CM tool for every dependency included in the build

One of the first breaking points occurs when the amount of time it takes for an application to build begins to limit unit- and integration-testing effectiveness. Only the items that need to be built should be built, in a true incremental approach. Another breaking point is excessive problem-resolution turnaround time, because the development environment scales beyond the capabilities of the in-house scripted manual build system. Developers find themselves spending most of their time tracking down what source code and common libraries went into a built object rather than resolving coding problems.

A sure sign that developers are reaching the limits of manual scripting efficiency is when they find themselves consistently spending as much as an hour a day working on build problems (either their own, or debugging build problems of a centralized CM team). Many companies actually assign a dedicated CM team whose sole responsibility is to execute builds. Developers find themselves waiting for the CM team to build their applications before they can move on to the next development effort. It can reach the point where the centralized CM team simply cannot keep up with the demand, especially when builds are cross-language, cross-platform and incredibly complex.

Conclusion
Through careful build-management planning and implementation, developers are freed up to focus on the development they were hired to perform. By optimizing a developer's workflow, companies are more capable of meeting market pressures with increasingly sophisticated and robust applications. The practices outlined above can be used to develop a build system that handles current build requirements, while paving the way for an orderly transition to automated build strategies that will enable significantly more flexible and manageable development environments.

Migrating to an Automated, Non-Scripting Build-Management Environment

To solve the problems described in the article, teams within medium- to large-sized development environments are now turning to tools based on a true client/server architecture with a central build knowledge base. Introduced over the past five years, this new class of build tool provides a standardized method for creating and managing Build Control files that replace Make and Ant/XML manual scripting. This approach eliminates the portability issues of rule-based programs derived from Make, while resolving the standardization challenges associated with scripted build processes based on Ant/XML.

One example of this approach is Openmake, a build management tool from Catalyst Systems Corporation that weaves together human and machine intelligence to automate and standardize the enterprise build process. It incorporates a browser-based user interface and a Tomcat, WebSphere, or OC4J Application Server to provide access to an Openmake Knowledge Base Server (see Figure 1). Enterprise-based features allow for the connection to multiple remote build servers. Simple Object Access Protocol (SOAP) is used as the communication layer between the browser and the application servers.

Developers interface with Openmake through a Web client, a command-line interface, or indirectly through IDE plug-ins. Build metadata is stored and managed via the central Openmake Knowledge Base Server and reused by multiple developers to generate Ant/XML scripts for Java support, or to generate "Make"-like scripts for traditional build requirements. Build Control files can be generated to build a single object (supporting developer daily compile activities) or a complete application (containing hundreds of interdependent modules). When a complete application Build Control file is generated, it eliminates the problem of recursive Make and ensures the accuracy of incremental builds. Builds can be managed from an empty build directory pulling source code from a predefined search path, or by retrieving source code from a version management tool. Openmake also allows control over environment variable settings such as LIB, INCLUDE, and CLASSPATH so that, regardless of the build machine, the build results are the same.

Developers begin by choosing from among an extensive list of predefined build types, with the option of modifying these templates, as needed. The build type templates are comprised of one or more preconfigured build tasks, enabling developers to quickly and easily craft a completely customized build framework without having to write custom scripts or build classes. Because the build types are stored centrally, all development teams work in a consistent, standardized manner throughout the distributed build process.

Openmake does not replace Ant for completing Java builds, but rather extends the use of Jakarta Ant without the need for manually coding XML scripts. In the place of hard-coded Make and Ant/XML scripts, for instance, Openmake's powerful rules engine takes advantage of its knowledge base of build metadata, such as target name and dependency information, to dynamically generate portable, PERL-based build processes at build time that can be referenced by multiple development teams.