Most current source code management tools deal with navigating source hierarchies and finding objects and methods. These are great problems to solve, but not ones that we're primarily interested in (JavaDeps comes relatively close in that it helps to discover some compilation dependencies that go unnoticed by some compilers).

Similarly, many revision control systems (RCS) provide check-in, checkout, branch, and merge capabilities, but none address source code tree structure and how to manage the requisite dependencies involved.

Target Audience
The target audience is developers, testers, and operational support staff who are interested in taking the necessary steps to actively manage their Java-based projects in terms of building source code for test and operational deployment; developing multiple versions of a product or service in parallel; and replicating operational, test, and development environments to reproduce unexpected behavior and fix bugs.

Large numbers of Java source files are in the range of 500..10K with large numbers of dependent third-party JARs in the range of 50..1K. All told, we're talking about a set of development projects that have 0(50L) total document and code artifacts...not very big, but large enough that it's worth examining how the code base evolves and how to keep it from turning into a liability instead of the asset it's intended to be.

Due to its complex nature, this topic is too large to be covered in a single article. I'll start by covering the basics of source code management and builds, and finish by touching on the topics of managing deployments and documentation. Future areas for discussion include managing properties files and build tools and building WAR files.

The Evolution of Java Source Code Hierarchies (aka Back to Basics)
Every Java shop I've ever worked in has followed an eerily similar evolutionary path as far as its Java source code is concerned:
1.   Starts the root branch off by creating Java package com.mycompany
2.   Begins to populate the source tree with a layer of utility and/or base classes, many of them the usual suspects like com.mycompany.db, com.mycompany.utils, com.mycompany. regexp, and com.mycompany.xml
3.   Continues to populate this source tree with a set of servlets, beans, data access, and JSPs that depend on the set of common classes (the aforementioned usual suspects)

This approach is extremely intuitive and works for a while ­ for about as long as the codebase remains simple enough that dependencies between distinct packages are well understood.

The first dependencies introduced are usually servlets, beans, and JSPs importing common/shared utility classes. These dependencies are distinct, simple, and well understood. However, soon thereafter, more complex references are introduced as developers try to reuse as much code as possible while minimizing the time they spend repackaging code. A servlet from one package begins to look like a utility to another package and is subsequently imported. This type of import can create a circular reference (see Figure 1) between source code files and sets the stage to make even simple SCM prohibitively complex.

Introducing circular references in Java is surprisingly easy and extremely common, though, interestingly enough, I've never actually heard of a developer admitting to such a practice. Understanding these relationships, plus your source code's dependencies on third-party JAR files, is key to having a modular, branchable, buildable, testable, and deployable codebase.

Partitioning Source Code: The Introduction of Components
The first step in decoupling direct source code dependencies is to partition your source code into components.

A component is a set of Java packages that provides a specific set of functionality and has its own development cycle. It doesn't matter whether it's one package or 20, one source file or 200 source files (though using more than a few hundred source files in one component will bring you right back where you started, in terms of problematic source code management). Having their own development cycle means that, relative to the other components, the source files need to be built/tested/deployed n times a day while other source code needs to go through this cycle m times a day.

Partitioning source code into components will become fairly intuitive after a few examples:

You could end up with as many as several hundred components, each with anywhere between 10 and perhaps 350 source files. Although partitioning your source code looks complicated, it's actually easy (the difficult part is getting your builds started).

All this source code needs to be checked into a revision control system (RCS). Any/all RCS syntax in this article will be in reference to Perforce (www.perforce.com), as it is has many features that make SCM very simple.

In Perforce parlance, the component source code is checked into location:

//depot/components/<component_name>/src

For example:

//depot/components/FileUtils/src/com/mycompany/...
//depot/components/DataParser/src
//depot/components/UserData/src

Builds, branches, and documentation are also partitioned under each component for RCS:

//depot/components/<component_name>/src
//depot/components/<component_name>/branch
//depot/components/<component_name>/builds
//depot/components/<component_name>/docs

Third-Party JAR and ZIP Files
The other source for build dependencies are between your source code and JARs provided by a third party.

This necessitates actively managing these files to keep on top of their multiple versions and frequent name collisions. It's very easy to impede the progress of debugging and building through the mismanagement of third-party JARs and ZIPs (e.g., opening up JARs manually to try to find a version number to find out what you built against, or what version you have in production), and yet remarkably simple to organize them intuitively and efficiently.

Because successive versions of third-party JARs sometimes result in name collisions, it's necessary to use the version numbers to maintain them under RCS. In Perforce, the JARs might look like the following (using the JDK and JSDK as examples):

//depot/jars/jdk/1.2.2/rt.jar
//depot/jars/jdk/1.3.0/rt.jar
//depot/jars/jdk/1.3.1/rt.jar
//depot/jars/jsdk/2.0/jsdk.jar
//depot/jars/jsdk/2.1/server.jar
//depot/jars/jsdk/2.1/servlet.jar
//depot/jars/jsdk/2.2/servlet.jar

This versioning scheme allows components that might depend on the 2.1 version of servlet.jar to reside next to components that might depend on the 2.2 version. Both components can be built and deployed in parallel and their dependencies tracked accordingly.

This approach also has the added bonus of allowing for any client that has access to your RCS server to be able to run builds, as every server has access to the requisite JARs via RCS.

Building Components
Now that your source code is partitioned and third-party JARs are under RCS, it's time to start building. Build requirements are very simple:

This first component built must be entirely self-contained ­ it can be built using only its own source code and (optionally) third-party JAR files. Components built this way are seed builds and start your build process. Build each of these components one at a time by compiling their Java source, JARing up the resultant class files, and checking these JARs into your RCS (build scripts should do all this for you).

If you can't isolate a component so that it's entirely self-contained, either repackage your source code (not often done due to time constraints) or generate an invalid build so you can begin to generate seed builds. (An invalid build is when a component is built against its own source code plus the source code of another component. Sometimes it's impossible to isolate even one component so it's self-contained, so you'll need to build it against multiple components' source code to get started. After this initial build, you'll be able to build it against its own source code and JARs created from this first build.)

A high-level overview of a build involves the following steps:
1.   Sync up source code and third-party JARs from your RCS to your local machine.
2.   Make sure your target build number hasn't been built already.
3. Set up your CLASSPATH, which contains three sets of entries:

Once you have all of your seed builds, begin to build those components that have only one level of dependence on other source code within your repository, i.e., they can be built using only their own source code, the seed JARs, and, optionally, third-party JARs. Build each of these components individually, JAR up their resultant class files, and check these JARs into RCS.

Once all your one-level dependence builds are complete, it's open season to build the rest of your components, usually done in the order of increasing number of dependencies. The goal is to make sure no component is compiled against any source code except its own.

What you're effectively doing here is isolating like branches of Java code in sets of Java packages against changes in other branches of Java code (also grouped in Java packages). This is probably the most important aspect of the build strategy. This allows stakeholders of your SCM system to isolate, and therefore understand, the dependencies between source code and JARs, and trace any build to the source code that was used to generate that build as well as replicate an environment by easily reproducing the JARs used to construct the environment.

Equally important is that the source code for a component is associated with its build JAR via a label, so it's easy to trace any class file you have in production back to its source files, and then from there to trace other dependent components' class files back to their corresponding source code.

This method of organizing your builds also frees up any components that share a dependency on a common component (e.g., utils). The common component is now on its own development cycle, so it can iterate through many build cycles while allowing dependent components to migrate to newer builds when it makes the most sense. Said another way, it allows for independent/parallel development of components that have a dependency on a single, shared component.

Build Example
At a high level, consider the following scenario for building your first three components:
1.   Component 1 ­ Utils: No dependencies. Build it against its own Java source code to generate a JAR file that contains the resultant .class files.
2.   Component 2 ­ DataParser: Depends on a previous build of the Utils component, as well as a third-party JAR called xerces.jar (v1.4.1). Build it against its own source code, a previous Utils component build, and the xerces.jar file from xerces v1.4.1.
3.   Component 3 ­ DataCaptureUI: Depends on a previous build of DataParser, a previous build of Utils, and a third-party JAR called servlet.jar (v2.0). Build it against its own source code, a previous DataParser component build, a previous Utils component build, and the servlet.jar file from jsdk 2.0.

Note that because Component 3 depends on DataParser, and DataParser depends on xerces.jar, you'll need to add xerces.jar as a dependent JAR for the DataCaptureUI build.

The above set of builds and dependencies is shown in Figure 2.

Conclusion
Managing source code dependencies is only the tip of the iceberg for comprehensive SCM. Other facets of SCM that fit into the component model include:

Partitioning Java source code into components and formalizing dependencies will provide several key benefits for your Java-based projects, some of which are implicit thus far: