Decades ago, when we were all computing on mainframes, the application stack was pretty simple. Programs were all running in core memory on the same machine as the operating system and the data store. There was typically one transactional log for activity and one for errors - not too many places to look for evidence of what had gone wrong and why.

Then the application stack started to grow up. Client software communicated over a network to a server running a more featured operating system and an RDBMS (relational database management system). The biggest challenge for developers was debugging those fat clients - typically by looking at something like Windows event viewer logs. System administrators were generally busy responding to alerts from SNMP traps and rummaging through network device logs or sniffing TCP traffic looking for the source of network failures.

Fast forward to the year 2005. Now our compute environments are distributed architectures based on a multitude of open source and proprietary components.

Each application requires many tiers of technology. Physical data center and logical application stack complexity is exploding. It's not uncommon for a mission-critical application in a single deployment to require a hundred or more physical machines and devices.

What Does This Mean for Java Developers? Mountains of Data...
The significance of all of these new dependencies for the typical enterprise application is that managing the modern stack requires the ability to understand large amounts of evidential data produced by all these different tiers, components, and technologies. This is especially true for J2EE applications.

A recent survey by JBoss of their customers revealed that the typical J2EE application gets deployed in four environments on its way to going live: development, QA, staging, and production. Each migration becomes increasingly difficult, time-consuming, and expensive. Each environment introduces new troubleshooting tasks and involves more people and resources for resolution. Code defects, configuration errors, resource conflicts, and multi-tier dependencies are key sources of the pain.

Developing, deploying, and managing J2EE applications today means being able to understand how Web servers, application servers, databases, storage systems, security, and networking all work together. At the Web tier we're greping Apache and IIS logs. Our J2EE application servers are generating JMS, log4j, custom Java logging, and JMX. If we want to figure out persistence dependencies, the database tier requires an understanding of JDBC (Java Database Connectivity) exceptions, redo or audit tables, slow query information, and replication status.

When we look at an application server, a Web server, or a database in a medium-sized enterprise data center, it could be generating (depending on the verbosity level) anywhere from 10 to 100 megabytes of data per day in its logs. If you add up all the subcomponents of the typical enterprise data center, log data can account for upwards of a terabyte of data in a single day.

Unfortunately, there's a tremendous amount of disparity between different types of log file data. An Apache log, for example, has little, if anything, in common with a MySQL slow query log or a JBoss log4j server log. However, entries from different sources of evidence do typically have a timestamp and an IP address from the host generating the evidence. Occasionally there will be one or two common values like a session ID, thread, or process number, or, in the case of a database, a SELECT statement. These are the kinds of clues we need to look for to figure out what is actually happening at runtime within the application stack.

Let's look at a few of the environments that a J2EE application gets deployed in on its way to going live. For each, I'll show a common breakdown that Java developers are experiencing today.

Development and QA Environments
Deployment of a J2EE development environment for building and debugging Web-based applications requires at least three tiers of the application stack - a Web server, an application server, and a database. Initial installation and configuration, debugging container interaction problems, or optimizing code for performance are often troublesome tasks that require access to multiple sources of evidence, including Web server access and error logs, application server logs, the application logs themselves, EJB and JMS activity, JVM state data through JMX, JDBC exceptions, and database Redo or activity logs.

Consider the following example of debugging a Java container-managed persistence problem. This is a very common situation where, after deploying an application in a development environment, we decided to upgrade the database to one that resembles more of what we'll use in production when our application goes live. Many development environments by default come bundled with a simple database. In our case, we're using Hypersonic, which comes bundled with JBoss, and we want to switch it out for MySQL. We've redeployed our application after switching from Hypersonic to MySQL, and the MySQL JDBC library is getting loaded, but we get an exception when trying to run the application.

[root@localhost 2005-10-12 06:55:20]# ant -f jboss-build.xml run-cmp
Buildfile: jboss-build.xml
run-cmp:
   [java] Caught an exception:
   [java] java.rmi.Server Exception: RemoteException occurred in server thread;
   nested exception is:...

The rest of the exception was not helpful so we decided to look through the JBoss log4j server for entries with a severity level of "error." Narrowing the search to the same time-frame as the exception helps minimize the number of errors on which we need to focus. In our case, we found the following error indicating that our EJB could not be instantiated, but it doesn't tell us why.

2005-10-12 06:55:29,111 ERROR [org.jboss.ejb.plugins.LogInterceptor]
EJBException in method: public abstract void
roster.Roster.createLeague(util.LeagueDetails) throws java.rmi.RemoteException:
javax.ejb.EJBException: Could not instantiate bean; CaughtByException is:
*Could not instantiate bean
*at roster.RosterBean.createLeague(RosterBean.java:439)...

Again, the rest of the log entry was not really helpful in this situation, so we decided to search the log for "debug"-level entries for our RosterBean around the same time-frame as the EJB error. We discovered that a state file was missing. Aha! The application server was probably holding on to the old application deployment state file, so we deleted the state files and restarted the application server.

2005-10-12 06:55:29, 159 DEBUG [org.jboss.ejb.plugins.AbstractInstanceCache] Activation
failure javax.ejb.EJBException: Could not activate; failed to restore state;
CausedByException is:
*/opt/live_jboss/jboss-4.0.2/server/alltmp/sessions/
RosterBean-eepnx16n-1m/eepo2fs6-1p.ser
(No such file or directory)
*at org.jboss.rjb.plugins.StatefulSessionFilePersistenceManager.activateSessions
(StatefulSessionFilePersistenceManager)...

Moving to Staging Environments
Once a J2EE application has been developed and debugged, it typically gets moved into a staging environment that more accurately represents a model of the destination production systems. Staging an application involves not only developers but also system administrators who are responsible for setting up and managing the staging environment. Because the environment can be significantly more involved than development, production migration issues often include IP address and port binding issues, incorrect database and service access permissions, and wrong Java class paths and/or versions. Correcting these types of problems now requires a more detailed understanding of the database tier and system configurations. A knowledge of networking and security also becomes very important.