Scaling Up in Production Environments
Provided your production environment is not too dissimilar from your staging environment, most production issues typically relate to troubleshooting hard-to-find defects, threshold conditions, latent dependency issues, and resource conflicts. Unfortunately, these are the types of complex system problems that often take hours and sometimes days to track down. These are the types of problems that can suck up a lot of an IT organization's time and money. In fact, most production environments running a modern application stack have become too difficult for help desk and first-line support staff to assist with troubleshooting. So the initial evidence gathering and analysis falls to system administrators (the ones who understand physical complexity) and developers (the ones who understand logical complexity). (see Figure 1)

We've all been through the reported incidents of application availability problems resulting in the multi-tier "blame game" drill. Investigating the chain of evidence from a Web server to the application server to the database across a network can be very painful. There are many sources of IT data and it often takes two or more domain experts to understand the data.

In this example, we see a series of events from three tiers that needs to be located to track the trail of a single Web browser posting data into a database. The transaction in question is a $200 ATM withdrawal from account ID 5005.

First, from the Apache access log, we're able to find the following by looking through the logs for the account ID:

127.0.0.1 - 200 [18/Oct/2005:08:53:31 -0800] "GET /bank/atm?accountId=5005&operation=0
HTTP/1.1" 200 1987 "http://localhost:8080/bank/main" "Mozilla/5.0 (X11; U; Linux i686;
en-US; rv:1.7.8) Gecko/20050524 Fedora/1.0.4-4 Firefox/1.0.4"

127.0.0.1 - 200 [18/Oct/2005:08:53:40 -0800] "POST /bank/atmAck HTTP/1.1" 200 1148
"http://localhost:8080/bank/atm?accountId=5005&operation=0" "Mozilla/5.0 (X11; U; Linux
i686; en-US; rv:1.7.8) Gecko/20050524 Fedora/1.0.4-4 Firefox/1.0.4"

Next, from the application server logs, we are able to find the trail of events processing the application logic. We find the first event by looking at the account ID and follow the events in time to the remainder of the trail:

2005-10-18 08:53:39,793 INFO [STDOUT] Debug: 5005

2005-10-18 08:53:39,793 INFO [STDOUT] Debug: Setting operation to: 0

2005-10-18 08:53:39,793 INFO [STDOUT] Debug: Setting amount to: 220.00

2005-10-18 08:53:39,793 INFO [STDOUT] Debug: Setting account id to: 5005

2005-10-18 08:53:39,783 INFO [STDOUT] Debug: /atmAck

Finally, from the database, we see the connection to the SQL statements with the account ID and EJB name:

/*C10*/SET AUTOCOMMIT FALSE
DELETE FROM NEXT_ID WHERE BEANNAME='tx'
INSERT INTO NEXT_ID VALUES('tx',137)
COMMIT
/*C6*/INSERT INTO TX VALUES('137',NULL,'2005-10-18 08:53:39.931',-220.00,7108.28,'ATM
Withdrawal')
DELETE FROM ACCOUNT WHERE ACCOUNT_ID='5005'
INSERT INTO TX VALUES('137','5005','2005-10-18 08:53:39.931',-220.00,7108.28,'ATM
Withdrawal')
COMMIT

In this simple example we had enough associative data information to follow a series of cascading events from one tier to the next; however, it's usually not this easy. Most multi-tier systems don't have this kind of identifier threading through each tier of technology. More advanced troubleshooting techniques, such as searching for time-based correlations or transitive data connections, are usually required. Obviously, knowing your data and knowing what to look for helps. Having automated tools that can quickly search through multiple data sources is a good idea too.

Look What's Coming Next
Someone once told me that data centers are like roach motels...stuff comes in but it never comes out. As Java and J2EE technologies become even more mission-critical, Web services become more widely deployed, and new layers like virtualization arrive, look out!

At the recent JBoss World conference in Barcelona, I met several developers working on the next generation of large-scale, truly mission-critical applications. Samuel Stammbach from SkyGuide in Switzerland is building the world's first air traffic control system based on J2EE. Ron Eller from HP talked about the real-time reticketing application they've built for Continental Airlines to provide instant changes to tickets and the ability to change a ticket online. Jean-Marie Lapeyre is the director of project Copernic, a 10-year program with a 1B Euro budget to build a simplified online tax system for 55M citizens and 3M businesses in France. Copernic has 4,000 Linux servers and a committed goal to solve every problem within 48 hours. The Copernic Web site had 5B hits for the current tax season during a two-month period and 3.8M returns were filed online last year.

All of these developers are looking to Web services and service-oriented architectures to reduce complexity through the reuse of software components and easier integrations between J2EE containers, services, and legacy systems. According to Gartner Group, we're all buying into the vision. Gartner estimates by 2008, 80% of software development projects will be based on service-oriented architectures and, by 2006, more than 60% of the $500B market for IT services will be based on Web services standards and technologies. Looks like we'll all be adding Web services requests, WS*, and WSTransaction logs to the pile of stuff we already need to regularly sift through.

Another obvious trend to watch is the continued popularity of open source, which is also changing the characteristics of the modern application stack. No longer do the third-party tools and components we use come from a small number of vendors. More than two-thirds of enterprises are using open source technologies and there are more than 100,000 open source projects on SourceForge.net. All this choice is great, but at what cost?

What's the next technology to jump on the stack? It's called virtualization. Some say virtualization will be the solution to the explosion of physical machines and devices while others believe it will only create additional logical complexity, rendering it unmanageable. Time will tell, but for sure all that virtualization state information and activity logs will need to be added to the mountain of IT data we need to grok.

It's been 10 years since the birth of Java, and 4.5 million Java developers would agree that J2EE has become the standard for enterprise applications. Now we're all spending an increasing amount of time and effort trying to keep our applications running as the "application stack" has become more complex. Analyst firms like IDC have sized the total expenditure on managing data centers upwards of $100B. I'd be interested to know how much of that is consumed by the human labor element of troubleshooting J2EE applications.