Developing custom applications using Microsoft's .NET Framework is a growing trend. According to Forrester Research, 56% of enterprises are choosing .NET versus 44% opting for J2EE, while IDC reports that 35.7% of large corporations use .NET for their most important applications compared to 25.3% for Java. While .NET lets your development team develop applications quickly and efficiently, it's still a challenge to make sure that:

• Applications get into production smoothly (and stay in production).
• Performance is optimized under real-world conditions involving the unruly Internet, shared servers, impatient users, broken links, etc.
• Applications are manageable by the operations team (so you don't have to be dragged back into fixing production problems).

The last point is the most painful for developers. According to Gartner, up to 40% of a developer's time can be spent dealing with issues affecting applications in production. Even with all the pre-production testing and production monitoring tools, Forrester reports that 71% of application problems are still found by real users and not by existing tools or testing procedures.

In other words, despite all the effort and expense to assure application quality, actual users are still being used as QA and optimization tools. Developers need the ability to monitor end-user transaction performance efficiently, and tag and trace slow transactions through the various application tiers back to the specific method calls or SQL queries causing the delays. A call tree like the one shown below is indispensable in efficiently identifying bugs or areas for tuning.

By spanning multiple components inside and outside the data center, the complexity of .NET Web applications has diminished the efficacy of existing lab-oriented simulation or load-testing tools in predicting and monitoring application performance. For example, your development environment probably has a totally different configuration and load from your production infrastructure. Furthermore, the increasing trend toward virtualization and shared infrastructure means that an application that runs well in isolation in the lab could be sluggish "in the wild." As a result, actionable information like the call tree shown above is needed to efficiently complete a .NET development project.

The solution to this conundrum is to have a suite of tools and procedures that can span both development and operations so that developers can:

• Thoroughly test and optimize the applications as they are being load tested and under real-world conditions during beta release.
• Hand off to the operations team the same monitoring and diagnostic capability needed to minimize the time and effort consumed by production problems.

The Moment of Truth: Optimizing as Apps Go from Deployment & Production
A tried and true procedure that's worked well starts when the code has been completed and is installed onto staging servers. It's critical that the staging servers mimic the production servers as much as possible since applications sharing the same server can and will impact each other's performance.

At this point, a load generation tool like Microsoft's Team Server can be used to generate load so as to test the complete application under stress and assure that it does meet the service level objectives of the application.

While a synthetic load from a load-generation tool is good, real users should also test the application during a beta cycle. Every real user interacts with the application differently under different network and use conditions, so boundary conditions not emulated by load-generation scripts need to be uncovered to avoid surprises when the application goes into full-scale production.

The next step in the process is to measure the transaction performance from browser to database. If the test load is from a load-generation tool then the browser isn't really involved and most load-testing tools can estimate transaction round-trip time. If measuring true real-user experience is desired (as it should be), then the real user's browser will have to be instrumented and various parameters like network latency, page load time, broken links, errors, aborts, etc. should be monitored to give a complete picture of the service level that the application and infrastructure are capable of providing. Once the transaction response time of the application is measured, it will be tremendously helpful for debugging or tuning so that any transaction that doesn't meet the pre-defined performance objective can be tagged and traced through the entire application. This is analogous to injecting dye into the bloodstream to highlight the diseased area on an X-ray.

Tagging and tracing problematic transactions from browser to backend will help pinpoint method or SQL calls that cause performance problems. This information can also help identify bugs or performance bottlenecks in the application or infrastructure.

Once the application is fully debugged and tuned, you should deploy the same suite of tools in production. This will let the development and operations teams maintain a common reference for pre-deployment and post-deployment (production) performance comparison and diagnostics. The need to monitor the real-user experience and tag and trace problematic transactions applies in production as well. It's not uncommon for applications to work great in the lab, and then develop mind-boggling performance problems because of the differences between the development and production servers.

Mundane items like configuration, service packs, load, and interaction between applications sharing the same servers are all potential culprits. If a single tool provides both development and production operation personnel with the same shared information, they can collaborate to solve production problems quickly instead of engaging in a futile blame game.

For the same suite of tools to function in both a pre-deployment as well as a production environment, it must satisfy four criteria:

• The monitoring and diagnostics information provided by the suite of tools has to be meaningful and actionable to both the development and operations teams. Furthermore, the tools suite has to facilitate information sharing and effective collaboration between the teams.
• The suite of tools has to be extremely light on bandwidth or capacity consumption since it has to be used in both development and production environments. An added requirement by the operations staff is that the tools suite has to be scalable to handle the varied amount of production volume.
• It has to be simple to deploy and easy to use since the teams using the suite might have different skills set or level of programming sophistication.
• The tools should also be able to monitor and report on common business metrics so the development team can prove that the app meets performance objectives, and for the operations team to show that these objectives are continuing to be met in the real world. (see Figure 1)