Welcome!

Java Authors: Sematext Blog, PagerDuty Blog, Pat Romanski, Jayaram Krishnaswamy, Liz McMillan

Related Topics: Java

Java: Article

Detecting J2EE Problems Before They Happen

A runtime abstract application model derived automatically from app server using stored knowledge of Java EE construction

This article introduces a new form of analysis for Java EE applications: a runtime abstract application model derived automatically from an application server using stored knowledge of Java EE construction. The model is used dynamically to do extensive automatic checks for a range of construction errors that could produce poor performance or unreliability. The model also lets server behavior be dynamically visualized in real-time or retrospectively.

There has been a lot of attention given lately to the topic of of Model Driven Architecture (MDA), which aims to create working systems by generating source code from successively transformed high-level component models. While doubts have been cast on the real-world robustness of this idea - and previous code-generation solutions haven't been a big success - there's no doubt that the possibility of working with software at a more abstract level holds a strong appeal for engineers.

Although the inauspicious history of CASE tools suggests that making a project dependent on model-driven code generation could be limiting, the central tenet of MDA - the ability to view and analyze our application at an abstract level - is a powerful and attractive goal. Even if our application grew beyond an initial set of predefined patterns and code templates we'd still like to be able to validate and understand it based on a design-level description of its operation.

Derived Model Analysis (DMA)
If we don't have a predefined model, how are we going to get one? Well, if you try to describe your application to someone else you'll almost certainly use architecture-level abstractions: the services it uses; the main business and data components and how these relate. So it would be good if similar high-level abstractions could be derived and presented automatically by analyzing and monitoring the execution of your application. Model elements would include application components, the application server services they use, and the data access, transaction management, and calling relationships between them.

Once application model elements were identified they would be updated dynamically during execution. Monitoring the changing patterns of inter-relationships in the model would automatically detect construction-quality problems by detecting unlikely relationships, unnecessary and duplicated relationships, and undesirable model entity states. Instead of trying to spot problems in the clutter of source code we could see key abstractions directly in the model.

eoLogic terms this form of indirect application monitoring Derived Model Analysis (DMA): tools analyze Java EE applications both statically and during server execution to derive an abstract model that includes both application components and Java EE services. Subsequent changes to the model form a dynamic event sequence that can be used to (a) track and validate application execution and (b) visualize the model. Lower-level application execution details can be recorded in the context of the sequence of model changes.

Note that DMA is not a profiling technique - it doesn't aim to identify current code hotspots; instead, it analyses how services have been constructed and are being used. The idea is to identify places where hotspots or unreliability may occur under load. This deeper form of analysis can be used to find problems before they manifest themselves and without the application being loaded during testing. These problems include incorrect or inefficient transaction grouping, inefficient database access, unreliable sequences of inter-component communication, and failure to control service lifecycles correctly. There's no need to drive the application to a point at which it exhibits slowdown, and the results need little interpretation.

Deriving a DMA Model
To generate and validate an abstract model of an application a tool must be able to monitor events in the server and interpret them in light of the relevant stored knowledge.

This includes definitions of the main abstract entities we're interested in (transaction manager, transaction resources, transactions, EJB containers, JMS destinations, etc.), the possible relationships between these entities, and invalid and valid patterns of relationships and states. DMA forms them into an abstract Entity-Relationship-Attribute (ERA) model as the system executes, with model changes triggering annotated definitions of problem states.

Relationship to JMX
The model sounds a lot like Java Management Extensions (JMX) - which essentially define a form of abstract model for purposes of managing and monitoring Java applications, and it suggests that possibly DMA could be layered on top of the information available from JMX MBeans. In detail, what characteristics does a DMA model require?

  • It must be an accurate and complete abstract model of an application, linking static (source) and runtime application components.
  • It must be able to be updated in real-time as the server executes generating meaningful sequential event flows.
  • It must support a wide range of relationship types including application-level call relationships.
  • It must be able to be intimately combined with knowledge about valid and potentially invalid model forms.
  • It must be possible to relate model-level information easily back to application source.
  • It must be easily filtered to focus on different aspects of server operation.
  • And it must be easily and intuitively understood.
JMX goes some way towards what is needed: It provides an abstract model of an application for both its static and dynamic aspects; it allows easy selection of MBeans; many MBeans relate directly to easily understood aspects of server operation; there's a notification system for attribute changes and there's even an MBean relation service.

However, for our purposes it also has some serious limitations. Many of the relationships we have to monitor are based on calling sequences and application component relationships. Designed primarily for system management and threshold monitoring, JMX doesn't provide the source-level monitoring and mapping that the detection and (especially) the explanation of application construction errors requires. Also, the level of coverage is generally insufficiently detailed to provide a coherent execution model for the purposes of visualization. And if we want to use the product to investigate problems requiring the ability to freeze the server at the point of problem detection and extract stack and related data information then JMX isn't precise enough.

So the approach that we adopted is to create a more detailed runtime ERA model specialized for the following purposes:

  • Representing sequences of server operation precisely and clearly
  • Detecting construction errors based on component interrelations, including call sequences and transaction membership
  • Explaining construction errors by relating model entities and relationships to precise source references
  • Providing an intuitive visual model of sequential server operation
  • Supporting model tracing and playback
  • Supporting integrated debugging
This specialized model then provides the structure for attaching knowledge about model entity roles and valid and invalid patterns of model relationships and attributes, together with details on problem descriptions and suggested fixes.

The need for detailed tracking of calls and object states means that the DMA engine moves from the realm of JMX and more towards an application of Aspect-Oriented Programming (AOP), combining the planned abstraction of JMX with the detailed and flexible monitoring and intervention of AOP. Having said this, it would be wasteful not to exploit the JMX information provided by a server. Some JMX MBeans serve as important internal DMA monitoring and access points, but are augmented with additional monitoring and updating points in the server.

DMA Error Detection
As shown in Figure 1 DMA abstracts from the underlying framework and application objects to a conceptual ERA model. Queries against this model then provide the means for problem recognition.

Usually the abstraction stage is primarily one of selection as key objects are monitored, but it can also require composition of elements from more than one underlying object.

DMA Use Case
To look at how the abstraction mechanisms of DMA allow construction problems to be detected and explained let's look at it operating on a sample application. We'll aim to show how we can identify a pattern of application and framework components that indicates a problem. We'll then show how the problem can be visualized and explained back to the source level by exploring the model at the point of detection.

Example Application
Figure 2 shows a simple Web-based order-processing example that accepts orders and processes them in the following way:

  • An order invoice is created and queued to an existing invoice service using JMS to create and process the invoice.
  • The order details are queued to an order processing system using JMS to process and deliver the order separately.
However there's a problem: The invoices don't arrive at the invoice processing application although the order entries are processed correctly.

Monitoring the Application
To monitor the sample application we'll run the WebLogic server from our DMA analyzer called eoSense, which comprises a server agent and a client. The agent constructs and checks the abstract model as WebLogic executes. When a problem is detected, the agent signals an alert to the client.

Transaction-Related Alerts
Running the example application results in the initial alerts shown in Figure 3 being detected (after several less serious alerts).

Looking at the alerts in more detail, there was a:

  • JMS Message sent inside a JTA Transaction using a non-XA Connection - A JMS Connection created from a non-XA Factory was used to create a JMS session and sender. The sender was then used with the context of a JTA transaction. This may indicate that an XA JMS Connection should have been used instead.
  • Mixed Transactions - The JMS sender has been used from a JMS Session marked as transacted, but there's already a JTA transaction active on the current thread.
DMA Visualization
When the Mixed Transaction alert is recorded a diagram of the ERA model allows the context of the problem to be understood. In eoSense this is called the Server View and an image of the server view is shown in Figure 4.

We can see that there are two active transactions, one linked to the Order Processing Servlet and the other linked to a JMS Session. We can also see that the Order Processing Servlet has communicated with two JMS Senders. Figure 5 shows diagrammatically the named key entities and relationships from Figure 4.

  • There are two in-flight transactions held by the Transaction Manager
  • There is an "initiated By" relationship between the OrderProcessingExample Servlet and Transaction 1
  • There is an "Initiated By" relationship between the JMS Session 1 and Transaction 2
  • The OrderProcessingExample Servlet has sent two Messages: There is a "Has_Called" relationship to JMS Sender 1, which is attached to the Order JMS Destination, and a "Has_Called" Relationship to JMS Sender 2, which is attached to the Invoice JMS Destination.

More Stories By Alan West

Alan West is CTO of eoLogic (http://www.eologic.com), responsible for all product development. He was previously a founder of Object Software Technology Ltd, and has over 20 years of experience in software tool design and architecting large software systems.

More Stories By Gordon Cruickshank

Gordon Cruickshank is co-founder of eoLogic (http://www.eologic.com), a software tools company created to develop innovative testing and debugging solutions. He was previously development manager at Wind River Systems and Objective Software Technology, building C++ debugging and object visualization tools.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
n d 08/22/06 10:36:16 AM EDT

This article introduces a new form of analysis for Java EE applications: a runtime abstract application model derived automatically from an application server using stored knowledge of Java EE construction. The model is used dynamically to do extensive automatic checks for a range of construction errors that could produce poor performance or unreliability. The model also lets server behavior be dynamically visualized in real-time or retrospectively.

@ThingsExpo Stories

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's platform-as-a-service. The new platform enables developers to build ap...

Building low-cost wearable devices can enhance the quality of our lives. In his session at Internet of @ThingsExpo, Sai Yamanoor, Embedded Software Engineer at Altschool, provided an example of putting together a small keychain within a $50 budget that educates the user about the air quality in their surroundings. He also provided examples such as building a wearable device that provides transit or recreational information. He then reviewed the resources available to build wearable devices at home including open source hardware, the raw materials required and the options available to power s...
The Internet of Things promises to transform businesses (and lives), but navigating the business and technical path to success can be difficult to understand. In his session at @ThingsExpo, Sean Lorenz, Technical Product Manager for Xively at LogMeIn, demonstrated how to approach creating broadly successful connected customer solutions using real world business transformation studies including New England BioLabs and more.
Since 2008 and for the first time in history, more than half of humans live in urban areas, urging cities to become “smart.” Today, cities can leverage the wide availability of smartphones combined with new technologies such as Beacons or NFC to connect their urban furniture and environment to create citizen-first services that improve transportation, way-finding and information delivery. In her session at @ThingsExpo, Laetitia Gazel-Anthoine, CEO of Connecthings, will focus on successful use cases.
Enthusiasm for the Internet of Things has reached an all-time high. In 2013 alone, venture capitalists spent more than $1 billion dollars investing in the IoT space. With "smart" appliances and devices, IoT covers wearable smart devices, cloud services to hardware companies. Nest, a Google company, detects temperatures inside homes and automatically adjusts it by tracking its user's habit. These technologies are quickly developing and with it come challenges such as bridging infrastructure gaps, abiding by privacy concerns and making the concept a reality. These challenges can't be addressed w...
The Domain Name Service (DNS) is one of the most important components in networking infrastructure, enabling users and services to access applications by translating URLs (names) into IP addresses (numbers). Because every icon and URL and all embedded content on a website requires a DNS lookup loading complex sites necessitates hundreds of DNS queries. In addition, as more internet-enabled ‘Things' get connected, people will rely on DNS to name and find their fridges, toasters and toilets. According to a recent IDG Research Services Survey this rate of traffic will only grow. What's driving t...
The Internet of Things is a misnomer. That implies that everything is on the Internet, and that simply should not be - especially for things that are blurring the line between medical devices that stimulate like a pacemaker and quantified self-sensors like a pedometer or pulse tracker. The mesh of things that we manage must be segmented into zones of trust for sensing data, transmitting data, receiving command and control administrative changes, and peer-to-peer mesh messaging. In his session at @ThingsExpo, Ryan Bagnulo, Solution Architect / Software Engineer at SOA Software, focused on desi...
"For over 25 years we have been working with a lot of enterprise customers and we have seen how companies create applications. And now that we have moved to cloud computing, mobile, social and the Internet of Things, we see that the market needs a new way of creating applications," stated Jesse Shiah, CEO, President and Co-Founder of AgilePoint Inc., in this SYS-CON.tv interview at 15th Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, data security and privacy.
The Internet of Things is tied together with a thin strand that is known as time. Coincidentally, at the core of nearly all data analytics is a timestamp. When working with time series data there are a few core principles that everyone should consider, especially across datasets where time is the common boundary. In his session at Internet of @ThingsExpo, Jim Scott, Director of Enterprise Strategy & Architecture at MapR Technologies, discussed single-value, geo-spatial, and log time series data. By focusing on enterprise applications and the data center, he will use OpenTSDB as an example t...
The industrial software market has treated data with the mentality of “collect everything now, worry about how to use it later.” We now find ourselves buried in data, with the pervasive connectivity of the (Industrial) Internet of Things only piling on more numbers. There’s too much data and not enough information. In his session at @ThingsExpo, Bob Gates, Global Marketing Director, GE’s Intelligent Platforms business, to discuss how realizing the power of IoT, software developers are now focused on understanding how industrial data can create intelligence for industrial operations. Imagine ...
Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water, are pursuing SmartGrid initiatives that represent one of the more mature examples of SAE. We have s...
There is no doubt that Big Data is here and getting bigger every day. Building a Big Data infrastructure today is no easy task. There are an enormous number of choices for database engines and technologies. To make things even more challenging, requirements are getting more sophisticated, and the standard paradigm of supporting historical analytics queries is often just one facet of what is needed. As Big Data growth continues, organizations are demanding real-time access to data, allowing immediate and actionable interpretation of events as they happen. Another aspect concerns how to deliver ...
Scott Jenson leads a project called The Physical Web within the Chrome team at Google. Project members are working to take the scalability and openness of the web and use it to talk to the exponentially exploding range of smart devices. Nearly every company today working on the IoT comes up with the same basic solution: use my server and you'll be fine. But if we really believe there will be trillions of these devices, that just can't scale. We need a system that is open a scalable and by using the URL as a basic building block, we open this up and get the same resilience that the web enjoys.
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world.
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.