Calculating Rates... It's Harder Than It Looks
Once the data are made available and properly identified, the problem then becomes presentation. One common requirement is to plot metrics like Msgs Sent and Rcvd in a trend chart.

Obviously, raw Total Msgs data shouldn't be plotted since it will increase continually. The delta from one event to the next must be computed and used to plot the trend.

This problem isn't unique to JMX - it's found in many systems that gather data in real-time. It's simple to count the messages, export that raw data, and let the monitoring client deal with the job of calculating deltas and rates.

However, at the client end it's not that simple, particularly when data are gathered from multiple sources. Table 5 and Table 6 contain data at two instants in time for just two channels, 12 and 13, on Server 1.

As long as the number of channels is the same at each time instant, the problem isn't so difficult. New values for Channel 12 are compared with previous values for Channel 12 to calculate a delta. It's the same for Channel 13. (Table 7)

To do this on the client side requires that one keeps a "cache" of prior data as obtained in the previous time interval. Additionally, each row of data must be cached by one or more "index" columns that uniquely identify the source. In this case, it's a combination of the Server and Channel columns that identify the rows to be compared.

This isn't overly difficult; there are simple algorithms for constructing a "key" from the index columns and storing a data row in a hashtable using that key. For each new row of data, old data are extracted using the key, a delta is computed, and the new data stored in place of the old.

However, it does put a burden on the developer of the monitoring client to maintain a cache and calculate deltas against the incoming streams of data. Had the delta values been computed and exposed in the MBean no such effort would be required.

In practice, it gets worse. Often, the number of sources doesn't remain constant. Channels may be created dynamically and the ID of each new channel is continually changing, starting at 1 and incrementing each time. Old channels may be closed and their IDs never reused.

In this situation, the cache we maintain to do the delta calculation keeps growing. (Table 8)

Here, channels 12 and 13 may be long gone, but the previous values stored in the cache are still there. To avoid a serious memory issue, the client code must provide a mechanism by which the cache can be cleared of items that are no longer needed. To do this requires that an event be generated, indicating that a channel has been closed.

Of course, we may need a "channel closed" event for other reasons, but to require it just so we can safely calculate deltas seems excessively burdensome.

Because this problem is seen so often, advanced visualization products usually provide built-in caching capability and transformations that can be used to perform the required work. However, when trying to apply a low-level charting package to the problem, one quickly finds that the problem is much bigger than originally thought.

The obvious recommendation here is to move the delta calculation back to the MBean code. In most cases, a "count" metric is going to be plotted, so provide the delta calculation upfront. It's usually much easier to do this at the source. There's no need for indexing of the data as the source contains both the old and new data. There's no need to worry about the comings and goings of the objects (e.g., channels) since the delta processing is self-contained in the object.

This is not to minimize the problems on the data-collection side either. There are difficulties that arise when moving the job back to the server. The problem now becomes how to manage the information so that it can be accessed by multiple clients without having to keep track of which client knows what.

The most common way of dealing with this is to provide the delta in terms of a "rate" rather than an absolute delta. This way the time interval for the calculation can be maintained independent of the client. The client can be supplied both the rate information and the original total information. In many cases, it's the rate that users want to see anyway, since the collection interval can vary and is really irrelevant.

Recommendation: Do delta calculations and/or rate computations on the server side rather than on the client. It's much more efficient and easier on the client-side developers.

Give Me a Break - Part 1
Another problem often seen in data collection systems is the inefficient encoding of information that's passed to the client for presentation. A good example of this is the "statistics" column in our sample MBean. (Table 9)

Often this is seen when the default view of the data is a simple HTML page. Having the metrics already available in a string form makes it easy to display in an HTML table without having to do numeric formatting.

Sometimes, developers are inclined to use XML since it's portable, easily parsed in client systems, and not binary:

<?xml version="1.0"?>
    <Statistics>
      <ProcessTime units="ms">124</ProcessTime>
      <WaitTime units="ms">34</WaitTime>
    </Statistics>

One argument for using either form is that the metrics encoded in the string are to be accessed as a unit, so it's clear that they represent a metric at the same instant in time.

However, most monitoring and visualization systems are fully capable of dealing with structured numeric data. JMX provides a simple OpenMBean data type called CompositeData. Several related measurements can easily be put in a Composite structure using JMX API calls. The data are transferred via JMX in an efficient binary form.

With XML or string encoding, the server must pack the data into a string and the client has to unpack it at the other end, knowing the schema for the data. Using CompositeData, a client has information immediately available about the semantics of the data. With strings, it has none.

Recommendation: Use structured Composite and Tabular data types wherever possible - and avoid string encoding that results in extra network overhead and more development work on the client side.