This MBean would be assigned a domain name like "ChannelManager" and a key like "type=ChannelInfo." Each channel would create a unique MBean with two additional key components, "server=XX" and "channel=NN," to indicate the server on which the channel is running and the ID of the channel itself. The full name of the bean used to return information about channel 12 on Server1 would be:

ChannelManager:type=ChannelInfo,server=Server1,channel=12

We might also have other information related to configuration, perhaps not as dynamic. This might include an IP Address and Port, and a count of active connections. (Table 2)

From a developer perspective, the information collected here is obtained from a different source and, as such, it's natural to have a separate MBean. It would have a different type key, "type=NetworkInfo," indicating it's network-related data. The full name for this bean might be:

ChannelManager:type=NetworkInfo,server=Server1,channel=12

So we have two simple MBeans, a ChannelInfo and a NetworkInfo bean. These map closely to the internal data structures of the system so it's quick and easy for developers to implement.

At this point, things are looking good. Lots of information is available for monitoring. Typically, an HTML page is provided as part of the system to view each of the MBeans. The system runs, the data show up in the HTML page and everyone (especially marketing) is excited about the new application monitoring capability available with this new version of the company's product.

Not So Fast... There Are Clouds on the Horizon
The first indication of a problem shows up when customers attempt to monitor the system in a useful way against live data. At first, looking at each channel via the HTML interface is exciting...there's so much data to explore and users see things they've never seen before about the behavior of the system.

But looking at one bean at a time in an HTML page gets old really fast, especially once you have more than just a few channels. Imagine how difficult this is if one of the servers has 100 channels running on it. It's practically impossible to extract useful information from the system this way.

It quickly becomes apparent that to understand the workings of the system fully, one must be able to perform calculations on the incoming data in aggregate. There must be a way to sum messages counts across all channels or take an average across the servers. Metrics can be dumped to a database for later analysis, but this isn't a real-time solution by any means.

One solution is to take advantage of the advanced capability provided by JMX to access multiple MBeans using the wildcard "*" syntax. In other words, request the names of all the beans matching a certain pattern, such as:

ChannelManager:type=ChannelInfo,server=Server1,*

This request will return the names of all ChannelInfo beans on Server1 for all channels. The data from each can be compiled into a single table, one row per bean. This seems like it will work, but there's a big problem lurking.

Identify Yourself... Or Else
In our sample MBean, it seemed natural for the developer to include the Channel ID in the data - on a single server, the internal data structures contain the ID of the channel being implemented. However, since all channels on a server live in the context of that server, it didn't seem reasonable to include the name of the server as part of the data.

In fact, from the developer's perspective, it made sense to minimize the data to be transferred and leave out the server name - since the bean name itself contains the server info, it shouldn't be necessary to include it in the data.

However, look what happens when the data from each channel MBean on each server are gathered into a single table (see Table 3).

Note that the table contains a row of data from each MBean on all servers. In this sample, the Channel ID field contains the number 13 twice. This is because channel 13 exists on more than one server.

The fact that the data elements don't identify the name of the server from which it came is a big problem. The name of the MBean contains the name of the server, but use of the wildcard syntax to reference all MBeans doesn't automatically provide information about the source. Without information about the server, the table produced here is useless. It seems like a simple oversight, but this is probably the most common problem encountered when visualization and analysis is attempted on data exposed via JMX (and many systems that pre-date JMX).

After seeing this situation come up time and again, some products like Enterprise RTView have evolved to provide automatic ways to supply this essential information to the presentation and analysis layer. This is done by parsing keys contained in the MBean name and creating new columns if they don't already exist. With the "Server" column added, Table 3 becomes Table 4.

In this example, the "Channel" column already exists so it's not needed. The "Server" column isn't available but can be added from the name of the MBean sourcing the data.

In general, the problem can be avoided by properly identifying the source of the data in the original data table. In many of these systems, there are dozens of columns of data. Saving one or two columns isn't very effective when you consider what has to be done on the client side. There may be a question as to whether it's good modeling practice to include extra data in the MBean (it's duplicated in the name). However, if the goal is to minimize the development effort on the client side, then the extra information is helpful. In an Aggregator MBean, it would be essential.

Recommendation: When using * to collect data from multiple beans of the same type or multiple connections, include columns as attributes that identify the source - otherwise you can't tell which bean it came from.