Welcome!

Java Authors: Pat Romanski, Miguel Valdes Faura, Liz McMillan, Carmen Gonzalez, Elizabeth White

Related Topics: Big Data Journal, Java, SOA & WOA, .NET, Cloud Expo

Big Data Journal: Blog Feed Post

A New Breed of Actionable Analytics

Being smart about your monitoring data collection allows you to isolate and resolve problems much faster

Data is dumb but we can't seem to get enough of it these days. An entire industry has evolved to make sense of the massive amounts of data being generated every day. Massive data collection by itself does not guarantee the context required to solve business and IT problems... If we are smart about data, however, it will lead us in the right direction.

Today's businesses are defined by the software they run on, enabling them to innovate and create new services faster. Apple re-invented the music industry through software; Netflix changed the way we consume movies through software; grocery shopping, taxi-hire, parcel-logistics, car rental, travel booking... these are all industries that have been completely disrupted by software.

As more and more organisations depend upon the software they run on, the consumer demand for superior services, faster innovation, and improved performance continues to accelerate. As a result the software required to compete in this unforgiving market only continues to increase in its complexity.

It was only a few short years ago that when I wanted to book my family vacation, we all made the trip down to the local travel agent and sat around a desk while the agent tapped away on her green-screen searching for available flights and hotels. The process of booking our hotels has obviously changed dramatically, and so have the applications that support the process.

For example, below is a screen shot of the Business Transaction flow through a global travel site when a user makes a simple search for a hotel. All the user does is select a destination, some dates and maybe one or two preferences, but the resultant transaction kicks off more than 200 service requests before the results are returned to the user they can make their selection of preferred hotel.

Hotel Search

This rise in application complexity often means we generate increasingly more structured and unstructured data from our applications such as log files, email, alerts, infrastructure stats, network stats etc. Close to 200 billion emails will be sent in 2014, I wonder how many of those emails are false alerts from monitoring tools?

Keep nothing?

I have been helping companies build monitoring strategies for nearly a decade, and the problem I used to regularly face was dealing with organisations that just didn't have enough data. IT departments didn't have the required information to be able to quickly diagnose and remediate problems when they occurred.

Keep everything?

As organisations continue to have a greater dependency on the software they run on and application complexity continues to increase, there is a danger we swing the other way by collecting and storing too much data to be able to make sense of it in a timely manner. My colleague Jim Hirschauer often refers to this as the "home hoarders effect" drawing parallels with people who simply never throw anything out from their homes. The bigger the piles get the harder it is to find what you need, and at some point that pile is going to topple over.

Keep what you need

There are lots of good reasons to keep data, but "just in-case we need it" is not a good reason. It is important to think about why the data is being captured and stored and how it will help solve problems in the future. By keeping the relevant data and throwing away the clutter, you become more efficient and effective in troubleshooting and resolving problems.

But data alone isn't good enough to solve your business and IT problems. Data in and of itself is one-dimensional and dumb. Data doesn't tell you when there are problems, it doesn't tell you when business is going well, it doesn't tell you anything meaningful without some help and a lot more context. Let's explore an example to illustrate my point.

Let's say we have a couple of data points about a person. The data points are...

Heart Rate = 150 bpm

Blood Pressure = 200 over 100

Now tell me, is this person performing well? With these data points we have no idea. We need more data. The table below shows a list of some possible data points we can collect.

Screen Shot 2014-03-24 at 10.42.29 PM.png

Notice the last attribute in the table. The activity provides us with the context we need to focus on the proper data points to figure out if the person is performing well or not. Here are some more relevant data points...

Distance Run = 100 meters

Time = 9.58s

Now can we determine if the person is performing well or not? I'm not much of a track and field aficionado so I have no idea if this is a good performance or not. I need point of comparison to determine if 9.58 seconds in the 100-meter dash is good. So here is our baseline for comparison sake...

100-Meter World Record Time = 9.69s

Well it looks like the person was performing really well. They set a world record in the 100-meter dash. All of the data points individually didn't tell us anything. We required correlation (context) and analytics (comparison to baseline) in order to turn data into information. I like to refer to this concept as creating Smart Data.

Smart Data Defined

Smart Data is actionable, intelligent, information.

Smart Data is created by performing correlation and analytics on data sets. AppDynamics correlates end user business transaction details with completion status (success, error, exception), response times, and all other data points measured at any given time. It automatically analyzes the entire data set to provide information from which to draw conclusions and take the appropriate action. This information is called Smart Data.

Being smart about your monitoring data collection allows you to isolate and resolve problems much faster, and with a much lower cost of ownership and overhead.

I have recently been working with a company to replace their legacy monitoring tool with AppDynamics. The environment they are monitoring is a good size, consisting of about 1,200 servers and their main application processes approximately 300,000 transactions per minute. They invested in a monitoring tool to help manage the performance of their applications which captured and stored all the data they could "just in-case" it was needed. Unfortunately this approach required an additional 92 servers to be provisioned for the monitoring tool itself, which consumed approximately 80TB of data storage per year. The increasing investments this customer needed to make in hardware, storage, people, and maintenance was too much to manage for them and they decided to look for a different approach.

AppDynamics "smart data" approach to analytics means this particular customer now only requires two reporting servers and the storage requirements were reduced to just 1TB per year. Collecting only the data required to make smart decisions gave them both a 98% reduction in hardware costs and  more effective analytics in the process.

Adding business context

Smart data is not just about resolving problems faster though. In October last year we introduced Real-time Business Metrics and described how AppDynamics customers can use Real-time Business Metrics to extract and present business metrics directly from within their applications. These business metrics provide business context enabling customers to turn smart data into actionable information. Our customers, for example, can see the exact revenue impact of performance problems, the end user experience during an app upgrade, or the real-time impact of marketing campaigns. Smart analytics not only clearly show the business benefits of making immediate improvements, they help direct where limited resources should be invested for further business and application improvement going forward.

AppDynamics is focused on delivering actionable intelligence to sell problems for IT operations and development teams as well as business owners. To learn more about Real-time Business Metrics read here.

The post A New Breed of Actionable Analytics written by appeared first on Application Performance Monitoring Blog from AppDynamics.

Read the original blog entry...

More Stories By AppDynamics Blog

In high-production environments where release cycles are measured in hours or minutes — not days or weeks — there's little room for mistakes and no room for confusion. Everyone has to understand what's happening, in real time, and have the means to do whatever is necessary to keep applications up and running optimally. DevOps is a high-stakes world, but done well, it delivers the agility and performance to significantly impact business competitiveness.

@ThingsExpo Stories
The Internet of Things (IoT) promises to evolve the way the world does business; however, understanding how to apply it to your company can be a mystery. Most people struggle with understanding the potential business uses or tend to get caught up in the technology, resulting in solutions that fail to meet even minimum business goals. In his session at @ThingsExpo, Jesse Shiah, CEO / President / Co-Founder of AgilePoint Inc., showed what is needed to leverage the IoT to transform your business. He discussed opportunities and challenges ahead for the IoT from a market and technical point of vie...
IoT is still a vague buzzword for many people. In his session at @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, discussed the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. He also discussed how IoT is perceived by investors and how venture capitalist access this space. Other topics discussed were barriers to success, what is new, what is old, and what the future may hold. Mike Kavis is Vice President & Principal Cloud Architect at Cloud Technology Pa...
Dale Kim is the Director of Industry Solutions at MapR. His background includes a variety of technical and management roles at information technology companies. While his experience includes work with relational databases, much of his career pertains to non-relational data in the areas of search, content management, and NoSQL, and includes senior roles in technical marketing, sales engineering, and support engineering. Dale holds an MBA from Santa Clara University, and a BA in Computer Science from the University of California, Berkeley.
The Internet of Things (IoT) is rapidly in the process of breaking from its heretofore relatively obscure enterprise applications (such as plant floor control and supply chain management) and going mainstream into the consumer space. More and more creative folks are interconnecting everyday products such as household items, mobile devices, appliances and cars, and unleashing new and imaginative scenarios. We are seeing a lot of excitement around applications in home automation, personal fitness, and in-car entertainment and this excitement will bleed into other areas. On the commercial side, m...
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
An entirely new security model is needed for the Internet of Things, or is it? Can we save some old and tested controls for this new and different environment? In his session at @ThingsExpo, New York's at the Javits Center, Davi Ottenheimer, EMC Senior Director of Trust, reviewed hands-on lessons with IoT devices and reveal a new risk balance you might not expect. Davi Ottenheimer, EMC Senior Director of Trust, has more than nineteen years' experience managing global security operations and assessments, including a decade of leading incident response and digital forensics. He is co-author of t...
Performance is the intersection of power, agility, control, and choice. If you value performance, and more specifically consistent performance, you need to look beyond simple virtualized compute. Many factors need to be considered to create a truly performant environment. In his General Session at 15th Cloud Expo, Harold Hannon, Sr. Software Architect at SoftLayer, discussed how to take advantage of a multitude of compute options and platform features to make cloud the cornerstone of your online presence.
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by mining large volumes of unstructured data, and how data tracking delivers uptime when it matters most.
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete at launch. DevOps may be disruptive, but it is essential.
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happens, where data lives and where the interface lies. For instance, it's a mix of architectural styles ...
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
The Internet of Things will put IT to its ultimate test by creating infinite new opportunities to digitize products and services, generate and analyze new data to improve customer satisfaction, and discover new ways to gain a competitive advantage across nearly every industry. In order to help corporate business units to capitalize on the rapidly evolving IoT opportunities, IT must stand up to a new set of challenges. In his session at @ThingsExpo, Jeff Kaplan, Managing Director of THINKstrategies, will examine why IT must finally fulfill its role in support of its SBUs or face a new round of...
The BPM world is going through some evolution or changes where traditional business process management solutions really have nowhere to go in terms of development of the road map. In this demo at 15th Cloud Expo, Kyle Hansen, Director of Professional Services at AgilePoint, shows AgilePoint’s unique approach to dealing with this market circumstance by developing a rapid application composition or development framework.

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's platform-as-a-service. The new platform enables developers to build ap...

Advanced Persistent Threats (APTs) are increasing at an unprecedented rate. The threat landscape of today is drastically different than just a few years ago. Attacks are much more organized and sophisticated. They are harder to detect and even harder to anticipate. In the foreseeable future it's going to get a whole lot harder. Everything you know today will change. Keeping up with this changing landscape is already a daunting task. Your organization needs to use the latest tools, methods and expertise to guard against those threats. But will that be enough? In the foreseeable future attacks w...
As enterprises move to all-IP networks and cloud-based applications, communications service providers (CSPs) – facing increased competition from over-the-top providers delivering content via the Internet and independently of CSPs – must be able to offer seamless cloud-based communication and collaboration solutions that can scale for small, midsize, and large enterprises, as well as public sector organizations, in order to keep and grow market share. The latest version of Oracle Communications Unified Communications Suite gives CSPs the capability to do just that. In addition, its integration ...