Welcome!

Java IoT Authors: Elizabeth White, Liz McMillan, AppDynamics Blog, Jonathan Fries, Tim Hinds

Related Topics: @BigDataExpo, Java IoT, IoT User Interface, @CloudExpo, Apache, SDN Journal

@BigDataExpo: Article

So What? – Monitoring Hadoop Beyond Ganglia

Don’t just run Hadoop jobs at scale, run them efficiently and at scale

Over the last couple of months I have been talking to more and more customers who are either bringing their Hadoop clusters into production or have already done so and are now getting serious about operations. This leads to some interesting discussions about how to monitor Hadoop properly and one thing pops up quite often: Do they need anything beyond Ganglia? If yes, what should they do beyond it?

The Basics
As in every other system, monitoring in a Hadoop environment starts with the basics: System Metrics - CPU, Disk, Memory you know the drill. Of special importance in a Hadoop system is a well-balanced cluster; you don't want to have some nodes being much more (or less) utilized then others. Besides CPU and memory utilization, Disk utilization and of course I/O throughput is of high importance. After all the most likely bottleneck in a Big Data system is I/O - either with ingress (network and disk), moving data around (e.g., MapReduce shuffle on the network) and straightforward read/write to disk.

The problem in a Hadoop system is of course its size. Nothing new for us, some of our customers monitor well beyond 1000+ JVMs with CompuwareAPM. The "advantage" in a Hadoop system is its relative conformity - every node looks pretty much like the other. This is what Ganglia leverages.

Cluster Monitoring with Ganglia
What Ganglia is very good at is providing an overview over how a cluster is utilized. The load chart is particularly interesting:

This chart shows the CPU load on a 1000 Server cluster that has roughly 15.000 CPUs

It tells us the number of available cores in the system and the number of running processes (in theory a core can never handle more than one process at a time) and the 1-min load average. If the system is getting fully utilized the 1-min load average would approach the total number of CPUs. Another view on this is the well-known CPU utilization chart:

CPU Utilization over the last day. While the utilization stays well below 10% we see a lot of I/O wait spikes.

While the load chart gives a good overall impression of usage, the utilization tells us the story of how the CPUs are used. While typical CPU charts show a single server, Ganglia specializes in showing whole clusters (the picture shows CPU usage of a 1000 machine cluster). In the case of the depicted chart we see that the CPUs are experiencing a lot of I/O wait spikes, which points toward heavy disk I/O. Basically it seems the disk I/O is the reason that we cannot utilize our CPU better at these times. But in general our cluster is well underutilized in terms of CPU.

Trends are also easy to understand, as can be seen in this memory chart over a year.

Memory capacity and usage over a year

All this looks pretty good, so what is missing? The "so what" and "why" is what is missing. If my memory demand is growing, I have no way of knowing why it is growing. If the CPU chart tells me that I spend a lot of time waiting, it does not tell what to do, or why that is so? These questions are beyond the scope of Ganglia.

What about Hadoop specifics?
Ganglia also has a Hadoop plugin, which basically gives you access to all the usual Hadoop metrics (unfortunately a comprehensive list of Hadoop metrics is really hard to find, appreciate if somebody commented the link). There is a good explanation on what is interesting on Edward Caproli's page: JoinTheGrid. Basically you can use those metrics to monitor the capacity and usage trends of HDFS and the NameNodes and also how many jobs, mappers and reducers are running.

Capacity of the DataNodes over time

Capacity of the Name Nodes over time

The DataNode Operations give me an impression of I/O pressure on the Hadoop cluster

All these charts can of course be easily built in any modern monitoring or APM solution like CompuwareAPM, but Ganglia gives you a simple starting point; and it's Free as in Beer.

What's missing again is the so what? If my jobs are running a lot longer than yesterday, what should I do? Why do they run longer? A Hadoop expert might dig into 10 different charts around I/O and Network, spilling, look at log files among other things and try an educated guess as to what might be the problem. But we aren't all experts, neither do we have the time to dig into all of these metrics and log files all the time.

This is the reason that we and our customers are moving beyond Ganglia - to solve the "Why" and "So What" within time constraints.

Beyond the Basics #1 - Understanding Cluster Utilization
A use case that we get from customers is that they want to know which users or which pools (in case of the fair scheduler) are responsible for how much of the cluster utilization. LinkedIn just released White Elephant, a tool that parses MapReduce logs and builds some nice dashboards and shows you which of your users occupy how much of your cluster. This is of course based on log file analysis and thus okay for analysis but not for monitoring. With proper tools in place we can do the same thing in near real time.

The CPU Usage in the Hadoop Cluster on per User basis

In this example I wanted to monitor which user consumed how much of my Amazon EMR cluster. If we see a user or pool that occupies a lot of the cluster we can course also see which jobs are running and how much of the cluster they occupy.

The CPU Usage in the Hadoop Cluster on per Job basis

And this will also tell us if that job has always been there, and just uses a lot more resources now. This would be our cue to start analyzing what has changed.

Beyond the Basics #2 - Understanding why my jobs are slow(er)
If we want to understand why a job is slow we need to look at a high-level break down first.

In which phase of the map reduce do we spend the most time, or did we spend more time than yesterday? Understanding these timings in context with the respective job counters, like Map Input or Spilled Records, helps us understand why the phase took longer.

Overview of the time spent in different phases and the respective input/output counters

At this point we will already have a pretty good idea as to what happened. We either simply have more data to crunch (more input data) or a portion of the MapReduce job consumes more CPU (code change?) or we spill more records to disk (code change or Hadoop config change?). We might also detect an unbalanced cluster in the performance breakdown.

This job is executing nearly exclusively on a single node instead of distributing

In this case we want to check whether all the involved nodes processed the same amount of data

Here we see that there is a wide range from minimum, average to maximum on mapped input and output records. The data is not balanced

or if the difference can again be found in the code (different kinds of computations). If we are running against HBase we might of course have an issue with HBase performance or distribution.

At the beginning of the job only a single HBase region Server consumes CPU while all others remain idle

On the other hand, if a lot of mapping time is spent in the garbage collector then you should maybe invest in larger JVMs.

The Performance Breakdown of this particular job shows considerable time in GC suspension

If spilling data to disk is where we spend our time, we should take a closer look at that phase. It might turn out that all of our time is spent on disk wait.

If the Disk were the bottleneck we would see it on disk I/O here

Now if disk write is our bottleneck, then really the only thing that we can do is reduce the map output records. Adding a combiner will not reduce the disk write (it will actually increase it, read here). In other words combining only optimizes the shuffle phase, thus the amount of data sent over the network, but not spill time!!

And at the very detailed level we can look at single task executions and understand in detail what is really going on.

The detailed data about each Map, Reduce Task Atttempt as well as the spills and shuffles

Conclusion
Ganglia is a great tool for high-level monitoring of your Hadoop cluster utilization, but it is not enough. The fact that everybody is working on additional means to understand the Hadoop cluster (Hortonworks with Ambari, Cloudera with their Manager, LinkedIn with White Elephant, the Star Fish project...) shows that there is a lot more needed beyond simple monitoring. Even those more advanced monitoring tools are not always answering the "why" though, which is what we really need to do. This is where the Performance Management discipline can add a lot of value and really help you get the best out of your Hadoop cluster. In other words don't just run Hadoop jobs at scale, run them efficiently and at scale!

More Stories By Michael Kopp

Michael Kopp has over 12 years of experience as an architect and developer in the Enterprise Java space. Before coming to CompuwareAPM dynaTrace he was the Chief Architect at GoldenSource, a major player in the EDM space. In 2009 he joined dynaTrace as a technology strategist in the center of excellence. He specializes application performance management in large scale production environments with special focus on virtualized and cloud environments. His current focus is how to effectively leverage BigData Solutions and how these technologies impact and change the application landscape.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
We're entering the post-smartphone era, where wearable gadgets from watches and fitness bands to glasses and health aids will power the next technological revolution. With mass adoption of wearable devices comes a new data ecosystem that must be protected. Wearables open new pathways that facilitate the tracking, sharing and storing of consumers’ personal health, location and daily activity data. Consumers have some idea of the data these devices capture, but most don’t realize how revealing and...
There is an ever-growing explosion of new devices that are connected to the Internet using “cloud” solutions. This rapid growth is creating a massive new demand for efficient access to data. And it’s not just about connecting to that data anymore. This new demand is bringing new issues and challenges and it is important for companies to scale for the coming growth. And with that scaling comes the need for greater security, gathering and data analysis, storage, connectivity and, of course, the...
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, will explain how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.
Artificial Intelligence has the potential to massively disrupt IoT. In his session at 18th Cloud Expo, AJ Abdallat, CEO of Beyond AI, will discuss what the five main drivers are in Artificial Intelligence that could shape the future of the Internet of Things. AJ Abdallat is CEO of Beyond AI. He has over 20 years of management experience in the fields of artificial intelligence, sensors, instruments, devices and software for telecommunications, life sciences, environmental monitoring, process...
SYS-CON Events announced today that Ericsson has been named “Gold Sponsor” of SYS-CON's @ThingsExpo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. Ericsson is a world leader in the rapidly changing environment of communications technology – providing equipment, software and services to enable transformation through mobility. Some 40 percent of global mobile traffic runs through networks we have supplied. More than 1 billion subscribers around the world re...
We’ve worked with dozens of early adopters across numerous industries and will debunk common misperceptions, which starts with understanding that many of the connected products we’ll use over the next 5 years are already products, they’re just not yet connected. With an IoT product, time-in-market provides much more essential feedback than ever before. Innovation comes from what you do with the data that the connected product provides in order to enhance the customer experience and optimize busi...
Increasing IoT connectivity is forcing enterprises to find elegant solutions to organize and visualize all incoming data from these connected devices with re-configurable dashboard widgets to effectively allow rapid decision-making for everything from immediate actions in tactical situations to strategic analysis and reporting. In his session at 18th Cloud Expo, Shikhir Singh, Senior Developer Relations Manager at Sencha, will discuss how to create HTML5 dashboards that interact with IoT devic...
The increasing popularity of the Internet of Things necessitates that our physical and cognitive relationship with wearable technology will change rapidly in the near future. This advent means logging has become a thing of the past. Before, it was on us to track our own data, but now that data is automatically available. What does this mean for mHealth and the "connected" body? In her session at @ThingsExpo, Lisa Calkins, CEO and co-founder of Amadeus Consulting, will discuss the impact of wea...
Manufacturers are embracing the Industrial Internet the same way consumers are leveraging Fitbits – to improve overall health and wellness. Both can provide consistent measurement, visibility, and suggest performance improvements customized to help reach goals. Fitbit users can view real-time data and make adjustments to increase their activity. In his session at @ThingsExpo, Mark Bernardo Professional Services Leader, Americas, at GE Digital, will discuss how leveraging the Industrial Interne...
The IoTs will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm and share the must-have mindsets for removing complexity from the development proc...
trust and privacy in their ecosystem. Assurance and protection of device identity, secure data encryption and authentication are the key security challenges organizations are trying to address when integrating IoT devices. This holds true for IoT applications in a wide range of industries, for example, healthcare, consumer devices, and manufacturing. In his session at @ThingsExpo, Lancen LaChance, vice president of product management, IoT solutions at GlobalSign, will teach IoT developers how t...
A critical component of any IoT project is the back-end systems that capture data from remote IoT devices and structure it in a way to answer useful questions. Traditional data warehouse and analytical systems are mature technologies that can be used to handle large data sets, but they are not well suited to many IoT-scale products and the need for real-time insights. At Fuze, we have developed a backend platform as part of our mobility-oriented cloud service that uses Big Data-based approache...
In his session at @ThingsExpo, Chris Klein, CEO and Co-founder of Rachio, will discuss next generation communities that are using IoT to create more sustainable, intelligent communities. One example is Sterling Ranch, a 10,000 home development that – with the help of Siemens – will integrate IoT technology into the community to provide residents with energy and water savings as well as intelligent security. Everything from stop lights to sprinkler systems to building infrastructures will run ef...
Digital payments using wearable devices such as smart watches, fitness trackers, and payment wristbands are an increasing area of focus for industry participants, and consumer acceptance from early trials and deployments has encouraged some of the biggest names in technology and banking to continue their push to drive growth in this nascent market. Wearable payment systems may utilize near field communication (NFC), radio frequency identification (RFID), or quick response (QR) codes and barcodes...
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...
You deployed your app with the Bluemix PaaS and it's gaining some serious traction, so it's time to make some tweaks. Did you design your application in a way that it can scale in the cloud? Were you even thinking about the cloud when you built the app? If not, chances are your app is going to break. Check out this webcast to learn various techniques for designing applications that will scale successfully in Bluemix, for the confidence you need to take your apps to the next level and beyond.
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus inter...
So, you bought into the current machine learning craze and went on to collect millions/billions of records from this promising new data source. Now, what do you do with them? Too often, the abundance of data quickly turns into an abundance of problems. How do you extract that "magic essence" from your data without falling into the common pitfalls? In her session at @ThingsExpo, Natalia Ponomareva, Software Engineer at Google, will provide tips on how to be successful in large scale machine lear...
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry's single source for the cloud. Fusion's advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including cloud...