|By Michael Kopp||
|April 29, 2013 10:15 AM EDT||
Over the last couple of months I have been talking to more and more customers who are either bringing their Hadoop clusters into production or have already done so and are now getting serious about operations. This leads to some interesting discussions about how to monitor Hadoop properly and one thing pops up quite often: Do they need anything beyond Ganglia? If yes, what should they do beyond it?
As in every other system, monitoring in a Hadoop environment starts with the basics: System Metrics - CPU, Disk, Memory you know the drill. Of special importance in a Hadoop system is a well-balanced cluster; you don't want to have some nodes being much more (or less) utilized then others. Besides CPU and memory utilization, Disk utilization and of course I/O throughput is of high importance. After all the most likely bottleneck in a Big Data system is I/O - either with ingress (network and disk), moving data around (e.g., MapReduce shuffle on the network) and straightforward read/write to disk.
The problem in a Hadoop system is of course its size. Nothing new for us, some of our customers monitor well beyond 1000+ JVMs with CompuwareAPM. The "advantage" in a Hadoop system is its relative conformity - every node looks pretty much like the other. This is what Ganglia leverages.
Cluster Monitoring with Ganglia
What Ganglia is very good at is providing an overview over how a cluster is utilized. The load chart is particularly interesting:
This chart shows the CPU load on a 1000 Server cluster that has roughly 15.000 CPUs
It tells us the number of available cores in the system and the number of running processes (in theory a core can never handle more than one process at a time) and the 1-min load average. If the system is getting fully utilized the 1-min load average would approach the total number of CPUs. Another view on this is the well-known CPU utilization chart:
CPU Utilization over the last day. While the utilization stays well below 10% we see a lot of I/O wait spikes.
While the load chart gives a good overall impression of usage, the utilization tells us the story of how the CPUs are used. While typical CPU charts show a single server, Ganglia specializes in showing whole clusters (the picture shows CPU usage of a 1000 machine cluster). In the case of the depicted chart we see that the CPUs are experiencing a lot of I/O wait spikes, which points toward heavy disk I/O. Basically it seems the disk I/O is the reason that we cannot utilize our CPU better at these times. But in general our cluster is well underutilized in terms of CPU.
Trends are also easy to understand, as can be seen in this memory chart over a year.
Memory capacity and usage over a year
All this looks pretty good, so what is missing? The "so what" and "why" is what is missing. If my memory demand is growing, I have no way of knowing why it is growing. If the CPU chart tells me that I spend a lot of time waiting, it does not tell what to do, or why that is so? These questions are beyond the scope of Ganglia.
What about Hadoop specifics?
Ganglia also has a Hadoop plugin, which basically gives you access to all the usual Hadoop metrics (unfortunately a comprehensive list of Hadoop metrics is really hard to find, appreciate if somebody commented the link). There is a good explanation on what is interesting on Edward Caproli's page: JoinTheGrid. Basically you can use those metrics to monitor the capacity and usage trends of HDFS and the NameNodes and also how many jobs, mappers and reducers are running.
Capacity of the DataNodes over time
Capacity of the Name Nodes over time
The DataNode Operations give me an impression of I/O pressure on the Hadoop cluster
All these charts can of course be easily built in any modern monitoring or APM solution like CompuwareAPM, but Ganglia gives you a simple starting point; and it's Free as in Beer.
What's missing again is the so what? If my jobs are running a lot longer than yesterday, what should I do? Why do they run longer? A Hadoop expert might dig into 10 different charts around I/O and Network, spilling, look at log files among other things and try an educated guess as to what might be the problem. But we aren't all experts, neither do we have the time to dig into all of these metrics and log files all the time.
This is the reason that we and our customers are moving beyond Ganglia - to solve the "Why" and "So What" within time constraints.
Beyond the Basics #1 - Understanding Cluster Utilization
A use case that we get from customers is that they want to know which users or which pools (in case of the fair scheduler) are responsible for how much of the cluster utilization. LinkedIn just released White Elephant, a tool that parses MapReduce logs and builds some nice dashboards and shows you which of your users occupy how much of your cluster. This is of course based on log file analysis and thus okay for analysis but not for monitoring. With proper tools in place we can do the same thing in near real time.
The CPU Usage in the Hadoop Cluster on per User basis
In this example I wanted to monitor which user consumed how much of my Amazon EMR cluster. If we see a user or pool that occupies a lot of the cluster we can course also see which jobs are running and how much of the cluster they occupy.
The CPU Usage in the Hadoop Cluster on per Job basis
And this will also tell us if that job has always been there, and just uses a lot more resources now. This would be our cue to start analyzing what has changed.
Beyond the Basics #2 - Understanding why my jobs are slow(er)
If we want to understand why a job is slow we need to look at a high-level break down first.
In which phase of the map reduce do we spend the most time, or did we spend more time than yesterday? Understanding these timings in context with the respective job counters, like Map Input or Spilled Records, helps us understand why the phase took longer.
Overview of the time spent in different phases and the respective input/output counters
At this point we will already have a pretty good idea as to what happened. We either simply have more data to crunch (more input data) or a portion of the MapReduce job consumes more CPU (code change?) or we spill more records to disk (code change or Hadoop config change?). We might also detect an unbalanced cluster in the performance breakdown.
This job is executing nearly exclusively on a single node instead of distributing
In this case we want to check whether all the involved nodes processed the same amount of data
Here we see that there is a wide range from minimum, average to maximum on mapped input and output records. The data is not balanced
or if the difference can again be found in the code (different kinds of computations). If we are running against HBase we might of course have an issue with HBase performance or distribution.
At the beginning of the job only a single HBase region Server consumes CPU while all others remain idle
On the other hand, if a lot of mapping time is spent in the garbage collector then you should maybe invest in larger JVMs.
The Performance Breakdown of this particular job shows considerable time in GC suspension
If spilling data to disk is where we spend our time, we should take a closer look at that phase. It might turn out that all of our time is spent on disk wait.
If the Disk were the bottleneck we would see it on disk I/O here
Now if disk write is our bottleneck, then really the only thing that we can do is reduce the map output records. Adding a combiner will not reduce the disk write (it will actually increase it, read here). In other words combining only optimizes the shuffle phase, thus the amount of data sent over the network, but not spill time!!
And at the very detailed level we can look at single task executions and understand in detail what is really going on.
The detailed data about each Map, Reduce Task Atttempt as well as the spills and shuffles
Ganglia is a great tool for high-level monitoring of your Hadoop cluster utilization, but it is not enough. The fact that everybody is working on additional means to understand the Hadoop cluster (Hortonworks with Ambari, Cloudera with their Manager, LinkedIn with White Elephant, the Star Fish project...) shows that there is a lot more needed beyond simple monitoring. Even those more advanced monitoring tools are not always answering the "why" though, which is what we really need to do. This is where the Performance Management discipline can add a lot of value and really help you get the best out of your Hadoop cluster. In other words don't just run Hadoop jobs at scale, run them efficiently and at scale!
GENBAND has announced that SageNet is leveraging the Nuvia platform to deliver Unified Communications as a Service (UCaaS) to its large base of retail and enterprise customers. Nuvia’s cloud-based solution provides SageNet’s customers with a full suite of business communications and collaboration tools. Two large national SageNet retail customers have recently signed up to deploy the Nuvia platform and the company will continue to sell the service to new and existing customers. Nuvia’s capabilities include HD voice, video, multimedia messaging, mobility, conferencing, Web collaboration, deskt...
Mar. 27, 2015 01:00 AM EDT Reads: 1,309
The Open Compute Project is a collective effort by Facebook and a number of players in the datacenter industry to bring lessons learned from the social media giant's giant IT deployment to the rest of the world. Datacenters account for 3% of global electricity consumption – about the same as all of Switzerland or the Czech Republic -- according to people I met at the recent Open Compute Summit in San Jose. With increasing mobility at the edge of the cloud and vast new dataflows being predicted with the growth of the Internet of Things (and The Coming Age of Many Zettabytes) in the near...
Mar. 27, 2015 01:00 AM EDT Reads: 1,697
Wearable technology was dominant at this year’s International Consumer Electronics Show (CES) , and MWC was no exception to this trend. New versions of favorites, such as the Samsung Gear (three new products were released: the Gear 2, the Gear 2 Neo and the Gear Fit), shared the limelight with new wearables like Pebble Time Steel (the new premium version of the company’s previously released smartwatch) and the LG Watch Urbane. The most dramatic difference at MWC was an emphasis on presenting wearables as fashion accessories and moving away from the original clunky technology associated with t...
Mar. 26, 2015 11:45 PM EDT Reads: 972
The WebRTC Summit 2014 New York, to be held June 9-11, 2015, at the Javits Center in New York, NY, announces that its Call for Papers is open. Topics include all aspects of improving IT delivery by eliminating waste through automated business models leveraging cloud technologies. WebRTC Summit is co-located with 16th International Cloud Expo, @ThingsExpo, Big Data Expo, and DevOps Summit.
Mar. 26, 2015 10:45 PM EDT Reads: 1,203
SYS-CON Events announced today that Cisco, the worldwide leader in IT that transforms how people connect, communicate and collaborate, has been named “Gold Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Cisco makes amazing things happen by connecting the unconnected. Cisco has shaped the future of the Internet by becoming the worldwide leader in transforming how people connect, communicate and collaborate. Cisco and our partners are building the platform for the Internet of Everything by connecting the...
Mar. 26, 2015 07:00 PM EDT Reads: 5,014
15th Cloud Expo, which took place Nov. 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA, expanded the conference content of @ThingsExpo, Big Data Expo, and DevOps Summit to include two developer events. IBM held a Bluemix Developer Playground on November 5 and ElasticBox held a Hackathon on November 6. Both events took place on the expo floor. The Bluemix Developer Playground, for developers of all levels, highlighted the ease of use of Bluemix, its services and functionality and provide short-term introductory projects that developers can complete between sessions.
Mar. 26, 2015 06:30 PM EDT Reads: 4,594
Temasys has announced senior management additions to its team. Joining are David Holloway as Vice President of Commercial and Nadine Yap as Vice President of Product. Over the past 12 months Temasys has doubled in size as it adds new customers and expands the development of its Skylink platform. Skylink leads the charge to move WebRTC, traditionally seen as a desktop, browser based technology, to become a ubiquitous web communications technology on web and mobile, as well as Internet of Things compatible devices.
Mar. 26, 2015 06:00 PM EDT Reads: 1,642
SYS-CON Events announced today that robomq.io will exhibit at SYS-CON's @ThingsExpo, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. robomq.io is an interoperable and composable platform that connects any device to any application. It helps systems integrators and the solution providers build new and innovative products and service for industries requiring monitoring or intelligence from devices and sensors.
Mar. 26, 2015 06:00 PM EDT Reads: 1,226
The list of ‘new paradigm’ technologies that now surrounds us appears to be at an all time high. From cloud computing and Big Data analytics to Bring Your Own Device (BYOD) and the Internet of Things (IoT), today we have to deal with what the industry likes to call ‘paradigm shifts’ at every level of IT. This is disruption; of course, we understand that – change is almost always disruptive.
Mar. 26, 2015 05:15 PM EDT Reads: 827
WebRTC is an up-and-coming standard that enables real-time voice and video to be directly embedded into browsers making the browser a primary user interface for communications and collaboration. WebRTC runs in a number of browsers today and is currently supported in over a billion installed browsers globally, across a range of platform OS and devices. Today, organizations that choose to deploy WebRTC applications and use a host machine that supports audio through USB or Bluetooth can use Plantronics products to connect and transit or receive the audio associated with the WebRTC session.
Mar. 26, 2015 05:00 PM EDT Reads: 1,464
Docker is an excellent platform for organizations interested in running microservices. It offers portability and consistency between development and production environments, quick provisioning times, and a simple way to isolate services. In his session at DevOps Summit at 16th Cloud Expo, Shannon Williams, co-founder of Rancher Labs, will walk through these and other benefits of using Docker to run microservices, and provide an overview of RancherOS, a minimalist distribution of Linux designed expressly to run Docker. He will also discuss Rancher, an orchestration and service discovery platf...
Mar. 26, 2015 04:15 PM EDT Reads: 2,287
Sonus Networks introduced the Sonus WebRTC Services Solution, a virtualized Web Real-Time Communications (WebRTC) offer, purpose-built for the Cloud. The WebRTC Services Solution provides signaling from WebRTC-to-WebRTC applications and interworking from WebRTC-to-Session Initiation Protocol (SIP), delivering advanced real-time communications capabilities on mobile applications and on websites, which are accessible via a browser.
Mar. 26, 2015 04:00 PM EDT Reads: 1,612
SYS-CON Events announced today that Aria Systems, the leading innovator in recurring revenue, has been named “Bronze Sponsor” of SYS-CON's @ThingsExpo, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Proven by the world’s most demanding enterprises, including AAA NCNU, Constant Contact, Falck, Hootsuite, Pitney Bowes, Telekom Denmark, and VMware, Aria helps enterprises grow their recurring revenue businesses. With Aria’s end-to-end active monetization platform, global brands can get to market faster with a wider variety of products and services, while maximizin...
Mar. 26, 2015 04:00 PM EDT Reads: 1,451
SYS-CON Media announced today that @WebRTCSummit Blog, the largest WebRTC resource in the world, has been launched. @WebRTCSummit Blog offers top articles, news stories, and blog posts from the world's well-known experts and guarantees better exposure for its authors than any other publication. @WebRTCSummit Blog can be bookmarked ▸ Here @WebRTCSummit conference site can be bookmarked ▸ Here
Mar. 26, 2015 04:00 PM EDT Reads: 1,551
SYS-CON Events announced today that Alert Logic, the leading provider of Security-as-a-Service solutions for the cloud, has been named “Bronze Sponsor” of SYS-CON's 16th International Cloud Expo® and DevOps Summit 2015 New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY, and the 17th International Cloud Expo® and DevOps Summit 2015 Silicon Valley, which will take place November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
Mar. 26, 2015 04:00 PM EDT Reads: 1,457
SYS-CON Events announced today that Vitria Technology, Inc. will exhibit at SYS-CON’s @ThingsExpo, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Vitria will showcase the company’s new IoT Analytics Platform through live demonstrations at booth #330. Vitria’s IoT Analytics Platform, fully integrated and powered by an operational intelligence engine, enables customers to rapidly build and operationalize advanced analytics to deliver timely business outcomes for use cases across the industrial, enterprise, and consumer segments.
Mar. 26, 2015 03:30 PM EDT Reads: 2,040
SYS-CON Events announced today that Solgenia will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY, and the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. Solgenia is the global market leader in Cloud Collaboration and Cloud Infrastructure software solutions. Designed to “Bridge the Gap” between Personal and Professional Social, Mobile and Cloud user experiences, our solutions help large and medium-sized organizations dr...
Mar. 26, 2015 03:00 PM EDT Reads: 2,521
SYS-CON Events announced today that Liaison Technologies, a leading provider of data management and integration cloud services and solutions, has been named "Silver Sponsor" of SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York, NY. Liaison Technologies is a recognized market leader in providing cloud-enabled data integration and data management solutions to break down complex information barriers, enabling enterprises to make smarter decisions, faster.
Mar. 26, 2015 03:00 PM EDT Reads: 3,305
Connected devices and the Internet of Things are getting significant momentum in 2014. In his session at Internet of @ThingsExpo, Jim Hunter, Chief Scientist & Technology Evangelist at Greenwave Systems, examined three key elements that together will drive mass adoption of the IoT before the end of 2015. The first element is the recent advent of robust open source protocols (like AllJoyn and WebRTC) that facilitate M2M communication. The second is broad availability of flexible, cost-effective storage designed to handle the massive surge in back-end data in a world where timely analytics is e...
Mar. 26, 2015 02:45 PM EDT Reads: 4,629
SYS-CON Events announced today that Akana, formerly SOA Software, has been named “Bronze Sponsor” of SYS-CON's 16th International Cloud Expo® New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. Akana’s comprehensive suite of API Management, API Security, Integrated SOA Governance, and Cloud Integration solutions helps businesses accelerate digital transformation by securely extending their reach across multiple channels – mobile, cloud and Internet of Things. Akana enables enterprises to share data as APIs, connect and integrate applications, drive part...
Mar. 26, 2015 02:15 PM EDT Reads: 1,306