Welcome!

Java Authors: Elizabeth White, Pat Romanski, Liz McMillan, Jason Bloomberg, Trevor Parsons

Related Topics: Oracle, Java, SOA & WOA, Websphere

Oracle: Article

How to Diagnose Java Resource Starvation

Using the IBM Thread & Monitor Dump Analyzer for Java

We can visualize resource starvation using an elaborate rendition of the Dining Philosophers Problem. This classic metaphor of resource allocation among processes was first introduced in 1971 by Edsger Dijkstra in his paper "Hierarchical Ordering of Sequential Processes." It's been a model and universal method for verifying theories on resource allocation ever since.

The metaphor goes like this: There are three well-known philosophers in an Asian bistro. Dinner is served but they are only given three chopsticks because the restaurant's supply truck has been stuck in a snow storm for a couple of days. Naturally each philosopher needs two chopsticks to eat his dinner and each is protected from interference while he uses a chopstick.

Plato skipped lunch that day and insists that he should have priority or else he'll faint. If he doesn't give up his chopsticks, the other philosophers won't be able to eat and eventually they'll starve. If Plato drops his chopsticks and immediately picks them up before the other philosophers can get them then they won't be able to eat and they'll eventually starve.

Dijkstra's Dining Philosophers Problem employed spaghetti and forks but I used chopsticks and Asian cuisine for a more realistic illustration since you can't eat spaghetti with a single chopstick (see Figure 1).

These are examples of resource starvation. You could also encounter deadlock. There are a couple of solutions like semaphores and monitors. But we won't think about solutions or deadlocks in this article. We'll focus on detecting resource starvation using IBM Thread and Monitor Dump Analyzer for Java.

What Is the IBM Thread and Monitor Dump Analyzer for Java?
During the runtime of a Java process, some Java Virtual Machines (JVMs) may not respond predictably and often seem to hang up for a long time or indefinitely (until the JVM shuts down). It's sometimes very challenging to determine the root causes of these sorts of problems.

By triggering Java thread dumps or Javacores when a Java process doesn't respond, it's possible to collect diagnostic information related to the JVM and a Java application captured at a particular point during execution.

On some platforms, a Javacore is known as "javadump." The code that creates a Javacore is part of the JVM. One can control it by using environment variables and runtime switches. By default, a Javacore occurs when the JVM terminates unexpectedly. A Javacore can also be triggered by sending specific signals to the JVM. Although a Javacore or javadump can occur in Solaris JVMs, much of the content of the Javacore is added by IBM and, so, is present only in IBM JVMs.

IBM Thread and Monitor Dump Analyzer for Java analyzes thread dumps/Javacores and diagnoses monitor locks and thread activities to identify the root causes of hangs, deadlocks, and resource contention or monitor bottlenecks. It can analyze IBM, Solaris, and HP-UX Java thread dumps from Java Virtual Machine 1.3.1, 1.4.x, 5.0, and 6.0

More than three years ago, I created IBM Thread and Monitor Dump Analyzer for Java as a garage project and spent thousands of hours during weeknights, weekends, and vacations to provide my clients and colleagues with something that can help diagnose these problems easily, motivated by the same reasons as my other garage projects like IBM HeapAnalyzer, IBM Pattern Modeling and Analysis Tool, and IBM Performance Analyzer for Java for Windows. All the algorithms used have been filed with the U.S. Patent and Trademark Office.

The widgetry was introduced to the public about two years ago through alphaWorks where it's been one of the top downloads for 30 consecutive months. It's recorded 23,296 downloads so far and it's been used by more than 14,600 companies, government agencies, research facilities, and universities worldwide (these statistics are good as of October 2008).

What Is a Thread Dump?
A Java thread dump is the one of the traces/dumps that JVM provides to help diagnosis a hang, deadlock, or monitor contention issue. It contains diagnostic information related to the JVM and a Java application captured at a point during execution. For example, the information can be a list of all the threads that run on a Java virtual machine. Usually thread dumps have more than just thread information. Thread dumps can produce information about the operating system, application environment, threads, stacks, locks, and memory. IBM Thread Dumps/Javacores provide much more information like core interface, data conversion/unicode, class, diagnosis, execution management/thread management, lock, execution engine, mixed mode interpreter, JIT (Just-in-Time compiler), storage/JVM heap, and hardware portability interface/extended hardware portability interface.

The contents and formats of Java thread dumps depend on the platform you're running on and the JVM provider.

Since a thread dump's format isn't a part of the SDK specification, each SDK vendor provides a unique thread dump format and its own JVM information. Currently IBM Thread and Monitor Dump Analyzer for Java understands IBM, Solaris, and HP-UX Java thread dump formats.

By default, IBM JVM thread dumps occur when the JVM terminates unexpectedly. A thread dump can also be triggered by sending specific signals to the JVM. Thread dumps are usually human readable text. In an IBM JVM, the thread dump is often called a Javadump or Javacore. Javacore is NOT the same as a core file, which is generated by a system dump.

Before IBM Thread and Monitor Dump Analyzer for Java was introduced, people had to manually check hundreds of threads in the thread dumps using their favorite editors. With IBM Thread and Monitor Dump Analyzer for Java, we can easily diagnose a variety of issues without reading through hundreds of thread stack traces.

Usually thread dumps aren't as large as heap dumps. Usually its size is around 1MB or less unless you have large number of threads or classes. It's not a big deal to manually review thread dumps if you only have a handful of threads running on your JVM. If you have hundreds of threads with dozens of Java monitors, it gets complicated.

How Does It Work?
This technology analyzes the information in each thread and provides diagnostic information, such as current thread information, the signal that caused the Javacore/Java heap information (maximum Java heap size, initial Java heap size, garbage collector counter, allocation failure counter, free Java heap size, and allocated Java heap size), number of runnable threads, total number of threads, number of monitors locked, native memory usage of each JVM component, and deadlock information.

IBM Thread and Monitor Dump Analyzer for Java also provides you with the recommended size of the Java heap cluster (applicable only to IBM SDK 1.4.2 and 1.3.1 SR7 or above) based on a heuristic analysis engine.

It compares each Javacore and provides process ID information for threads, a time stamp of the first Javacore, a time stamp of the last Javacore, the number of garbage collections a minute, the number of allocation failures a minute, the time between the first Javacore and the last Javacore, the number of hang suspects, and a list of hang suspects.

This technology also compares all monitor information in a Javacore and detects deadlock and resource contention or monitor bottlenecks, if there are any.

Let's Get Started
We can simulate a starvation problem with a simple Java application.

In DiningPhilosophers.java, we created three chopsticks, c1, c2, and c3 with the names, chopstick 1, chopstick 2, and chopstick3. Then we created three philosophers, p1, p2, and p3 with the names, Socrates, Plato, and Aristotle. You might have noticed that each philosopher's name has a suffix that represents a priority since we want to give Plato more of a chance to eat. Besides a name and priority, each philosopher is also given a left chopstick and a right chopstick. We start with each philosopher, p1, p2,  and p3 as shown in Listing 1. (Download Listings 1-6 here.)

In Chopstick.java, we just create a constructor, a getter, and a setter for the name of chopstick as shown in Listing 2

In Philosopher.java, we create a constructor, getters and setters for right and left chopsticks, and the philosopher's name. We also override the run() method in which we acquire Java's built-in monitors for left and right chopsticks and call the eat() method. In each () method, we pause for two seconds to simulate eating with the sleep() method but keep in mind that the philosopher is eating, not sleeping in the eat() method. See Listing 3.

Now let's put all the classes in a subdirectory, compile, and run them.

Compiling classes on a Unix platform
javac com/ibm/Jinwoo/starvation/DiningPhilosophers.java

Compiling classes on a Windows platform
javac com\ibm\jinwoo\starvation\DiningPhilosophers.java

I used the latest release of IBM Java virtual machine 6.0 (build jvmwi3260-20080816_22093 VM build 20080816_022093_lHdSMr) and another vendor's latest release of a Java virtual machine 6.0 when I wrote this for a fair comparison.

Executing classes
java com.ibm.jinwoo.starvation.DiningPhilosophers
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1

Most of the time Plato will be eating and the other two philosophers will have less of a chance to eat. We can generate thread dumps by sending signal 3 to the process. On the Windows platform, you can use keyboard combinations like Ctrl-Break to send signal 3 to a process running in a command line window. Listing 4 has the results from the IBM Java virtual machine.

We used another vendor's Java virtual machine and redirected output to a file.

Generating thread dumps with another vendor's Java virtual machine
java com.ibm.jinwoo.starvation.DiningPhilosophers 1> sun.log

Now let's talk about thread dump a little more. In most Java platforms, thread dumps are generated when the JVM terminates unexpectedly because of an operating system signal or when the user enters a reserved key combination control-break on Windows, say. We can also generate thread dumps with IBM's JavaDump API: com.ibm.jvm.Dump.JavaDump() if you're running a IBM Java virtual machine.

You can use the kill command on most Unix systems:

  • SIGQUIT[3] : JVM will not terminate

For example, kill -3 $PID

  • SIGSEGV[11] : JVM will terminate

For example, kill -11 $PID

On the IBM WebSphere Application Server 5/6 environment, you can use wsadmin script to generate thread dumps:

wsadmin.sh -conntype SOAP -port <port number>
> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
> $AdminControl invoke $jvm dumpThreads

More Stories By Jinwoo Hwang

Jinwoo Hwang is a software engineer, inventor, author, and technical leader at IBM WebSphere Application Server Technical Support in Research Triangle Park, North Carolina. He joined IBM in 1995 and worked with IBM Global Learning Services, IBM Consulting Services, and software development teams prior to his current position at IBM. He is an IBM Certified Solution Developer and IBM Certified WebSphere Application Server System Administrator as well as a SUN Certified Programmer for the Java platform. He is the architect and creator of the following technologies:

Mr. Hwang is the author of the book C Programming for Novices (ISBN:9788985553643, Yonam Press, 1995) as well as the following webcasts and articles:

Mr. Hwang is the author of the following IBM technical articles:

  • VisualAge Performance Guide,1999
  • CORBA distributed object applet/servlet programming for IBM WebSphere Application Server and VisualAge for Java v2.0E ,1999
  • Java CORBA programming for VisualAge for Java ,1998
  • MVS/CICS application programming for VisualAge Generator ,1998
  • Oracle Native/ODBC application programming for VisualAge Generator ,1998
  • MVS/CICS application Web connection programming for VisualAge Generator ,1998
  • Java applet programming for VisualAge WebRunner ,1998
  • VisualAge for Java/WebRunner Server Works Java Servlet Programming Guide ,1998
  • RMI Java Applet programming for VisualAge for Java ,1998
  • Multimedia Database Java Applet Programming Guide ,1997
  • CICS ECI Java Applet programming guide for VisualAge Generator 3.0 ,1997
  • CICS ECI DB2 Application programming guide for VigualGen, 1997
  • VisualGen CICS ECI programming guide, 1997
  • VisualGen CICS DPL programming guide, 1997

Mr. Hwang holds the following patents in the U.S. / other countries:


Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
One of the biggest challenges when developing connected devices is identifying user value and delivering it through successful user experiences. In his session at Internet of @ThingsExpo, Mike Kuniavsky, Principal Scientist, Innovation Services at PARC, described an IoT-specific approach to user experience design that combines approaches from interaction design, industrial design and service design to create experiences that go beyond simple connected gadgets to create lasting, multi-device experiences grounded in people's real needs and desires.
Enthusiasm for the Internet of Things has reached an all-time high. In 2013 alone, venture capitalists spent more than $1 billion dollars investing in the IoT space. With "smart" appliances and devices, IoT covers wearable smart devices, cloud services to hardware companies. Nest, a Google company, detects temperatures inside homes and automatically adjusts it by tracking its user's habit. These technologies are quickly developing and with it come challenges such as bridging infrastructure gaps, abiding by privacy concerns and making the concept a reality. These challenges can't be addressed w...
The Domain Name Service (DNS) is one of the most important components in networking infrastructure, enabling users and services to access applications by translating URLs (names) into IP addresses (numbers). Because every icon and URL and all embedded content on a website requires a DNS lookup loading complex sites necessitates hundreds of DNS queries. In addition, as more internet-enabled ‘Things' get connected, people will rely on DNS to name and find their fridges, toasters and toilets. According to a recent IDG Research Services Survey this rate of traffic will only grow. What's driving t...
Scott Jenson leads a project called The Physical Web within the Chrome team at Google. Project members are working to take the scalability and openness of the web and use it to talk to the exponentially exploding range of smart devices. Nearly every company today working on the IoT comes up with the same basic solution: use my server and you'll be fine. But if we really believe there will be trillions of these devices, that just can't scale. We need a system that is open a scalable and by using the URL as a basic building block, we open this up and get the same resilience that the web enjoys.
Connected devices and the Internet of Things are getting significant momentum in 2014. In his session at Internet of @ThingsExpo, Jim Hunter, Chief Scientist & Technology Evangelist at Greenwave Systems, examined three key elements that together will drive mass adoption of the IoT before the end of 2015. The first element is the recent advent of robust open source protocols (like AllJoyn and WebRTC) that facilitate M2M communication. The second is broad availability of flexible, cost-effective storage designed to handle the massive surge in back-end data in a world where timely analytics is e...
We are reaching the end of the beginning with WebRTC, and real systems using this technology have begun to appear. One challenge that faces every WebRTC deployment (in some form or another) is identity management. For example, if you have an existing service – possibly built on a variety of different PaaS/SaaS offerings – and you want to add real-time communications you are faced with a challenge relating to user management, authentication, authorization, and validation. Service providers will want to use their existing identities, but these will have credentials already that are (hopefully) i...
"Matrix is an ambitious open standard and implementation that's set up to break down the fragmentation problems that exist in IP messaging and VoIP communication," explained John Woolf, Technical Evangelist at Matrix, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happens, where data lives and where the interface lies. For instance, it's a mix of architectural styles ...
The Internet of Things will put IT to its ultimate test by creating infinite new opportunities to digitize products and services, generate and analyze new data to improve customer satisfaction, and discover new ways to gain a competitive advantage across nearly every industry. In order to help corporate business units to capitalize on the rapidly evolving IoT opportunities, IT must stand up to a new set of challenges. In his session at @ThingsExpo, Jeff Kaplan, Managing Director of THINKstrategies, will examine why IT must finally fulfill its role in support of its SBUs or face a new round of...
Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water, are pursuing SmartGrid initiatives that represent one of the more mature examples of SAE. We have s...
The Internet of Things will greatly expand the opportunities for data collection and new business models driven off of that data. In her session at @ThingsExpo, Esmeralda Swartz, CMO of MetraTech, discussed how for this to be effective you not only need to have infrastructure and operational models capable of utilizing this new phenomenon, but increasingly service providers will need to convince a skeptical public to participate. Get ready to show them the money!
P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at @ThingsExpo, Robin Raymond, Chief Architect at Hookflash, will walk through the shifting landscape of traditional telephone and voice services ...
The Internet of Things is tied together with a thin strand that is known as time. Coincidentally, at the core of nearly all data analytics is a timestamp. When working with time series data there are a few core principles that everyone should consider, especially across datasets where time is the common boundary. In his session at Internet of @ThingsExpo, Jim Scott, Director of Enterprise Strategy & Architecture at MapR Technologies, discussed single-value, geo-spatial, and log time series data. By focusing on enterprise applications and the data center, he will use OpenTSDB as an example t...
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, described how to revolutioniz...
Bit6 today issued a challenge to the technology community implementing Web Real Time Communication (WebRTC). To leap beyond WebRTC’s significant limitations and fully leverage its underlying value to accelerate innovation, application developers need to consider the entire communications ecosystem.
The definition of IoT is not new, in fact it’s been around for over a decade. What has changed is the public's awareness that the technology we use on a daily basis has caught up on the vision of an always on, always connected world. If you look into the details of what comprises the IoT, you’ll see that it includes everything from cloud computing, Big Data analytics, “Things,” Web communication, applications, network, storage, etc. It is essentially including everything connected online from hardware to software, or as we like to say, it’s an Internet of many different things. The difference ...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world.
SYS-CON Events announced today that Windstream, a leading provider of advanced network and cloud communications, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Windstream (Nasdaq: WIN), a FORTUNE 500 and S&P 500 company, is a leading provider of advanced network communications, including cloud computing and managed services, to businesses nationwide. The company also offers broadband, phone and digital TV services to consumers primarily in rural areas.
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
The major cloud platforms defy a simple, side-by-side analysis. Each of the major IaaS public-cloud platforms offers their own unique strengths and functionality. Options for on-site private cloud are diverse as well, and must be designed and deployed while taking existing legacy architecture and infrastructure into account. Then the reality is that most enterprises are embarking on a hybrid cloud strategy and programs. In this Power Panel at 15th Cloud Expo (http://www.CloudComputingExpo.com), moderated by Ashar Baig, Research Director, Cloud, at Gigaom Research, Nate Gordon, Director of T...