Welcome!

Java IoT Authors: Pat Romanski, Carmen Gonzalez, Elizabeth White, Liz McMillan, Yeshim Deniz

Related Topics: Recurring Revenue, Java IoT, Microservices Expo, IBM Cloud

Recurring Revenue: Article

How to Diagnose Java Resource Starvation

Using the IBM Thread & Monitor Dump Analyzer for Java

We can visualize resource starvation using an elaborate rendition of the Dining Philosophers Problem. This classic metaphor of resource allocation among processes was first introduced in 1971 by Edsger Dijkstra in his paper "Hierarchical Ordering of Sequential Processes." It's been a model and universal method for verifying theories on resource allocation ever since.

The metaphor goes like this: There are three well-known philosophers in an Asian bistro. Dinner is served but they are only given three chopsticks because the restaurant's supply truck has been stuck in a snow storm for a couple of days. Naturally each philosopher needs two chopsticks to eat his dinner and each is protected from interference while he uses a chopstick.

Plato skipped lunch that day and insists that he should have priority or else he'll faint. If he doesn't give up his chopsticks, the other philosophers won't be able to eat and eventually they'll starve. If Plato drops his chopsticks and immediately picks them up before the other philosophers can get them then they won't be able to eat and they'll eventually starve.

Dijkstra's Dining Philosophers Problem employed spaghetti and forks but I used chopsticks and Asian cuisine for a more realistic illustration since you can't eat spaghetti with a single chopstick (see Figure 1).

These are examples of resource starvation. You could also encounter deadlock. There are a couple of solutions like semaphores and monitors. But we won't think about solutions or deadlocks in this article. We'll focus on detecting resource starvation using IBM Thread and Monitor Dump Analyzer for Java.

What Is the IBM Thread and Monitor Dump Analyzer for Java?
During the runtime of a Java process, some Java Virtual Machines (JVMs) may not respond predictably and often seem to hang up for a long time or indefinitely (until the JVM shuts down). It's sometimes very challenging to determine the root causes of these sorts of problems.

By triggering Java thread dumps or Javacores when a Java process doesn't respond, it's possible to collect diagnostic information related to the JVM and a Java application captured at a particular point during execution.

On some platforms, a Javacore is known as "javadump." The code that creates a Javacore is part of the JVM. One can control it by using environment variables and runtime switches. By default, a Javacore occurs when the JVM terminates unexpectedly. A Javacore can also be triggered by sending specific signals to the JVM. Although a Javacore or javadump can occur in Solaris JVMs, much of the content of the Javacore is added by IBM and, so, is present only in IBM JVMs.

IBM Thread and Monitor Dump Analyzer for Java analyzes thread dumps/Javacores and diagnoses monitor locks and thread activities to identify the root causes of hangs, deadlocks, and resource contention or monitor bottlenecks. It can analyze IBM, Solaris, and HP-UX Java thread dumps from Java Virtual Machine 1.3.1, 1.4.x, 5.0, and 6.0

More than three years ago, I created IBM Thread and Monitor Dump Analyzer for Java as a garage project and spent thousands of hours during weeknights, weekends, and vacations to provide my clients and colleagues with something that can help diagnose these problems easily, motivated by the same reasons as my other garage projects like IBM HeapAnalyzer, IBM Pattern Modeling and Analysis Tool, and IBM Performance Analyzer for Java for Windows. All the algorithms used have been filed with the U.S. Patent and Trademark Office.

The widgetry was introduced to the public about two years ago through alphaWorks where it's been one of the top downloads for 30 consecutive months. It's recorded 23,296 downloads so far and it's been used by more than 14,600 companies, government agencies, research facilities, and universities worldwide (these statistics are good as of October 2008).

What Is a Thread Dump?
A Java thread dump is the one of the traces/dumps that JVM provides to help diagnosis a hang, deadlock, or monitor contention issue. It contains diagnostic information related to the JVM and a Java application captured at a point during execution. For example, the information can be a list of all the threads that run on a Java virtual machine. Usually thread dumps have more than just thread information. Thread dumps can produce information about the operating system, application environment, threads, stacks, locks, and memory. IBM Thread Dumps/Javacores provide much more information like core interface, data conversion/unicode, class, diagnosis, execution management/thread management, lock, execution engine, mixed mode interpreter, JIT (Just-in-Time compiler), storage/JVM heap, and hardware portability interface/extended hardware portability interface.

The contents and formats of Java thread dumps depend on the platform you're running on and the JVM provider.

Since a thread dump's format isn't a part of the SDK specification, each SDK vendor provides a unique thread dump format and its own JVM information. Currently IBM Thread and Monitor Dump Analyzer for Java understands IBM, Solaris, and HP-UX Java thread dump formats.

By default, IBM JVM thread dumps occur when the JVM terminates unexpectedly. A thread dump can also be triggered by sending specific signals to the JVM. Thread dumps are usually human readable text. In an IBM JVM, the thread dump is often called a Javadump or Javacore. Javacore is NOT the same as a core file, which is generated by a system dump.

Before IBM Thread and Monitor Dump Analyzer for Java was introduced, people had to manually check hundreds of threads in the thread dumps using their favorite editors. With IBM Thread and Monitor Dump Analyzer for Java, we can easily diagnose a variety of issues without reading through hundreds of thread stack traces.

Usually thread dumps aren't as large as heap dumps. Usually its size is around 1MB or less unless you have large number of threads or classes. It's not a big deal to manually review thread dumps if you only have a handful of threads running on your JVM. If you have hundreds of threads with dozens of Java monitors, it gets complicated.

How Does It Work?
This technology analyzes the information in each thread and provides diagnostic information, such as current thread information, the signal that caused the Javacore/Java heap information (maximum Java heap size, initial Java heap size, garbage collector counter, allocation failure counter, free Java heap size, and allocated Java heap size), number of runnable threads, total number of threads, number of monitors locked, native memory usage of each JVM component, and deadlock information.

IBM Thread and Monitor Dump Analyzer for Java also provides you with the recommended size of the Java heap cluster (applicable only to IBM SDK 1.4.2 and 1.3.1 SR7 or above) based on a heuristic analysis engine.

It compares each Javacore and provides process ID information for threads, a time stamp of the first Javacore, a time stamp of the last Javacore, the number of garbage collections a minute, the number of allocation failures a minute, the time between the first Javacore and the last Javacore, the number of hang suspects, and a list of hang suspects.

This technology also compares all monitor information in a Javacore and detects deadlock and resource contention or monitor bottlenecks, if there are any.

Let's Get Started
We can simulate a starvation problem with a simple Java application.

In DiningPhilosophers.java, we created three chopsticks, c1, c2, and c3 with the names, chopstick 1, chopstick 2, and chopstick3. Then we created three philosophers, p1, p2, and p3 with the names, Socrates, Plato, and Aristotle. You might have noticed that each philosopher's name has a suffix that represents a priority since we want to give Plato more of a chance to eat. Besides a name and priority, each philosopher is also given a left chopstick and a right chopstick. We start with each philosopher, p1, p2,  and p3 as shown in Listing 1. (Download Listings 1-6 here.)

In Chopstick.java, we just create a constructor, a getter, and a setter for the name of chopstick as shown in Listing 2

In Philosopher.java, we create a constructor, getters and setters for right and left chopsticks, and the philosopher's name. We also override the run() method in which we acquire Java's built-in monitors for left and right chopsticks and call the eat() method. In each () method, we pause for two seconds to simulate eating with the sleep() method but keep in mind that the philosopher is eating, not sleeping in the eat() method. See Listing 3.

Now let's put all the classes in a subdirectory, compile, and run them.

Compiling classes on a Unix platform
javac com/ibm/Jinwoo/starvation/DiningPhilosophers.java

Compiling classes on a Windows platform
javac com\ibm\jinwoo\starvation\DiningPhilosophers.java

I used the latest release of IBM Java virtual machine 6.0 (build jvmwi3260-20080816_22093 VM build 20080816_022093_lHdSMr) and another vendor's latest release of a Java virtual machine 6.0 when I wrote this for a fair comparison.

Executing classes
java com.ibm.jinwoo.starvation.DiningPhilosophers
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1

Most of the time Plato will be eating and the other two philosophers will have less of a chance to eat. We can generate thread dumps by sending signal 3 to the process. On the Windows platform, you can use keyboard combinations like Ctrl-Break to send signal 3 to a process running in a command line window. Listing 4 has the results from the IBM Java virtual machine.

We used another vendor's Java virtual machine and redirected output to a file.

Generating thread dumps with another vendor's Java virtual machine
java com.ibm.jinwoo.starvation.DiningPhilosophers 1> sun.log

Now let's talk about thread dump a little more. In most Java platforms, thread dumps are generated when the JVM terminates unexpectedly because of an operating system signal or when the user enters a reserved key combination control-break on Windows, say. We can also generate thread dumps with IBM's JavaDump API: com.ibm.jvm.Dump.JavaDump() if you're running a IBM Java virtual machine.

You can use the kill command on most Unix systems:

  • SIGQUIT[3] : JVM will not terminate

For example, kill -3 $PID

  • SIGSEGV[11] : JVM will terminate

For example, kill -11 $PID

On the IBM WebSphere Application Server 5/6 environment, you can use wsadmin script to generate thread dumps:

wsadmin.sh -conntype SOAP -port <port number>
> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
> $AdminControl invoke $jvm dumpThreads

More Stories By Jinwoo Hwang

Jinwoo Hwang is a software engineer, inventor, author, and technical leader at IBM WebSphere Application Server Technical Support in Research Triangle Park, North Carolina. He joined IBM in 1995 and worked with IBM Global Learning Services, IBM Consulting Services, and software development teams prior to his current position at IBM. He is an IBM Certified Solution Developer and IBM Certified WebSphere Application Server System Administrator as well as a SUN Certified Programmer for the Java platform. He is the architect and creator of the following technologies:

Mr. Hwang is the author of the book C Programming for Novices (ISBN:9788985553643, Yonam Press, 1995) as well as the following webcasts and articles:

Mr. Hwang is the author of the following IBM technical articles:

  • VisualAge Performance Guide,1999
  • CORBA distributed object applet/servlet programming for IBM WebSphere Application Server and VisualAge for Java v2.0E ,1999
  • Java CORBA programming for VisualAge for Java ,1998
  • MVS/CICS application programming for VisualAge Generator ,1998
  • Oracle Native/ODBC application programming for VisualAge Generator ,1998
  • MVS/CICS application Web connection programming for VisualAge Generator ,1998
  • Java applet programming for VisualAge WebRunner ,1998
  • VisualAge for Java/WebRunner Server Works Java Servlet Programming Guide ,1998
  • RMI Java Applet programming for VisualAge for Java ,1998
  • Multimedia Database Java Applet Programming Guide ,1997
  • CICS ECI Java Applet programming guide for VisualAge Generator 3.0 ,1997
  • CICS ECI DB2 Application programming guide for VigualGen, 1997
  • VisualGen CICS ECI programming guide, 1997
  • VisualGen CICS DPL programming guide, 1997

Mr. Hwang holds the following patents in the U.S. / other countries:


Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that Systena America will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Systena Group has been in business for various software development and verification in Japan, US, ASEAN, and China by utilizing the knowledge we gained from all types of device development for various industries including smartphones (Android/iOS), wireless communication, security technology and IoT serv...
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real r...
Five years ago development was seen as a dead-end career, now it’s anything but – with an explosion in mobile and IoT initiatives increasing the demand for skilled engineers. But apart from having a ready supply of great coders, what constitutes true ‘DevOps Royalty’? It’ll be the ability to craft resilient architectures, supportability, security everywhere across the software lifecycle. In his keynote at @DevOpsSummit at 20th Cloud Expo, Jeffrey Scheaffer, GM and SVP, Continuous Delivery Busine...
In order to meet the rapidly changing demands of today’s customers, companies are continually forced to redefine their business strategies in order to meet these needs, stay relevant and continue to see profitable growth. IoT deployment and development is integral in this transformation, and today businesses are increasingly seeing the value of investing their resources into IoT deployments. These technologies are able increase ROI through projects such as connecting supply chains or enabling sm...
SYS-CON Events announced today that CollabNet, a global leader in enterprise software development, release automation and DevOps solutions, will be a Bronze Sponsor of SYS-CON's 20th International Cloud Expo®, taking place from June 6-8, 2017, at the Javits Center in New York City, NY. CollabNet offers a broad range of solutions with the mission of helping modern organizations deliver quality software at speed. The company’s latest innovation, the DevOps Lifecycle Manager (DLM), supports Value S...
SYS-CON Events announced today that Outscale, a global pure play Infrastructure as a Service provider and strategic partner of Dassault Systèmes, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2010, Outscale simplifies infrastructure complexities and boosts the business agility of its customers. Outscale delivers a secure, reliable and industrial strength solution for its customers, which in...
SYS-CON Events announced today that Peak 10, Inc., a national IT infrastructure and cloud services provider, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Peak 10 provides reliable, tailored data center and network services, cloud and managed services. Its solutions are designed to scale and adapt to customers’ changing business needs, enabling them to lower costs, improve performance and focus intern...
Everywhere we turn in our industry we can find strong opinions about the direction, type and nature of cloud’s impact on computing and business. Another word that is used in every context in our industry is “hybrid.” In his session at 20th Cloud Expo, Alvaro Gonzalez, Director of Technical, Partner and Field Marketing at Peak 10, will use a combination of a few conceptual props and some research recently commissioned by Peak 10 to offer a real-world consideration of how the various categories of...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Super Micro Computer, Inc., a global leader in compute, storage and networking technologies, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Supermicro (NASDAQ: SMCI), the leading innovator in high-performance, high-efficiency server technology, is a premier provider of advanced server Building Block Solutions® for Data Center, Cloud Computing, Enterprise IT, Hadoop/...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
Multiple data types are pouring into IoT deployments. Data is coming in small packages as well as enormous files and data streams of many sizes. Widespread use of mobile devices adds to the total. In this power panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists will look at the tools and environments that are being put to use in IoT deployments, as well as the team skills a modern enterprise IT shop needs to keep things running, get a handle on all this data, and deli...
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
In his opening keynote at 20th Cloud Expo, Michael Maximilien, Research Scientist, Architect, and Engineer at IBM, will motivate why realizing the full potential of the cloud and social data requires artificial intelligence. By mixing Cloud Foundry and the rich set of Watson services, IBM's Bluemix is the best cloud operating system for enterprises today, providing rapid development and deployment of applications that can take advantage of the rich catalog of Watson services to help drive insigh...
Existing Big Data solutions are mainly focused on the discovery and analysis of data. The solutions are scalable and highly available but tedious when swapping in and swapping out occurs in disarray and thrashing takes place. The resolution for thrashing through machine learning algorithms and support nomenclature is through simple techniques. Organizations that have been collecting large customer data are increasingly seeing the need to use the data for swapping in and out and thrashing occurs ...
SYS-CON Events announced today that delaPlex will exhibit at SYS-CON's @CloudExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. delaPlex pioneered Software Development as a Service (SDaaS), which provides scalable resources to build, test, and deploy software. It’s a fast and more reliable way to develop a new product or expand your in-house team.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend @CloudExpo | @ThingsExpo, June 6-8, 2017, at the Javits Center in New York City, NY and October 31 - November 2, 2017, Santa Clara Convention Center, CA. Learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
SYS-CON Events announced today that EARP Integration will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. EARP Integration is a passionate software house. Since its inception in 2009 the company successfully delivers smart solutions for cities and factories that start their digital transformation. EARP provides bespoke solutions like, for example, advanced enterprise portals, business intelligence systems an...
SYS-CON Events announced today that Progress, a global leader in application development, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Enterprises today are rapidly adopting the cloud, while continuing to retain business-critical/sensitive data inside the firewall. This is creating two separate data silos – one inside the firewall and the other outside the firewall. Cloud ISVs oft...
As cloud adoption continues to transform business, today's global enterprises are challenged with managing a growing amount of information living outside of the data center. The rapid adoption of IoT and increasingly mobile workforce are exacerbating the problem. Ensuring secure data sharing and efficient backup poses capacity and bandwidth considerations as well as policy and regulatory compliance issues.