Click here to close now.

Welcome!

Java Authors: Pat Romanski, Elizabeth White, Carmen Gonzalez, Yeshim Deniz, Liz McMillan

Related Topics: Oracle, Java, Microservices Journal, Websphere

Oracle: Article

How to Diagnose Java Resource Starvation

Using the IBM Thread & Monitor Dump Analyzer for Java

We can visualize resource starvation using an elaborate rendition of the Dining Philosophers Problem. This classic metaphor of resource allocation among processes was first introduced in 1971 by Edsger Dijkstra in his paper "Hierarchical Ordering of Sequential Processes." It's been a model and universal method for verifying theories on resource allocation ever since.

The metaphor goes like this: There are three well-known philosophers in an Asian bistro. Dinner is served but they are only given three chopsticks because the restaurant's supply truck has been stuck in a snow storm for a couple of days. Naturally each philosopher needs two chopsticks to eat his dinner and each is protected from interference while he uses a chopstick.

Plato skipped lunch that day and insists that he should have priority or else he'll faint. If he doesn't give up his chopsticks, the other philosophers won't be able to eat and eventually they'll starve. If Plato drops his chopsticks and immediately picks them up before the other philosophers can get them then they won't be able to eat and they'll eventually starve.

Dijkstra's Dining Philosophers Problem employed spaghetti and forks but I used chopsticks and Asian cuisine for a more realistic illustration since you can't eat spaghetti with a single chopstick (see Figure 1).

These are examples of resource starvation. You could also encounter deadlock. There are a couple of solutions like semaphores and monitors. But we won't think about solutions or deadlocks in this article. We'll focus on detecting resource starvation using IBM Thread and Monitor Dump Analyzer for Java.

What Is the IBM Thread and Monitor Dump Analyzer for Java?
During the runtime of a Java process, some Java Virtual Machines (JVMs) may not respond predictably and often seem to hang up for a long time or indefinitely (until the JVM shuts down). It's sometimes very challenging to determine the root causes of these sorts of problems.

By triggering Java thread dumps or Javacores when a Java process doesn't respond, it's possible to collect diagnostic information related to the JVM and a Java application captured at a particular point during execution.

On some platforms, a Javacore is known as "javadump." The code that creates a Javacore is part of the JVM. One can control it by using environment variables and runtime switches. By default, a Javacore occurs when the JVM terminates unexpectedly. A Javacore can also be triggered by sending specific signals to the JVM. Although a Javacore or javadump can occur in Solaris JVMs, much of the content of the Javacore is added by IBM and, so, is present only in IBM JVMs.

IBM Thread and Monitor Dump Analyzer for Java analyzes thread dumps/Javacores and diagnoses monitor locks and thread activities to identify the root causes of hangs, deadlocks, and resource contention or monitor bottlenecks. It can analyze IBM, Solaris, and HP-UX Java thread dumps from Java Virtual Machine 1.3.1, 1.4.x, 5.0, and 6.0

More than three years ago, I created IBM Thread and Monitor Dump Analyzer for Java as a garage project and spent thousands of hours during weeknights, weekends, and vacations to provide my clients and colleagues with something that can help diagnose these problems easily, motivated by the same reasons as my other garage projects like IBM HeapAnalyzer, IBM Pattern Modeling and Analysis Tool, and IBM Performance Analyzer for Java for Windows. All the algorithms used have been filed with the U.S. Patent and Trademark Office.

The widgetry was introduced to the public about two years ago through alphaWorks where it's been one of the top downloads for 30 consecutive months. It's recorded 23,296 downloads so far and it's been used by more than 14,600 companies, government agencies, research facilities, and universities worldwide (these statistics are good as of October 2008).

What Is a Thread Dump?
A Java thread dump is the one of the traces/dumps that JVM provides to help diagnosis a hang, deadlock, or monitor contention issue. It contains diagnostic information related to the JVM and a Java application captured at a point during execution. For example, the information can be a list of all the threads that run on a Java virtual machine. Usually thread dumps have more than just thread information. Thread dumps can produce information about the operating system, application environment, threads, stacks, locks, and memory. IBM Thread Dumps/Javacores provide much more information like core interface, data conversion/unicode, class, diagnosis, execution management/thread management, lock, execution engine, mixed mode interpreter, JIT (Just-in-Time compiler), storage/JVM heap, and hardware portability interface/extended hardware portability interface.

The contents and formats of Java thread dumps depend on the platform you're running on and the JVM provider.

Since a thread dump's format isn't a part of the SDK specification, each SDK vendor provides a unique thread dump format and its own JVM information. Currently IBM Thread and Monitor Dump Analyzer for Java understands IBM, Solaris, and HP-UX Java thread dump formats.

By default, IBM JVM thread dumps occur when the JVM terminates unexpectedly. A thread dump can also be triggered by sending specific signals to the JVM. Thread dumps are usually human readable text. In an IBM JVM, the thread dump is often called a Javadump or Javacore. Javacore is NOT the same as a core file, which is generated by a system dump.

Before IBM Thread and Monitor Dump Analyzer for Java was introduced, people had to manually check hundreds of threads in the thread dumps using their favorite editors. With IBM Thread and Monitor Dump Analyzer for Java, we can easily diagnose a variety of issues without reading through hundreds of thread stack traces.

Usually thread dumps aren't as large as heap dumps. Usually its size is around 1MB or less unless you have large number of threads or classes. It's not a big deal to manually review thread dumps if you only have a handful of threads running on your JVM. If you have hundreds of threads with dozens of Java monitors, it gets complicated.

How Does It Work?
This technology analyzes the information in each thread and provides diagnostic information, such as current thread information, the signal that caused the Javacore/Java heap information (maximum Java heap size, initial Java heap size, garbage collector counter, allocation failure counter, free Java heap size, and allocated Java heap size), number of runnable threads, total number of threads, number of monitors locked, native memory usage of each JVM component, and deadlock information.

IBM Thread and Monitor Dump Analyzer for Java also provides you with the recommended size of the Java heap cluster (applicable only to IBM SDK 1.4.2 and 1.3.1 SR7 or above) based on a heuristic analysis engine.

It compares each Javacore and provides process ID information for threads, a time stamp of the first Javacore, a time stamp of the last Javacore, the number of garbage collections a minute, the number of allocation failures a minute, the time between the first Javacore and the last Javacore, the number of hang suspects, and a list of hang suspects.

This technology also compares all monitor information in a Javacore and detects deadlock and resource contention or monitor bottlenecks, if there are any.

Let's Get Started
We can simulate a starvation problem with a simple Java application.

In DiningPhilosophers.java, we created three chopsticks, c1, c2, and c3 with the names, chopstick 1, chopstick 2, and chopstick3. Then we created three philosophers, p1, p2, and p3 with the names, Socrates, Plato, and Aristotle. You might have noticed that each philosopher's name has a suffix that represents a priority since we want to give Plato more of a chance to eat. Besides a name and priority, each philosopher is also given a left chopstick and a right chopstick. We start with each philosopher, p1, p2,  and p3 as shown in Listing 1. (Download Listings 1-6 here.)

In Chopstick.java, we just create a constructor, a getter, and a setter for the name of chopstick as shown in Listing 2

In Philosopher.java, we create a constructor, getters and setters for right and left chopsticks, and the philosopher's name. We also override the run() method in which we acquire Java's built-in monitors for left and right chopsticks and call the eat() method. In each () method, we pause for two seconds to simulate eating with the sleep() method but keep in mind that the philosopher is eating, not sleeping in the eat() method. See Listing 3.

Now let's put all the classes in a subdirectory, compile, and run them.

Compiling classes on a Unix platform
javac com/ibm/Jinwoo/starvation/DiningPhilosophers.java

Compiling classes on a Windows platform
javac com\ibm\jinwoo\starvation\DiningPhilosophers.java

I used the latest release of IBM Java virtual machine 6.0 (build jvmwi3260-20080816_22093 VM build 20080816_022093_lHdSMr) and another vendor's latest release of a Java virtual machine 6.0 when I wrote this for a fair comparison.

Executing classes
java com.ibm.jinwoo.starvation.DiningPhilosophers
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1
Plato priority9 is eating with chopstick 3 and chopstick 1

Most of the time Plato will be eating and the other two philosophers will have less of a chance to eat. We can generate thread dumps by sending signal 3 to the process. On the Windows platform, you can use keyboard combinations like Ctrl-Break to send signal 3 to a process running in a command line window. Listing 4 has the results from the IBM Java virtual machine.

We used another vendor's Java virtual machine and redirected output to a file.

Generating thread dumps with another vendor's Java virtual machine
java com.ibm.jinwoo.starvation.DiningPhilosophers 1> sun.log

Now let's talk about thread dump a little more. In most Java platforms, thread dumps are generated when the JVM terminates unexpectedly because of an operating system signal or when the user enters a reserved key combination control-break on Windows, say. We can also generate thread dumps with IBM's JavaDump API: com.ibm.jvm.Dump.JavaDump() if you're running a IBM Java virtual machine.

You can use the kill command on most Unix systems:

  • SIGQUIT[3] : JVM will not terminate

For example, kill -3 $PID

  • SIGSEGV[11] : JVM will terminate

For example, kill -11 $PID

On the IBM WebSphere Application Server 5/6 environment, you can use wsadmin script to generate thread dumps:

wsadmin.sh -conntype SOAP -port <port number>
> set jvm [$AdminControl completeObjectName type=JVM,process=server1,*]
> $AdminControl invoke $jvm dumpThreads

More Stories By Jinwoo Hwang

Jinwoo Hwang is a software engineer, inventor, author, and technical leader at IBM WebSphere Application Server Technical Support in Research Triangle Park, North Carolina. He joined IBM in 1995 and worked with IBM Global Learning Services, IBM Consulting Services, and software development teams prior to his current position at IBM. He is an IBM Certified Solution Developer and IBM Certified WebSphere Application Server System Administrator as well as a SUN Certified Programmer for the Java platform. He is the architect and creator of the following technologies:

Mr. Hwang is the author of the book C Programming for Novices (ISBN:9788985553643, Yonam Press, 1995) as well as the following webcasts and articles:

Mr. Hwang is the author of the following IBM technical articles:

  • VisualAge Performance Guide,1999
  • CORBA distributed object applet/servlet programming for IBM WebSphere Application Server and VisualAge for Java v2.0E ,1999
  • Java CORBA programming for VisualAge for Java ,1998
  • MVS/CICS application programming for VisualAge Generator ,1998
  • Oracle Native/ODBC application programming for VisualAge Generator ,1998
  • MVS/CICS application Web connection programming for VisualAge Generator ,1998
  • Java applet programming for VisualAge WebRunner ,1998
  • VisualAge for Java/WebRunner Server Works Java Servlet Programming Guide ,1998
  • RMI Java Applet programming for VisualAge for Java ,1998
  • Multimedia Database Java Applet Programming Guide ,1997
  • CICS ECI Java Applet programming guide for VisualAge Generator 3.0 ,1997
  • CICS ECI DB2 Application programming guide for VigualGen, 1997
  • VisualGen CICS ECI programming guide, 1997
  • VisualGen CICS DPL programming guide, 1997

Mr. Hwang holds the following patents in the U.S. / other countries:


Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
SYS-CON Events announced today that ProfitBricks, the provider of painless cloud infrastructure, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY., and the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. ProfitBricks is the IaaS provider that offers a painless cloud experience for all IT users, with no learning curve. ProfitBricks boasts flexible cloud servers and networking, an integrated Data Center Designer tool f...
SYS-CON Events announced today that Dyn, the worldwide leader in Internet Performance, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Dyn is a cloud-based Internet Performance company. Dyn helps companies monitor, control, and optimize online infrastructure for an exceptional end-user experience. Through a world-class network and unrivaled, objective intelligence into Internet conditions, Dyn ensures traffic gets delivered faster, safer, and more reliably than ever.
SYS-CON Events announced today that Open Data Centers (ODC), a carrier-neutral colocation provider, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. Open Data Centers is a carrier-neutral data center operator in New Jersey and New York City offering alternative connectivity options for carriers, service providers and enterprise customers.
SYS-CON Events announced today that On the Avenue Marketing Group, a sales and marketing firm that utilizes events to market and sell products to consumers, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. On the Avenue Marketing Group (OTA) is a sales and marketing firm that utilizes events to market and sell products to consumers. On behalf of our clients, we attend thousands of fairs, festivals, expos, concerts, conferences, and sporting events annually, helping them reach millions of individuals ...
SYS-CON Events announced today that ActiveState, the leading independent Cloud Foundry and Docker-based PaaS provider, has been named “Silver Sponsor” of SYS-CON's DevOps Summit New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. ActiveState believes that enterprises gain a competitive advantage when they are able to quickly create, deploy and efficiently manage software solutions that immediately create business value, but they face many challenges that prevent them from doing so. The Company is uniquely positioned to help address these challenges thro...
BroadSoft on Tuesday announced that it is a recipient of the 2014 Frost & Sullivan Market Leadership Award in the Hosted/Cloud Internet Protocol (IP) Telephony market for Latin America. According to Frost & Sullivan market research, the Latin America (LATAM) hosted/cloud Internet Protocol (IP) telephony market, including integrated unified communications and collaboration (UC&C) applications, is currently experiencing a rapid growth trajectory and is expected to exhibit a tenfold rise in annual revenues in the 2013-2020 period. With more than 600 cloud deployments internationally, BroadSoft w...
SYS-CON Events announced today that Vitria Technology, Inc. will exhibit at SYS-CON’s @ThingsExpo, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Vitria will showcase the company’s new IoT Analytics Platform through live demonstrations at booth #330. Vitria’s IoT Analytics Platform, fully integrated and powered by an operational intelligence engine, enables customers to rapidly build and operationalize advanced analytics to deliver timely business outcomes for use cases across the industrial, enterprise, and consumer segments.
SYS-CON Events announced today that Alert Logic, the leading provider of Security-as-a-Service solutions for the cloud, has been named “Bronze Sponsor” of SYS-CON's 16th International Cloud Expo® and DevOps Summit 2015 New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY, and the 17th International Cloud Expo® and DevOps Summit 2015 Silicon Valley, which will take place November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Akana, formerly SOA Software, has been named “Bronze Sponsor” of SYS-CON's 16th International Cloud Expo® New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. Akana’s comprehensive suite of API Management, API Security, Integrated SOA Governance, and Cloud Integration solutions helps businesses accelerate digital transformation by securely extending their reach across multiple channels – mobile, cloud and Internet of Things. Akana enables enterprises to share data as APIs, connect and integrate applications, drive part...
SYS-CON Events announced today that CommVault has been named “Bronze Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY, and the 17th International Cloud Expo®, which will take place on November 3–5, 2015, at the Santa Clara Convention Center in Santa Clara, CA. A singular vision – a belief in a better way to address current and future data management needs – guides CommVault in the development of Singular Information Management® solutions for high-performance data protection, universal availability and sim...
SYS-CON Events announced today that SafeLogic has been named “Bag Sponsor” of SYS-CON's 16th International Cloud Expo® New York, which will take place June 9-11, 2015, at the Javits Center in New York City, NY. SafeLogic provides security products for applications in mobile and server/appliance environments. SafeLogic’s flagship product CryptoComply is a FIPS 140-2 validated cryptographic engine designed to secure data on servers, workstations, appliances, mobile devices, and in the Cloud.
SYS-CON Events announced today that StorPool Storage will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. StorPool is distributed storage software that allows service providers, enterprises and other cloud builders to run data storage on standard x86 servers, instead of using expensive and inefficient storage arrays (SAN).
GENBAND has announced that SageNet is leveraging the Nuvia platform to deliver Unified Communications as a Service (UCaaS) to its large base of retail and enterprise customers. Nuvia’s cloud-based solution provides SageNet’s customers with a full suite of business communications and collaboration tools. Two large national SageNet retail customers have recently signed up to deploy the Nuvia platform and the company will continue to sell the service to new and existing customers. Nuvia’s capabilities include HD voice, video, multimedia messaging, mobility, conferencing, Web collaboration, deskt...
SYS-CON Events announced today that Site24x7, the cloud infrastructure monitoring service, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Site24x7 is a cloud infrastructure monitoring service that helps monitor the uptime and performance of websites, online applications, servers, mobile websites and custom APIs. The monitoring is done from 50+ locations across the world and from various wireless carriers, thus providing a global perspective of the end-user experience. Site24x7 supports monitoring H...
Sonus Networks introduced the Sonus WebRTC Services Solution, a virtualized Web Real-Time Communications (WebRTC) offer, purpose-built for the Cloud. The WebRTC Services Solution provides signaling from WebRTC-to-WebRTC applications and interworking from WebRTC-to-Session Initiation Protocol (SIP), delivering advanced real-time communications capabilities on mobile applications and on websites, which are accessible via a browser.
SYS-CON Events announced today that Intelligent Systems Services will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Established in 1994, Intelligent Systems Services Inc. is located near Washington, DC, with representatives and partners nationwide. ISS’s well-established track record is based on the continuous pursuit of excellence in designing, implementing and supporting nationwide clients’ mission-critical systems. ISS has completed many successful projects in Healthcare, Commercial, Manufacturing, ...
SYS-CON Events announced today that B2Cloud, a provider of enterprise resource planning software, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. B2cloud develops the software you need. They have the ideal tools to help you work with your clients. B2Cloud’s main solutions include AGIS – ERP, CLOHC, AGIS – Invoice, and IZUM
SYS-CON Events announced today that Tufin, the market-leading provider of Security Policy Orchestration Solutions, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. As the market leader of Security Policy Orchestration, Tufin automates and accelerates network configuration changes while maintaining security and compliance. Tufin's award-winning Orchestration Suite™ gives IT organizations the power and agility to enforce security policy across complex, multi-vendor enterprise networks. With more than 1...
SYS-CON Events announced today that Cloudian, Inc., the leading provider of hybrid cloud storage solutions, will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. Cloudian, Inc., is a Foster City, California - based software company specializing in cloud storage software. The main product is Cloudian, an Amazon S3-compliant cloud object storage platform, the bedrock of cloud computing systems, that enables cloud service providers and enterprises to build reliable, affordable and scalable cloud storage solu...
Temasys has announced senior management additions to its team. Joining are David Holloway as Vice President of Commercial and Nadine Yap as Vice President of Product. Over the past 12 months Temasys has doubled in size as it adds new customers and expands the development of its Skylink platform. Skylink leads the charge to move WebRTC, traditionally seen as a desktop, browser based technology, to become a ubiquitous web communications technology on web and mobile, as well as Internet of Things compatible devices.