Welcome!

Java IoT Authors: Stackify Blog, APM Blog, Liz McMillan, Elizabeth White, Yeshim Deniz

Related Topics: Java IoT, Weblogic

Java IoT: Article

Java Thread Dumps

Analyzing Java Thread Dumps

Software maintenance is a tedious and challenging job. As long as the software functions as expected, it looks great. Imagine the situation, your pager keeps buzzing in the midnight (Not a happy feeling, right?).

Any software system, no matter how well it has been built and quality tested might get into run-time performance issues. The reasons can be within the functional boundary or can be from the external environment. Software systems are built on certain assumptions and pre-conceived notions. However, when they go live, the assumptions may turn out to be false causing the system to malfunction.

In enterprise J2EE systems which generally have very large user base and involve interactions between heterogeneous systems, one of the general runtime issues reported is the system slowdown or system "hang". In such situations, the general trouble shooting pattern would be to analyze the Java thread dumps to isolate the threads which are causing the slow-down or hang. This article discusses the Java stack traces; anatomy of Java Threads and how to read thread dumps in general.

Exceptions and Stack Traces
All of us encounter and have encountered exceptions during our learning/development phases. Exception is the way Java reports a runtime error. Exceptions have two parts. Message and the stack trace. Message tells you what has gone wrong. Stack Traces provide a complete call flow through all the classes involved top-down as part of the runtime error.

The following is an example stack trace for an ArrayIndexOutOfBoundsException

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4

at Test.run(Test.java:13)

at Test.<init>(Test.java:5)

at Test.main(Test.java:20)

In the above exception, the first line "Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 4" tells you that the JVM encountered the reported exception when it tried to access the value at array index 4. The java thread which encountered the exception is "main" thread.

Next let's read through the stack trace. The rule of thumb for exceptions is to read from the top line (message line) to understand what the exception is and keep going down to understand the call flow. In the above example, the call flow started at Test.java line number 20 (main() method) and then it called the constructor of the Test class. Constructors are represented with <init> in the stack traces. Then it moved on to the run() method in the Test class and encountered the reported exception at line number 13.

From the above stack trace, we can conclude that in Test.java, an attempt is made to read past the size of the Array.

The Java Thread Dump
Java Thread dump can be defined as a snapshot of all threads running inside a Java Virtual Machine (JVM) at a given point of time. A thread dump might contain a single thread or multiple threads. In multi threaded environments like J2EE application servers, there will be many threads and Thread Groups. Each of these threads will have its own call stack and will be performing a separate function at a given time. Thread Dump will provide the stack traces of all the JVM threads and much more information about a particular thread.

The Java VM Process and Java Threads
The Java Virtual Machine or JVM is an operating system level process. Java Threads are child processes or Light-Weight Processes (in Solaris terms) of the JVM process.

Generating Java Thread Dumps
Thread Dump is generated by sending a SIGQUIT signal to the JVM process. There are different ways of sending this signal to the process.

In Unix, use "kill -3 <pid>" where pid is the Process ID of the JVM.

In Windows, press CTRL+BREAK on the window where the JVM is running

Java Thread State
Each Java Thread can be in one of the four states during its lifetime.

Runnable - The thread is either running or ready to run when it gets its CPU turn. JRockit thread dumps refer to this state as ACTIVE.

Waiting on Monitor - The thread is either sleeping or waiting on an object for said period of time or waiting to be notified by another thread. This will happen when any of the sleep() methods on Thread object or wait() method on an Object is called.

For example, in WebLogc server the idle execute threads are in this condition and they wait till a socket reader thread notify them of some new work to be done. The stack trace will look like this:

"ExecuteThread: '2' for queue: 'weblogic.admin.RMI'" daemon prio=5 tid=0x1752F040 nid=0x180c in Object.wait() [1887f000..1887fd8c]

at java.lang.Object.wait(Native Method) waiting on <04134D98> (a weblogic.kernel.ExecuteThread)

at java.lang.Object.wait(Object.java:426)

at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:126)

locked <04134D98> (a weblogic.kernel.ExecuteThread)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:145)

Some other versions of the JVM call this state as CW, Object.wait() (as above). JRockit refer to this as state as WAITING.

Waiting for Monitor Entry - The thread is waiting to get the lock for an object (some other thread may be holding the lock). This happens if two or more threads try to execute synchronized code. Note that the lock is always for an object and not for individual methods.

Sample stack trace of a thread in this condition:

"ExecuteThread: '24' for queue: 'DisplayExecuteQueue'" daemon prio=5 tid=0x5541b0 nid=0x3b waiting for monitor entry [49b7f000..49b7fc24]

at weblogic.cluster.replication.ReplicationManager.createSecondary (ReplicationManager.java:908)
- waiting to lock <6c4b9130> (a java.lang.Object)
at weblogic.cluster.replication.ReplicationManager.updateSecondary (ReplicationManager.java:715)
at weblogic.servlet.internal.session.ReplicatedSessionData.syncSession (ReplicatedSessonData.java:459)
- locked <6c408700> (a weblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.session.ReplicatedSessionContext.sync (ReplicatedSessionContext.java:134)
- locked <6c408700> (aweblogic.servlet.internal.session.ReplicatedSessionData)
at weblogic.servlet.internal.ServletRequestImpl.syncSession (ServletRequestImpl.java:2418)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:3137)
at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2544)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:153)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:134)

In the above stacktrace, you can see that this thread holds the lock an object (6c408700) and waiting to lock another object (6c4b9130)

Some other JVMs may not give the object Ids with lock information in the stack trace. The same state may also be called as ‘MW'. JRockit refers to this state as LOCKED.

Anatomy of a Java Thread
To be able to read/analyze thread dump, it is important to understand various parts of a Thread dump. Let's take a simple Thread Stack example and read various parts of it.

"ExecuteThread: '1' " daemon prio=5 tid=0x628330 nid=0xf runnable [0xe4881000..0xe48819e0]

at com.vantive.vanjavi.VanJavi.VanCreateForm(Native Method)

at com.vantive.vanjavi.VanMain.open(VanMain.java:53)

at jsp_servlet._so.__newServiceOrder.printSOSection( __newServiceOrder.java:3547)

at jsp_servlet._so.__newServiceOrder._jspService (__newServiceOrder.java:5652)

at weblogic.servlet.jsp.JspBase.service(JspBase.java:27)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:265)

at weblogic.servlet.internal.ServletStubImpl.invokeServlet (ServletStubImpl.java:200)

at weblogic.servlet.internal.WebAppServletContext.invokeServlet (WebAppServletContext.java:2495)

at weblogic.servlet.internal.ServletRequestImpl.execute (ServletRequestImpl.java:2204)

at weblogic.kernel.ExecuteThread.execute (ExecuteThread.java:139)

at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:120)

In the above Thread Dump, the interesting part to is the first line. The rest of the stuff is nothing more than a general stack trace. Lets analyze the first line here

Execute Thread : 1 This indicates the name of the thread.

daemon Indicates that this Thread is a daemon thread

prio=5 The priority of this thread (default is 5)

tid The Java Thread Id (Unique identifier for this thread in the running JVM instance).

nid Native Identifier of the thread. Indicates LWP id in Solaris. Identifier for this process

at the OS level.

runnable Indicate the Thread state (See above)

[x..y] Indicates the range of addresses in the heap where this thread is executing.

The remaining part of the thread dump show the call flow. In this example, the thread (Execute Thread 1) is a OS daemon which is currently executing a native method vanCreateForm().

Putting Thread Dumps to Use
In this section I describe few use cases in which Thread dumps are very useful.

High CPU consumption
Diagnosis

The application seems to consume almost 100% CPU and the throughput has come down drastically. Starts to perform very poorly with high load on the CPU.

Thread Dump
All threads in the dump generally show one or more threads struck doing the same operation through all the thread dumps.

Solution

  • Take a series of thread dumps (around 5-7) for one particular call flow (a web form submit for example) before the call is completed.
  • Looking for "runnable" threads in the Thread Dump. If each such thread seems to move (the method call in each thread dump is different), then the thread is progressing and probably not a culprit. If the thread is executing the same method (same line number) through all the thread dumps, then it is almost for sure the culprit. Go to the code and do the code level analysis. You are almost certain to catch an issue there.

Low CPU consumption and Poor Response time
Diagnosis

This happens in a high I/O bound systems under high load. The CPU consumption is low with only few threads consuming reasonable CPU chunk. However the application response time will be very slow.

Thread Dump
Some or all runnable threads seem to struck performing an I/O operation like a File read/write or a database operation.

Solution
Understand the I/O operation profile of your application. Use caches where applicable to reduce DB interactions.

Application/Server Hang
Diagnosis

An application or an server JVM hosting the application will hang (become unresponsive).

Thread Dump

  • All runnable threads are struck doing the same operation through all the thread dumps taken. The server runs out of any more threads as all the runnable threads "never" complete their operation.
  • There might be many threads waiting for monitor entry. This can happen if a "runnable" thread is holding a lock on an object and never returns it while others are waiting for the same lock.

Solution

  • Check for dead lock. JVM generally detects a deadlock if it is simple enough (Thread A waiting for Thread B and vice versa). However, you need to understand the lock profile at the given moment to see if there is any complex deadlock situation involved.
  • Revisit the synchronized methods/block in the code. Reduce the size of synchronization area to the extent possible.
  • One issue could be too long a timeout while accessing a remote resource/component. Implement a reasonable timeout on the remote object clients so that the throw appropriate exception when a remote system does not respond in a reasonable amount of time.
  • If all the threads are waiting for a resource (like an EJB/DB connection), consider increasing the object pool size for those resources.

Tools
There are both commercial and open source tools available for thread dump analysis. One such tool is Samurai. It is a light weight open source tool which runs as a Java Web Start application as well as from your command prompt. For more information and documentation on Samurai, visit

http://yusuke.homeip.net/samurai/en/index.html

Conclusion
Maintaining J2EE enterprise application in a production environment is a tough job. As the business dynamics change, J2EE application environments change which might cause runtime instability in the production application. One of the primary factors that effect an running J2EE application is high load. While most systems are designed to be scalable, environmental limitations might cause them to become non-responsive.

Java Thread Dumps is an excellent mechanism to identify, diagnose, detect and resolve typical production issues. While application profiling and other mechanisms do exist, analyzing Java Thread dumps will provide us an clear and early understanding of popular production level issues there by saving time and helping us provide better user experience with the production applications.

More Stories By Shankar Itchapurapu

Shankar Itchapurapu is a software engineer at Oracle in India. He holds a Master's degree in Computer Applications. You can e-mail Shankar at [email protected]

@ThingsExpo Stories
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
In his session at 21st Cloud Expo, Carl J. Levine, Senior Technical Evangelist for NS1, will objectively discuss how DNS is used to solve Digital Transformation challenges in large SaaS applications, CDNs, AdTech platforms, and other demanding use cases. Carl J. Levine is the Senior Technical Evangelist for NS1. A veteran of the Internet Infrastructure space, he has over a decade of experience with startups, networking protocols and Internet infrastructure, combined with the unique ability to it...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
Gemini is Yahoo’s native and search advertising platform. To ensure the quality of a complex distributed system that spans multiple products and components and across various desktop websites and mobile app and web experiences – both Yahoo owned and operated and third-party syndication (supply), with complex interaction with more than a billion users and numerous advertisers globally (demand) – it becomes imperative to automate a set of end-to-end tests 24x7 to detect bugs and regression. In th...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Large industrial manufacturing organizations are adopting the agile principles of cloud software companies. The industrial manufacturing development process has not scaled over time. Now that design CAD teams are geographically distributed, centralizing their work is key. With large multi-gigabyte projects, outdated tools have stifled industrial team agility, time-to-market milestones, and impacted P&L stakeholders.
"Space Monkey by Vivent Smart Home is a product that is a distributed cloud-based edge storage network. Vivent Smart Home, our parent company, is a smart home provider that places a lot of hard drives across homes in North America," explained JT Olds, Director of Engineering, and Brandon Crowfeather, Product Manager, at Vivint Smart Home, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
"Akvelon is a software development company and we also provide consultancy services to folks who are looking to scale or accelerate their engineering roadmaps," explained Jeremiah Mothersell, Marketing Manager at Akvelon, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
"There's plenty of bandwidth out there but it's never in the right place. So what Cedexis does is uses data to work out the best pathways to get data from the origin to the person who wants to get it," explained Simon Jones, Evangelist and Head of Marketing at Cedexis, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
SYS-CON Events announced today that Telecom Reseller has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
An increasing number of companies are creating products that combine data with analytical capabilities. Running interactive queries on Big Data requires complex architectures to store and query data effectively, typically involving data streams, an choosing efficient file format/database and multiple independent systems that are tied together through custom-engineered pipelines. In his session at @BigDataExpo at @ThingsExpo, Tomer Levi, a senior software engineer at Intel’s Advanced Analytics gr...