Welcome!

Java IoT Authors: Elizabeth White, Pat Romanski, Akhil Sahai, JP Morgenthal, Janakiram MSV

Related Topics: Java IoT, IBM Cloud, Weblogic

Java IoT: Article

Java Developer's Journal Feature: "Deadlocks in J2EE"

Most non-trivial applications involve high degrees of concurrency and many layers of abstraction

In the example, the updateBatchStatus method makes a "RequiresNew" EJB call to actually update the batch_status database table so the status change is immediately visible, even though the effects of the current transaction aren't yet visible. The call to executeUpdate isn't an EJB call, so it executes in the same transaction as the rest of bulkLoadData.

As described, this code will cause a deadlock even in the absence of concurrency. When bulkLoadData calls the executeUpdate method, it updates an existing database row, which involves acquiring a write-lock on that row. The nested EJB call to updateBatchStatus will execute on a separate database connection and try to execute a very similar query, but it will block because it can't acquire the necessary write-lock. From the database's perspective, this second connection will be allowed to proceed as soon as the first connection's transaction is committed or rolled back, which is why the database doesn't detect this as a deadlock. The VM, however, won't allow the call to bulkLoadData to complete before each call to updateBatchStatus completes, so we have a deadlock.

This example shows one update blocking another update, so it causes a deadlock on any database. If the initial update query was a simple select query instead, this example would only cause deadlocks on databases that use lock-based concurrency control, where one connection's read-lock can block another connection from acquiring a write-lock. In any case, a deadlock of this type is neither timing-dependent nor load-dependent, and the thread dump will show one Java thread waiting for a database response, but that thread will be associated with two active database connections. And one of those database connections will be idle, but blocking the other connection.

This scenario has many subtle variations, where more than one thread and more than two database connections may be involved. For example, the outer EJB call's database connection may have acquired a database lock that blocks another unrelated database connection from proceeding, but that unrelated database connection has acquired a lock that blocks the nested EJB call's database operation. This particular case is timing-dependent, and will show several Java threads waiting for database responses. At least one of those Java threads will be associated with two active database connections.

Cross-Resource Deadlock #3: JVM Locks Crossed with Database Locks
A third deadlock scenario occurs when mixing database locks with JVM locks. In this case, one thread holds a database lock and is trying to acquire a JVM lock (trying to enter a synchronized block), and another thread holds that JVM lock and is trying to acquire a database lock. Again, the database sees one connection blocking another, but nothing is stopping that connection from proceeding, so it won't detect a deadlock. The JVM sees one thread inside a synchronized block and another trying to enter, so even if the JVM could detect deadlocks and handle them somehow, it wouldn't detect this case.

For an example that illustrates this deadlock scenario, consider a simple (flawed) read-through cache. This cache is a HashMap backed by a database table. If a cache hit occurs, it will simply return the value from the HashMap; on a cache miss however, it will read the value from the database, add it to the HashMap, and return it as shown in Listing 2.

It's a simple read-through cache. Note that the get() method is synchronized, since we access a non-thread safe container, and we want the containsKey/put combination to be atomic in the case of a cache miss.

This cache is fairly straightforward: the contract is that if we change the data in the table backing the cache, we should call clearCache(), so the cache can avoid handing back stale data. The resulting cache misses will repopulate the cache appropriately.

Let's now consider some code that changes this data and clears the cache:

public void updateData(String key, String value) {
    executeUpdate("update cache_table set value='" + value +
       "' where key='" + key + "'");
    SimpleCache.getInstance().clearCache();
}

In a simple case, this will work with no problems. However, on a database that uses lock-based concurrency control, the query in updateData will prevent the select query in queryForValue from executing, because the update statement will have acquired a write-lock that prevents the select query from acquiring a read-lock on the same row. If the timing is just right, one thread can try to read a given value out of the cache and get a cache miss while another thread updates that value in the database. If the database executes the update statement first, it will block the select statement from continuing. However, the thread executing the select statement came through the synchronized get method, so it has acquired the lock on the SimpleCache. For the thread in updateData to return, it has to call clearCache(), but it can't acquire the lock (clearCache() is synchronized).

When dealing with an instance of this scenario, there will be one Java thread waiting for a response from the database, and one waiting to acquire a JVM lock. Each thread will be associated with a database connection, with one connection blocking the other. The fix is to avoid doing database operations while holding JVM locks: the get() method of our SimpleCache could be rewritten like this:

public Object get(String key) {
    synchronized(this) {
       if (cache.containsKey(key)) {
          return cache.get(key);
       }
    }
    Object value = queryForValue(key);
    synchronized(this) {
       cache.put(key, value);
    }
    return value;
}

Since we now know that this deadlock case can happen, we can add a check to our queryForValue method to try to avoid the deadlock condition by using Thread.holdsLock():

private Object queryForValue(String key) {
      assert(!Thread.holdsLock(this));
      return executeQuery(...);
}

While Thread.holdsLock() can be useful in this case, it only works if we know which locks we're specifically worried about. It would be useful to have a similar method that could determine whether the current thread holds any JVM locks. Then, any piece of code that made any kind of RPC call, database access, etc., could throw an exception or log a warning to indicate that these operations can be dangerous while holding a JVM lock.

Note: even though we've fixed the deadlock problem in this example, it's still flawed because the cache is cleared before updateData's transaction has been committed. If a cache miss happens after the clearCache call but before updateData's transaction is committed, the cache will load the old data because the new data isn't visible yet. The fix here is to clear the cache only after the changes have been committed. Observe that this would only happen in an MVCC database; in a lock-based database, the pending update would block the cache's read, so the cache would read the correct value after the update's transaction was committed.

Rules of Thumb
These guidelines should help you avoid these problems, or at least diagnose and fix them if they do happen.

  • Keep your transactions short and simple.
  • Understand the locking behavior of your database (and your transaction isolation level).
  • Assume any database access has a chance of failing with a database deadlock, and retry correctly.
  • Don't update any non-transactional state (in-memory state, caches, etc.) until after the transaction has completed.
  • Make sure resource pools are large enough for your peak concurrency.
  • Try not to acquire multiple resources at the same time. If you must, acquire them in the same order each time.
  • Learn how to get a full thread dump from your application server and a list of database connections from your database (including which connections are blocked by each), and how to know which Java thread each active database connection is associated with. The easiest way to understand the mapping between Java threads and database connections is to add some logging to your connection pool access code.
  • When making nested EJB calls, understand which ones will use the same database connection as the caller, and which ones will use new connections. Even if the nested call runs in the same global transaction, it may still use a different database connection, and this can cause cross-resource deadlocks.
  • Avoid making database calls and EJB calls, or doing other off-JVM operations while holding JVM locks. Use assert(!Thread.holdsLock(...)) if there are specific JVM locks you're concerned about to prevent future code changes from violating this rule unintentionally.
Conclusion
Cross-resource deadlocks in a J2EE application can be a big problem - they can cause the entire application to grind to a halt. They can also be difficult to isolate and fix, especially if developers aren't familiar with how to analyze a deadlocked environment. The scenarios we've discussed should help you understand a few common deadlocks, and give you some idea of what to look for. Even better, the rules of thumb we've outlined should give you a few conventions to follow in your code to avoid problems like this altogether.

Resources

More Stories By Michael Nonemacher

Michael Nonemacher is a lead software engineer for Lombardi Software. He has worked with Java since 1997, focusing on server-side database interaction and concurrency in Web-based enterprise applications.

Comments (3) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
debnathm 11/18/08 11:48:20 AM EST

You mention:
"It would be nice if the EJB vendors supplied a retry-on-deadlock parameter that would do this automatically with container-managed transactions. Without this automated feature, developers end up forcing EJB calls to use bean-managed transactions just to be able to retry on deadlocks. (One of the disadvantages to making your EJB calls use bean-managed transactions is that it's not obvious how to get the same semantics as a container-managed transaction with "RequiresNew" ...)"

Is BMT strictly required to retry? Suppose all business methods pass through an initial method with no transaction attributes. This method front_bizmethod() calls the actual business method which of course
has a transaction attribute of Required/RequiresNew etc.

public void front_bizmethod()
{
int retry_count = 0;
boolean retry = false;
do
{
try
{
bizmethod();
}
catch (RetryException re)
{
retry = true;
retry_count++;
}
} while (retry && retry_count < 10);
}

/** Business method -- has transaction attribute
* Required or RequiresNew
*/
bizmethod() throws RetryException
{
// Do business calls and throw a RetryException
// if a deadlock was detected by the database
}

What I mean is that there is a way around it by
coding an extra method.

What are your thoughts on the same ?

jnorris10 07/08/06 11:10:54 PM EDT

Thanks for this great article, Mike. I wasn't able to make it to JavaOne this year so I missed your BOF at JavaOne that looks like it was talking about this stuff (BOF-0534). Are there any slides from that that I can get ahold of?

With the power and simplicity that CMT (container managed transactions) brings, is there really no simple way to automatically handle database deadlock by retrying the transaction according to some given parameters (ie: number of retries, back off time, etc)? That would really be unfortunate.

Here are a few possible solutions, but they all seem sub-optimal for various reasons: (I would like to stay within the spec (EJB3/JEE5), but I'm not adament on this).

1) JBoss has a proprietary extension for this: (org.jboss.ejb.plugins.TxRetryExceptionHandler) However, besides being outside the spec, this implementation does not allow me to easily keep track of any state regarding the retry (ie: retry count numbers, data for back off algorithms, etc.).

2) I can use the EJB3 Interceptor spec, but this is really cumbersome because it ties in *after* the CMT stuff has already been setup.

3) I'm sure I could write a jboss-aop interceptor that would do the right thing here, but again it would be outside the spec. Has anyone done this already? I certainly don't want to reinvent the wheel on this one.

4) I could switch from CMT (container managed transactions) to BMT (bean managed transactions). This would give me more control over the the transaction endpoints but then I would have to give up all the niceties that CMT give me.

I really would like a nice solution for this...

SYS-CON Australia News Desk 04/11/06 10:14:26 AM EDT

Most non-trivial applications involve high degrees of concurrency and many layers of abstraction. Concurrency is associated with resource contention and an increase in deadlock conditions. The multiple layers of abstraction make it more difficult to isolate and fix the deadlock conditions.

@ThingsExpo Stories
The cloud market growth today is largely in public clouds. While there is a lot of spend in IT departments in virtualization, these aren’t yet translating into a true “cloud” experience within the enterprise. What is stopping the growth of the “private cloud” market? In his general session at 18th Cloud Expo, Nara Rajagopalan, CEO of Accelerite, explored the challenges in deploying, managing, and getting adoption for a private cloud within an enterprise. What are the key differences between wh...
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
SYS-CON Events announced today that Venafi, the Immune System for the Internet™ and the leading provider of Next Generation Trust Protection, will exhibit at @DevOpsSummit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Venafi is the Immune System for the Internet™ that protects the foundation of all cybersecurity – cryptographic keys and digital certificates – so they can’t be misused by bad guys in attacks...
There will be new vendors providing applications, middleware, and connected devices to support the thriving IoT ecosystem. This essentially means that electronic device manufacturers will also be in the software business. Many will be new to building embedded software or robust software. This creates an increased importance on software quality, particularly within the Industrial Internet of Things where business-critical applications are becoming dependent on products controlled by software. Qua...
In addition to all the benefits, IoT is also bringing new kind of customer experience challenges - cars that unlock themselves, thermostats turning houses into saunas and baby video monitors broadcasting over the internet. This list can only increase because while IoT services should be intuitive and simple to use, the delivery ecosystem is a myriad of potential problems as IoT explodes complexity. So finding a performance issue is like finding the proverbial needle in the haystack.
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning techniques such as Cognitive Fingerprinting, wind project operators can utilize these tools to learn from collected data, detect regular patterns, and optimize their own operations. In his session at 18th Cloud Expo, Stuart Gillen, Director of Business Development at SparkCognition, discussed how research has demonstrated the value of Machine Learning in delivering next generation analytics to imp...
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develo...
SYS-CON Events announced today that MangoApps will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. MangoApps provides modern company intranets and team collaboration software, allowing workers to stay connected and productive from anywhere in the world and from any device.
Large scale deployments present unique planning challenges, system commissioning hurdles between IT and OT and demand careful system hand-off orchestration. In his session at @ThingsExpo, Jeff Smith, Senior Director and a founding member of Incenergy, will discuss some of the key tactics to ensure delivery success based on his experience of the last two years deploying Industrial IoT systems across four continents.
Basho Technologies has announced the latest release of Basho Riak TS, version 1.3. Riak TS is an enterprise-grade NoSQL database optimized for Internet of Things (IoT). The open source version enables developers to download the software for free and use it in production as well as make contributions to the code and develop applications around Riak TS. Enhancements to Riak TS make it quick, easy and cost-effective to spin up an instance to test new ideas and build IoT applications. In addition to...
Identity is in everything and customers are looking to their providers to ensure the security of their identities, transactions and data. With the increased reliance on cloud-based services, service providers must build security and trust into their offerings, adding value to customers and improving the user experience. Making identity, security and privacy easy for customers provides a unique advantage over the competition.
"We've discovered that after shows 80% if leads that people get, 80% of the conversations end up on the show floor, meaning people forget about it, people forget who they talk to, people forget that there are actual business opportunities to be had here so we try to help out and keep the conversations going," explained Jeff Mesnik, Founder and President of ContentMX, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
"There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
Internet of @ThingsExpo, taking place November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with the 19th International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world and ThingsExpo Silicon Valley Call for Papers is now open.
I wanted to gather all of my Internet of Things (IOT) blogs into a single blog (that I could later use with my University of San Francisco (USF) Big Data “MBA” course). However as I started to pull these blogs together, I realized that my IOT discussion lacked a vision; it lacked an end point towards which an organization could drive their IOT envisioning, proof of value, app dev, data engineering and data science efforts. And I think that the IOT end point is really quite simple…
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...
WebRTC is bringing significant change to the communications landscape that will bridge the worlds of web and telephony, making the Internet the new standard for communications. Cloud9 took the road less traveled and used WebRTC to create a downloadable enterprise-grade communications platform that is changing the communication dynamic in the financial sector. In his session at @ThingsExpo, Leo Papadopoulos, CTO of Cloud9, discussed the importance of WebRTC and how it enables companies to focus...