Welcome!

Java IoT Authors: Liz McMillan, Yeshim Deniz, Zakia Bouachraoui, Elizabeth White, Pat Romanski

Related Topics: @DevOpsSummit, Java IoT, @DXWorldExpo

@DevOpsSummit: Article

Multi-Threaded Apps By @GrabnerAndi | @DevOpsSummit [#DevOps]

Why apps show high response time contribution to web requests coming from worker threads

How to Analyze Problems in Multi-Threaded Applications

As part of my Share Your PurePath and Performance Clinic initiatives I get to see lots of interesting problems out there. This time I picked two examples that just came in this week from Balasz and Daniel. Both wanted my opinion on why their apps show high response time contribution to their web requests coming from worker threads that seem to be either in I/O or in a Wait state. The question was what are these threads waiting for and whether this is something that could be optimized to speed up these slow response times they see on some of their critical web requests.

For both apps it turned out that the developers chose to "offload" work items to a pool of worker threads. This is of course a very valid design pattern. The way it was implemented though didn't fully leverage the advantage that multi-threading can give you. We identified two patterns that I will now describe in more detail in the hope that your multi-threaded applications are not suffering from these performance anti-patterns:

  1. Sequential instead of parallel execution of background threads
  2. Many parallel background threads using "non-shareable" resources

I also want to take the opportunity to give you some tips on how to "read" Dynatrace PurePaths. Over the years I've developed my own technique and I hope you find my approach worthwhile to test on your own data.

Pattern #1: Sequential Execution of Threads and a bit more ...
I already tweeted last week. I created the following slide for Balasz to present to his development team including tips on what I look for when analyzing a PurePath:

PurePath showing that the main web request thread executes 3 RMI calls followed by two background threads where the second one actually starts when the first is finished - that's sequential execution

The many callouts I put on this slide might be a bit overwhelming. Let me explain the highlights.

The overall request response time as measured on the web server is 9.896s. The contributors to this response time are pointed out in the callouts that I numbered. Here is short version of it:

  1. The main service request thread on the Java servlet container takes 3203s where 1.770s is almost entirely spent in I/O.
  2. The main service request thread makes 3 RMI calls. The "Elapsed Time" (=Timestamp on when that method was called in relation to the start of the request) shows us that the first one executes after 366ms followed by the second and third in sequential order
  3. After the third RMI call - at exactly 1.800s Elapsed Time the first background thread is started. This thread takes 1.429s to execute.
  4. The second background worker thread was supposed to run in parallel - at least based on what Balasz told me. The Elapsed Time column however shows that it was executed AFTER the first background thread was done. This was not intended that way. It also explains the 1.770s that show up as I/O time on the main service thread. It is mainly the time that the main thread waited until the first background thread was done just to kick off the second background thread.
  5. The second background thread was now really executed in parallel letting the main service request thread finish. This second background thread generates and writes the generated HTML using the HttpServletResponse Output Stream. This also explains the long waiting time on the Web Server as it was waiting 6.690s for that asynchronous background thread to write data to the output stream.

Lesson Learned: Verify if your threads execute as intended
Looking at the actual time stamp (=Elapsed Time) tells you when your threads really get started and whether they execute in parallel or not. It's a great way to verify your actual implementation and lets you learn which component is waiting for which other component.

For more patterns and lessons learned, click here for the full article.

More Stories By Andreas Grabner

Andreas Grabner has been helping companies improve their application performance for 15+ years. He is a regular contributor within Web Performance and DevOps communities and a prolific speaker at user groups and conferences around the world. Reach him at @grabnerandi

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


IoT & Smart Cities Stories
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
@DevOpsSummit at Cloud Expo, taking place November 12-13 in New York City, NY, is co-located with 22nd international CloudEXPO | first international DXWorldEXPO and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time t...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
CloudEXPO New York 2018, colocated with DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
Bill Schmarzo, Tech Chair of "Big Data | Analytics" of upcoming CloudEXPO | DXWorldEXPO New York (November 12-13, 2018, New York City) today announced the outline and schedule of the track. "The track has been designed in experience/degree order," said Schmarzo. "So, that folks who attend the entire track can leave the conference with some of the skills necessary to get their work done when they get back to their offices. It actually ties back to some work that I'm doing at the University of San...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
DXWorldEXPO | CloudEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
DXWorldEXPO LLC announced today that Telecom Reseller has been named "Media Sponsor" of CloudEXPO | DXWorldEXPO 2018 New York, which will take place on November 11-13, 2018 in New York City, NY. Telecom Reseller reports on Unified Communications, UCaaS, BPaaS for enterprise and SMBs. They report extensively on both customer premises based solutions such as IP-PBX as well as cloud based and hosted platforms.