Welcome!

Java IoT Authors: Liz McMillan, Pat Romanski, Elizabeth White, Yeshim Deniz, Frank Lupo

Related Topics: Java IoT

Java IoT: Article

JFluid: A New Way to Profile Java Applications

JFluid: A New Way to Profile Java Applications

Anyone who develops production applications eventually spends some time profiling. JFluid is an experimental new technology for profiling Java code. It was developed at Sun Microsystems Laboratories and can be a handy tool in your profiling toolbox.

Your application should run fast and not overconsume valuable resources such as memory. For production applications it's important to "scale well" by running quickly and within a reasonable memory footprint when the workload increases. Profiling tools help identify bottlenecks in your code - sections that contribute the most to execution time and resource consumption. Profiling can reveal unanticipated facts about application performance. In the world of Java application development, all too often performance assumptions tend to be wrong.

Until now there have been two ways to profile Java code. You can modify the code by hand or use a profiling tool that leverages the Java Virtual Machine Profiler Interface (JVMPI). The first approach (e.g., inserting calls to System.currentTimeMillis()) is tedious and error prone. The second approach requires learning one of the many JVMPI profiling tools. Some are open source; others are commercial products. JFluid provides a third alternative.

Features
First, an important note: currently JFluid works only with its own Java Virtual Machine (JVM). This JVM is a slightly modified version of Sun's v1.4.2 HotSpot JVM and is fully compatible with the unmodified JVM. The only difference is a small internal profiling API. Replacing the standard JVM with the JFluid JVM is easy - see the "Installation" section of this article.

The JFluid tool is a Swing application that is currently a bit rough around the edges - hopefully it will improve over time. The application to be profiled can be started by the tool. Alternatively, you can attach the tool to a JVM that is already running. Attaching to a running JVM is useful when you want to profile applications that run in a container such as a Web or application server. An important JFluid feature is that no special command-line settings are required when starting the JVM to which you eventually attach. Until the JFluid tool is attached, the application runs at full speed with no profiling overhead. Think about the implication of that. It means that applications can be profiled in their normal deployed environment without advance preparation and without incurring any overhead when profiling is not being done. In other words, JFluid allows profiling to be turned on and off at will.

Minimizing overhead is a cornerstone of JFluid's design. Its CPU profiling allows you to profile a subset of the application's methods. By not profiling the rest of the methods you can dramatically reduce profiling overhead. To profile a group of methods, first select one or more methods manually. JFluid treats each selected method as the root of a "call graph". Starting at the root method JFluid determines which methods are called by it, repeating the process recursively until the entire call subgraph is identified. Only the methods that belong to the call subgraph are instrumented and profiled; the rest of your code runs unchanged at full speed. Method selections can be changed arbitrarily while the program is running. This allows you to perform a "drill down" with decreasing overhead, or to successively profile different code areas for which there would be too much overhead if profiled together. Tools that allow only a selection of methods for profiling by name or package are unable to provide this functionality in such an easy-to-use form.

JFluid also tracks memory allocations with an eye on minimizing overhead. By default JFluid does not record complete information on every object allocation; instead it does statistical sampling by keeping a counter for each class. It decrements the class's counter when an object is allocated. When the allocation counter reaches zero, JFluid does detailed profiling and resets the counter. Detailed profiling of an allocation consists of capturing a stack trace and monitoring whether the object is still on the heap, in other words the object's liveness. By default the counter is 10 so 10% of object allocations are tracked. In production applications, particularly server applications, the number of objects allocated is so high that the information that is discarded is usually not significantly different from what is retained. The allocation counter can be decreased to improve precision or increased to reduce overhead.

For tracked allocations JFluid records the age of each object, where "age" means number of garbage collections the object has survived. It also reports the number of different ages for all tracked allocations of a class; this is called "surviving generations." The surviving generations value is useful for detecting memory leaks caused by objects that are constantly generated, but only partially garbage collected. In other words, the group of objects grows steadily. "Steadily" is the key word: it is not a group of objects that has been generated once within a short period of time in the past and remains fixed since then. Nor is it a group of objects that may grow for quite some time, but then get collected at once. In both of those cases, the number of different ages of objects within a group would be small, no matter how young or old the objects themselves are. It is typically in a leaking object group that the number of different ages grows steadily.

JFluid can also tell you where in your code an object was allocated. However, it does not help answer the question of why an object has not been garbage collected. In other words, no display is provided to show which objects are pointing to the suspected leaking object. Sometimes just knowing where an object was allocated is enough of a clue to help you figure out why it has not been garbage collected. When that is not enough, a heap graph would be helpful. We have yet to see what JFluid is going to offer in this area.

In addition to the instrumentation of CPU usage and heap allocations, JFluid provides four monitors. These graphs show thread count, current heap size and usage, time spent doing garbage collection, and the count of the number of different ages for all heap objects. The count of the number of different ages is the surviving generations value for the entire heap.

Installation
All the necessary .zip files are on the JFluid Web site. Support is included for SPARC Solaris, x86 Solaris, Linux, and Windows. Installation consists of two parts: replacing the JVM and installing the tool.

To replace the JVM, download the binary files (libjvm.so or jvm.dll, depending on your platform) and put them in place of the corresponding files in your standard JVM. Alternatively, you can download a complete JDK that has already been modified to support JFluid.

To install the JFluid tool, unzip the file into any directory. Edit the shell script or batch file that's used to start the JFluid tool so that your JVM and JFluid directories are specified, and you are ready to go.

Sample Program
To demonstrate JFluid I wrote a simple prime number generator that uses the Sieve of Eratosthenes. Its method accepts a single integer parameter and returns all prime numbers less than or equal to that integer. (The source code for this article can be downloaded from www.sys-con.com/java/sourcec.cfm.) The code is poorly written on purpose so that I can demonstrate some of JFluid's features. To test JFluid's ability to connect to a container, I also wrote a poorly implemented servlet that uses the prime number class; the servlet wastes an atrocious amount of heap space. Finally, I created a small .html form that invokes the servlet.

I ran JFluid on both a SunBlade 2000 running Solaris 9 and a Pentium3-based PC running Windows 2000. All screenshots in this article are from the PC, which is where I ran the example profiling session.

Example Profiling Session
After modifying my JVM with the JFluid files, I started the Tomcat servlet container. After starting the JFluid tool I modified its settings, which includes two different CLASSPATH values. The first is for the main application class loader, the second for any other class loader that gets used by your code. For JFluid to be able to find and instrument classes of my servlet I had to specify the servlet's classes in the path for the other class loader (see Figure 1). In the Instrumentation tab of the settings dialog I turned on profiling of the core Java classes, which is off by default to reduce overhead.

To begin a profiling session I chose Run>Attach from the menu. JFluid prompted me for the name of the working directory for the JVM process - with my Tomcat installation this is the directory in which I started Tomcat. After specifying the directory I clicked on the console window that displayed when I started Tomcat; that allowed me to send the JVM a signal by pressing Ctrl-Break on my keyboard. I then returned to JFluid and verified that it connected to the JVM. This process is awkward but painless. The steps for connecting to the JVM running on a Unix system are a bit different; I have some tips on that below.

After JFluid attached to the JVM I clicked the Monitors tab to see the graph of heap usage. This graph is always available, even when no detailed instrumentation of memory allocations is being done. My next task was to look into the slow performance of the servlet. Using the Select and add root method> From binary menu entry, I specified my PrimeServlet.doPost() method. With that done, I chose the Instrument> Selected root methods transitively from the menu and I was ready to go. In a browser window I brought up my HTML form, typed 123456 as the maximum value, and then clicked Submit. After a pause my browser displayed results.

Meanwhile, JFluid was profiling my code. Clicking on the Profile> Get latest results menu took me to the CPU results tab, which displayed the window in Figure 2. The top line showed that my doPost() method took 409 milliseconds (ms). Of that time, however, only 2.16 ms were actually spent in the code of doPost() itself; the rest of the time was spent in methods called by doPost(). In particular, the java.lang. String.split() method took up a huge amount of time - 243 ms. That's almost twice the amount of time used by PrimeNumbers.getEratosthenes() to generate the prime numbers. Drilling a little deeper, I saw that most of the time in PrimeNumbers getEratosthenes() was actually spent in a method I wrote to convert the answer to a string. Wasteful string processing was hurting performance.

In addition to wasting CPU cycles, the code was also wasting heap space. Using the Monitors tab to watch the graph of heap usage I noticed that the heap continued to grow as I used the servlet to request additional prime numbers. Some of that growth was expected, but the servlet was caching answers so requests with a previously requested value should not have caused much additional memory usage. But they did. This is due to a bug I put in on purpose: the cache was broken because the key used to add to the cache was not the same key used to request entries from the cache. So if 123456 is requested four times, there will be four instances of PrimeNumber in the cache, even though only one is needed. The extra instances of PrimeNumber cannot be garbage collected from the heap because the cache holds references to them.

To identify this memory leak I selected Object liveness from the Instrument menu. This turned off JFluid's CPU profiling and turned on detailed profiling of object allocations. Since this was a small application, I set the allocation counter to 1 to get complete profiling. To reduce the overhead incurred, I set the stack trace depth that it records to 3. Then I went back to my browser and did four more requests of prime numbers.

When I clicked on Profile > Get latest results, the Memory results tab displayed with the list of objects allocated since my selection of Object liveness. The line of interest was this one:

4 live obj. - 4 alloc obj. - 3.8 avg. age - 2 surv. gen. - 4 total alloc obj. -
PrimeNumbers

Each time I requested a prime number I used the same value: 123456. Only one live instance of the PrimeNumbers class should exist. There are three extra objects because of the cache bug. After "live objects," the other figures reported are:

  • alloc. obj: The number of allocations being tracked
  • avg. age: The average of the number of garbage collections the live objects have survived
  • surv. gen.: The number of different ages of the live objects
  • total alloc. obj: All allocations including those that are not being tracked
These numbers provide clues when you are looking for memory leaks. Double-clicking the line of figures brings up a window that shows the different stack traces that were captured for each profiled object. In my example application the problem was obvious from the number of live objects reported. Unfortunately, memory leak detection is not always so simple. This is why the average age and surviving generation tail values are displayed. As described in the "Features" section, classes with large values for these figures are potential sources of memory leaks.

Using JFluid on Unix Systems
The only difference between running JFluid on a Windows system and a Unix system is the method of attaching to a running JVM. On Unix systems the JFluid tool sends a SIGQUIT signal to the running JVM to establish a connection. To do that the JFluid client must be running with adequate privileges. For example, when testing on Solaris 9, I was attaching the JFluid client to the JVM used by Sun ONE Web Server v6.1. By default, that JVM was run by the user ID nobody. So I had to run the JFluid tool as nobody, not as another user.

Sun Microsystems Laboratories
4150 Network Circle
Santa Clara, CA 95054
Web: http://research.sun.com
Phone: 800 555-9SUN

Conclusion
JFluid is an experimental but powerful tool. It provides detailed profiling information on demand, allowing you to turn profiling on and off at will. When profiling is on you can control how much overhead is incurred. The JFluid tool lacks some features and polish, but it's easy to learn and installation is quick.

References

  • JFluid: http://research.sun.com/projects/jfluid/
  • JFluid research: http://research.sun.com/techrep/2003/abstract-125.html
  • JVMPI: http://java.sun.com/j2se/1.4.2/docs/guide/jvmpi/jvmpi.html
  • More Stories By Gregg Sporar

    Gregg Sporar is a Staff Engineer in the Services Division of Sun Microsystems and is a Sun Certified Java Developer.

    Comments (0)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    @ThingsExpo Stories
    In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It’s clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Tha...
    In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
    As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
    SYS-CON Events announced today that Ryobi Systems will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ryobi Systems Co., Ltd., as an information service company, specialized in business support for local governments and medical industry. We are challenging to achive the precision farming with AI. For more information, visit http:...
    Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
    High-velocity engineering teams are applying not only continuous delivery processes, but also lessons in experimentation from established leaders like Amazon, Netflix, and Facebook. These companies have made experimentation a foundation for their release processes, allowing them to try out major feature releases and redesigns within smaller groups before making them broadly available. In his session at 21st Cloud Expo, Brian Lucas, Senior Staff Engineer at Optimizely, will discuss how by using...
    In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lead...
    SYS-CON Events announced today that Daiya Industry will exhibit at the Japanese Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ruby Development Inc. builds new services in short period of time and provides a continuous support of those services based on Ruby on Rails. For more information, please visit https://github.com/RubyDevInc.
    As businesses evolve, they need technology that is simple to help them succeed today and flexible enough to help them build for tomorrow. Chrome is fit for the workplace of the future — providing a secure, consistent user experience across a range of devices that can be used anywhere. In her session at 21st Cloud Expo, Vidya Nagarajan, a Senior Product Manager at Google, will take a look at various options as to how ChromeOS can be leveraged to interact with people on the devices, and formats th...
    SYS-CON Events announced today that Yuasa System will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Yuasa System is introducing a multi-purpose endurance testing system for flexible displays, OLED devices, flexible substrates, flat cables, and films in smartphones, wearables, automobiles, and healthcare.
    SYS-CON Events announced today that Taica will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Taica manufacturers Alpha-GEL brand silicone components and materials, which maintain outstanding performance over a wide temperature range -40C to +200C. For more information, visit http://www.taica.co.jp/english/.
    Organizations do not need a Big Data strategy; they need a business strategy that incorporates Big Data. Most organizations lack a road map for using Big Data to optimize key business processes, deliver a differentiated customer experience, or uncover new business opportunities. They do not understand what’s possible with respect to integrating Big Data into the business model.
    Recently, REAN Cloud built a digital concierge for a North Carolina hospital that had observed that most patient call button questions were repetitive. In addition, the paper-based process used to measure patient health metrics was laborious, not in real-time and sometimes error-prone. In their session at 21st Cloud Expo, Sean Finnerty, Executive Director, Practice Lead, Health Care & Life Science at REAN Cloud, and Dr. S.P.T. Krishnan, Principal Architect at REAN Cloud, will discuss how they b...
    Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
    SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
    SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
    SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
    SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
    SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
    Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.