|By Andreas Grabner||
|April 11, 2013 03:25 PM EDT||
We have been blogging about the same problems and problem patterns we see while working with our customers over the past few of years. There have always been the classic application performance landmines in the areas of inefficient database access, misconfigured frameworks, excessive memory usage, bloated web pages and not following common web performance best practices among others.
More than two years ago we posted summary blogs of the Top Server-Side Performance Problems and the Top 10 Client-Side Performance Problems to give operations, architects, testers and developers easy-to-consume best practices. We feel that it is time to provide an update to these best practices as new problem patterns have since come into play. We also want to cover more than just problems that happen within your application by broadening the scope across the entire Application Delivery Chain. This includes all components between your end user and your back-end systems, databases and third-party services. The following illustrates which components are involved and what the typical errors are along the delivery chain.
Delivering an application to the end user has become more complex as it involves more components than ever before. This also leaves a lot of room for mistakes that impact end-user experience.
Let's now dig a little deeper in some of the highlighted problem areas. The following lists our Top Performance Landmines that have been reported by our customers such as BonTon and Swarovski. Other companies include those in the financial services industry, manufacturing industry and energy industry among others. To make it easier for you to decide which landmines to read we also added the target audience for each problem area.
Bloated Web Front Ends
Audience: Operations, Architects, Testers, Developers
Often companies focus on optimizing the performance of the applications they deliver by tuning the code, reducing SQL overhead, implementing application caching, and other items that are, for the most part, invisible to the customer using the application. However, all of this effort and activity can go completely unnoticed if the content being delivered to customers is bloated and inefficient.
Sources we track show that the average page delivered to customers has been steadily increasing in size and complexity over the last 3-4 years as well as customers' expectations of performance. This continuous conflict of business vs customer expectations needs to be understood in order to be effectively managed. What companies need to realize is that what they consider to be fast and efficient doesn't really matter. If the customers using the site believe that the site is slow and hard to use, they won't use it and they will tell their friends about their poor experience.
Comparing your performance to top competitors in your industry as well as Internet leaders helps you set performance goals that can be achieved over time. Additionally, understanding why your customers leave your site can help you resolve customer experience issues: Is it a particular subset of customers who leave? Which page caused them to leave? Is there an application function on that page that is bloated and slow?
Comparing your site against peers in the same industry will help you understand where you rank.
Using caching, compression, CDNs, and a critical eye that asks questions about every new image, function, and feature you add, you can trim the weight of your site and deliver a better customer experience.
We discuss the performance degradation that can be traced to bloated front ends and how this affects site performance in Performance Improvement is not Performance Optimization and Super Bowl Sunday 2013 - Winners, Losers, and Casualties.
Slow Third-Party Content and CDNs
Audience: Operations, Architects, Testers
Focusing on your own content can leave you exposed to performance issues that originate outside your organization. With companies adding more content from third-party sources to their site, managing application performance becomes increasingly complex, even when these services are designed to improve performance.
During peak performance events over the last 12 months - holiday shopping season and the Super Bowl - two primary trends were seen: third-party services were overwhelmed when more than one of their customers reached peak traffic simultaneously and CDNs buckled under flash loads that were far larger than even the busiest days their customers typically experience.
Monitoring and managing third parties means treating them as unique applications, with their own baselines and Service Level Agreements (SLAs) and Service Level Objectives (SLOs). It sometimes means asking tough questions of these services, such as:
- Have you load tested your systems to see what happens when three of your largest customers experience peak traffic simultaneously?
- What is the escalation path we should follow with your team when we discover a performance issue that is affecting our customers?
- How well did your system perform during the eight busiest hours over the last 12 months, not just the average performance?
Monitor the impact of slow third-party and CDN content on your page load time.
Finally, your team needs to be prepared for the scenario where a third-party service or CDN suffers a severe outage or begins to seriously degrade your site performance. Always have a Plan B, C, etc. that gives you the ability to mitigate the issue. These plans could include removing third-party tags, images, and content from your site entirely during peak traffic, load balancing between multiple CDNs, moving content to a secondary cloud provider, all the way to switching to a simple bare bones site that removes all rich media until traffic returns to a normal level.
Unless you know how third parties affect your performance, there is no way for you to manage them effectively. Once you manage your third parties, you can take control of all aspects of your site performance.
More on third-party services and their effects on application performance is covered in: You only control 1/3 of your Page Load Performance!, Third Party Content Management applied: Four steps to gain control of your Page Load Performance!, The Ripple Effect of Facebook's Outage, Third-Party Issues and the Performance Ripple Effect, and Website's Vulnerability to Third-Party Services Exposed.
We also discuss third parties, most notably CDN performance in: Super Bowl Sunday 2013 - Winners, Losers, and Casualties, and Why Bon Ton needs real-time visibility into 85% of its content delivered by Akamai.
Wrong Usage of Frameworks
Audience: Architects, Developers
The following screenshot shows that Hibernate executes the same SQL query multiple times instead of caching the result from the first query. This happens in case Hibernate has not been configured correctly to perform optimally for your specific needs:
Loading a person two times in a row, but no session cache involved
Finally, frameworks get constantly updated to improve functionality but also improve performance and stability. You want to watch out for these updates and also update your implemented framework version to benefit from the improvements. We have seen cases where, e.g., jQuery was never updated leaving websites with bad performance on older browsers and sometimes even on newer browsers when older versions of jQuery didn't leverage the capabilities of the latest IE, FF, Chrome or Safari browsers.
Long-running CSS Class Name Lookups contribute about 80% to the Client-Side Load Time.
If you want to read more about common problems when using these types of frameworks check out our blogs series on Hibernate (The Session Cache, The Query Cache, Second Level Cache), the Top SharePoint Performance Mistakes or the 101 on jQuery Selector Performance.
Network Infrastructure Problems
Audience: Operations, Architects, Testers
Network infrastructure is an important component of every successful business operation. Performance problems experienced by end users can have various origins. The operation teams need Application Performance Monitoring solutions that will enable them to isolate fault domains effortlessly and quickly.
Sometimes the answer is not obvious and performance problems can end up in a "war room" between infrastructure and application providers. The team needs to analyze whether the problem is present at all locations where the application is executed. In certain cases, the performance problems might be caused by external infrastructure used by some users.
Performance problems can be pretty costly. According to the report by the Aberdeen Group they can reduce revenue by 9% and productivity by 64%. When our services are based on the SAP infrastructure the costs can rise to even $15,000 per every minute of a service downtime. Even though SAP provides tools to monitor its components, the proper APM solution should deliver a holistic view over the entire infrastructure. Only then can the Operations team tell whether it is a problem with SAP components that were quite an investment to deploy or it's an infrastructure problem that's not related to the SAP or any application.
Overview of SAP tier with top most under-performing modules and most affected users
The most obvious hints on whether this is a network or an application problem can be seen by checking for the Network and Server time outliers compared to the values of the baseline traffic. But eyeballing the reports is not enough to avoid problems. The first step toward proactive application performance management is to learn to respond promptly to alerts triggered by the APM tool when key measures go outside of the usual range.
Audience: Operations, Architects
"The Cloud" comes with a great promise: endless resources for endless scalability and performance when I need it. This eliminates the need to buy a lot of hardware that sits idle most of the time but is only used during peak traffic periods. It also allows me to scale and perform far beyond what is expected without needing to wait for additional hardware to ship.
But there are some gotchas: throwing hardware at an application that is not designed to scale in a cloud environment won't leverage the possibilities that the cloud provides. In fact, it often ends up being a very costly endeavor. One must also understand that The Cloud - unless we talk about a private cloud setting - is an environment that is not owned by you. Direct access to the underlying hardware is not as easy as if the hardware is located in the next room, which makes troubleshooting or monitoring much harder. The cloud is also not just an endless resource pool of CPU, Memory or Disk On-Demand. It provides lots of other services such as storage, messaging and more which one must understand and monitor for performance, as these services are key components of your application.
It is recommended to live monitor cloud instance usage and cost in order to not fall into a cost trap
Relating to these problem areas you want to read the following blog posts: Managing Hybrid Cloud Environments, Analyzing Performance of Windows Azure Storage, Why Performance Monitoring is easier in Public than onPremise Clouds and Monitoring your Clouds.
Too Many Database Calls
Audience: Architects, Testers, Developers
Database Access is the problem we see the most within the application. It is nothing new - but - as we still see it on almost every application we work with, it is critical enough to mention it again. The first lesson learned is that the blame is often not on the database side but on the access patterns of the application to the database. All too often we see a single web request that queries thousands of database statements. There are multiple reasons for it: fetching too much data beyond just the data that is needed or inefficient fetching of data that then gets aggregated and computed in the application rather than in a stored procedure. What is really interesting is that we see this problem pattern not only in distributed applications running on modern application servers. We also see it on "legacy" applications such as VB6 or even the mainframe. The following screenshot highlights the transaction flow of an enterprise application that calls the mainframe. The mainframe transaction makes 225 SQL executions per transaction. A closer look typically reveals that the same statements are called hundreds of times due to the reasons mentioned above:
The Transaction Flow highlights how services interact with each other including the number of interactions to DB2 which indicate a potential architectural and performance problem.
Besides these access pattern problems we also see individual statements that take a long time to execute. In this case, it is important to not only focus on the database to optimize statements by tweaking indices or the like, it's also important to analyze whether these queries can be optimized from within the application. We often see that too much data is retrieved from the database, which first gets parsed by the application (using extra memory) and is then thrown away (more GC activity). Another landmine is misconfigured connection pools or application code that holds on to connections too long and ends up blocking other threads from accessing the database.
The following screenshot shows the database queries executed by a single transaction, most of them taking very long to execute. The fix to this problem was to optimize these statements in both the application and in the database:
The architects in this case started by optimizing SQL statements that took a long time to execute and those that got executed several times within the same transaction.
For further reading check out our blogs with more detailed background on these problem patterns such as Don't let your load balancers ruin your holiday business or Saving MIPS and Money. For connection pool problems we also have one interesting blog named The reason I don't monitor connection pool usage.
Big Data Not Optimized
Audience: Operations, Architects, Testers, Developers
The amount of data that we and our applications have to process is constantly growing. Big Data solutions (NoSQL, MapReduce...) provide new approaches to storing and processing large amount of data. But as with every technology it needs to be used in an optimized way to fit your specific needs. It is a misconception that you can simply process more data by adding additional resources to, e.g., a MapReduce cluster in order to speed up data processing. This only works if you have implemented your jobs in a way that allows them to scale. The same is true for accessing data from a NoSQL database. The same problems we see with relational databases also apply to accessing data in Big Data solutions. If you make inefficient queries or more queries than necessary, you are going to impact performance.
The following screenshot highlights a transaction that spends most of its time in MongoDB. A closer look into this revealed that the framework used to access MongoDB made a call to a size method of the cursor that then executed an additional query to MongoDB, which was totally unnecessary. In this example, eliminating that call reduced roundtrips to MongoDB and improved overall transaction performance by 15x:
Transactions that call JourneyCollection.getCount spend nearly half their time in MongoDB.
If you are using Big Data technologies such as Cassandra, MongoDB, Hadoop, or the like I suggest following up with the following blog posts that explain some of the problem patterns and highlight best practices: MongoDB Anti-Pattern, NoSQL vs Traditional Databases, Inside Cassandra Write Performance and What we can Learn from Cassandra Pagination. Also check out 15x Performance Improvements for Pig+HBase.
Undetected Memory Leaks
Audience: Architects, Testers, Developers
Memory and Garbage Collection problems are still very prominent issues in any enterprise application. One of the reasons is that the very nature of Garbage Collection is often misunderstood. Besides the traditional memory-related problems such as high memory usage, wrong cache usage strategies, we also see memory issues related to class loading, large classes or native memory. The following screenshot shows the problem of having single objects consuming a lot of memory. Not that this is a bad idea if necessary - but too often this happens because information is kept in memory for no apparent reason and with that consuming memory that is not available for others.
Single Object that is responsible for a big portion of the memory being leaked
Traditional memory leaks often lead to out of memory exceptions and typically to crashes of the virtual machines. This has a negative impact on the end user as the current context of user sessions and active transactions might be lost.
High memory usage on the other hand can result in high garbage collection, which has a direct impact on end user response time. Transactions that are suspended because of long running garbage collection processing can be optimized by tweaking garbage collection settings as well as being less "wasteful" with memory.
Even problems related to wrong implementations of equals/hashcode can lead to memory problems. To address this problem we wrote a full chapter on Memory Management in our Java Enterprise Performance book that explains concepts like How Garbage Collection works, Difference between JVMs, GC Tuning, High Memory Usage and the Root Cause, Class Load Related Problems and more. We have also blogged about specific memory scenarios - check out the following blogs: Memory Monitoring in WebSphere Environments, GC Bottlenecks in Heterogeneous Environments, Leak Detection in Production Environments, Top Memory Problems - Part I and Part II.
More to Come...
These landmines are some highlights with links to more detailed blog posts. As we continue to blog about these problem patterns, we plan to compile a second list of problems later this year. Keep watching our blog for more information and check out our online book on Java Enterprise Performance.
Eighty percent of a data scientist’s time is spent gathering and cleaning up data, and 80% of all data is unstructured and almost never analyzed. Cognitive computing, in combination with Big Data, is changing the equation by creating data reservoirs and using natural language processing to enable analysis of unstructured data sources. This is impacting every aspect of the analytics profession from how data is mined (and by whom) to how it is delivered. This is not some futuristic vision: it's ha...
Feb. 10, 2016 07:45 AM EST Reads: 432
Silver Spring Networks, Inc. (NYSE: SSNI) extended its Internet of Things technology platform with performance enhancements to Gen5 – its fifth generation critical infrastructure networking platform. Already delivering nearly 23 million devices on five continents as one of the leading networking providers in the market, Silver Spring announced it is doubling the maximum speed of its Gen5 network to up to 2.4 Mbps, increasing computational performance by 10x, supporting simultaneous mesh communic...
Feb. 10, 2016 07:00 AM EST
The cloud promises new levels of agility and cost-savings for Big Data, data warehousing and analytics. But it’s challenging to understand all the options – from IaaS and PaaS to newer services like HaaS (Hadoop as a Service) and BDaaS (Big Data as a Service). In her session at @BigDataExpo at @ThingsExpo, Hannah Smalltree, a director at Cazena, will provide an educational overview of emerging “as-a-service” options for Big Data in the cloud. This is critical background for IT and data profes...
Feb. 10, 2016 12:45 AM EST Reads: 195
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
Feb. 10, 2016 12:00 AM EST Reads: 202
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
Feb. 9, 2016 10:15 PM EST Reads: 238
With an estimated 50 billion devices connected to the Internet by 2020, several industries will begin to expand their capabilities for retaining end point data at the edge to better utilize the range of data types and sheer volume of M2M data generated by the Internet of Things. In his session at @ThingsExpo, Don DeLoach, CEO and President of Infobright, will discuss the infrastructures businesses will need to implement to handle this explosion of data by providing specific use cases for filte...
Feb. 9, 2016 05:00 PM EST Reads: 169
SYS-CON Events announced today that VAI, a leading ERP software provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. VAI (Vormittag Associates, Inc.) is a leading independent mid-market ERP software developer renowned for its flexible solutions and ability to automate critical business functions for the distribution, manufacturing, specialty retail and service sectors. An IBM Premier Business Part...
Feb. 9, 2016 04:00 PM EST Reads: 607
Fortunately, meaningful and tangible business cases for IoT are plentiful in a broad array of industries and vertical markets. These range from simple warranty cost reduction for capital intensive assets, to minimizing downtime for vital business tools, to creating feedback loops improving product design, to improving and enhancing enterprise customer experiences. All of these business cases, which will be briefly explored in this session, hinge on cost effectively extracting relevant data from ...
Feb. 9, 2016 03:15 PM EST
As enterprises work to take advantage of Big Data technologies, they frequently become distracted by product-level decisions. In most new Big Data builds this approach is completely counter-productive: it presupposes tools that may not be a fit for development teams, forces IT to take on the burden of evaluating and maintaining unfamiliar technology, and represents a major up-front expense. In his session at @BigDataExpo at @ThingsExpo, Andrew Warfield, CTO and Co-Founder of Coho Data, will dis...
Feb. 9, 2016 03:00 PM EST Reads: 178
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies adopt disruptive technologies to optimize revenue-generating systems, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Founded in 1997, Pythian is a global IT services company that helps companies compete by adopting disruptive technologies such as cloud, Big Data, advanced analytics, and DevO...
Feb. 9, 2016 02:45 PM EST Reads: 208
SYS-CON Events announced today that iDevices®, the preeminent brand in the connected home industry, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. iDevices, the preeminent brand in the connected home industry, has a growing line of HomeKit-enabled products available at the largest retailers worldwide. Through the “Designed with iDevices” co-development program and its custom-built IoT Cloud Infrastruc...
Feb. 9, 2016 02:45 PM EST
SYS-CON Events announced today that Alert Logic, Inc., the leading provider of Security-as-a-Service solutions for the cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Alert Logic, Inc., provides Security-as-a-Service for on-premises, cloud, and hybrid infrastructures, delivering deep security insight and continuous protection for customers at a lower cost than traditional security solutions. Ful...
Feb. 9, 2016 02:15 PM EST Reads: 403
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 ad...
Feb. 9, 2016 01:15 PM EST Reads: 383
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management...
Feb. 9, 2016 11:30 AM EST Reads: 414
With the Apple Watch making its way onto wrists all over the world, it’s only a matter of time before it becomes a staple in the workplace. In fact, Forrester reported that 68 percent of technology and business decision-makers characterize wearables as a top priority for 2015. Recognizing their business value early on, FinancialForce.com was the first to bring ERP to wearables, helping streamline communication across front and back office functions. In his session at @ThingsExpo, Kevin Roberts...
Feb. 9, 2016 08:00 AM EST Reads: 378
One of the bewildering things about DevOps is integrating the massive toolchain including the dozens of new tools that seem to crop up every year. Part of DevOps is Continuous Delivery and having a complex toolchain can add additional integration and setup to your developer environment. In his session at @DevOpsSummit at 18th Cloud Expo, Miko Matsumura, Chief Marketing Officer of Gradle Inc., will discuss which tools to use in a developer stack, how to provision the toolchain to minimize onboa...
Feb. 9, 2016 07:45 AM EST
SYS-CON Events announced today that Fusion, a leading provider of cloud services, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Fusion, a leading provider of integrated cloud solutions to small, medium and large businesses, is the industry's single source for the cloud. Fusion's advanced, proprietary cloud service platform enables the integration of leading edge solutions in the cloud, including clou...
Feb. 6, 2016 03:30 PM EST Reads: 757
Most people haven’t heard the word, “gamification,” even though they probably, and perhaps unwittingly, participate in it every day. Gamification is “the process of adding games or game-like elements to something (as a task) so as to encourage participation.” Further, gamification is about bringing game mechanics – rules, constructs, processes, and methods – into the real world in an effort to engage people. In his session at @ThingsExpo, Robert Endo, owner and engagement manager of Intrepid D...
Feb. 5, 2016 09:00 PM EST Reads: 816
WebRTC has had a real tough three or four years, and so have those working with it. Only a few short years ago, the development world were excited about WebRTC and proclaiming how awesome it was. You might have played with the technology a couple of years ago, only to find the extra infrastructure requirements were painful to implement and poorly documented. This probably left a bitter taste in your mouth, especially when things went wrong.
Feb. 2, 2016 04:30 AM EST Reads: 885
Learn how IoT, cloud, social networks and last but not least, humans, can be integrated into a seamless integration of cooperative organisms both cybernetic and biological. This has been enabled by recent advances in IoT device capabilities, messaging frameworks, presence and collaboration services, where devices can share information and make independent and human assisted decisions based upon social status from other entities. In his session at @ThingsExpo, Michael Heydt, founder of Seamless...
Feb. 1, 2016 05:00 AM EST Reads: 975