Welcome!

Java IoT Authors: Liz McMillan, Pat Romanski, Elizabeth White, Akhil Sahai, Ruxit Blog

Related Topics: Java IoT, Industrial IoT, Open Source Cloud, IoT User Interface, Apache, SDN Journal

Java IoT: Article

Tracing Black Boxes III: Solr Query Performance Tuning

Solr Query Performance Tuning

Solr Server provides JMX statistics that show performance details such as query speed and cache hit/miss rates in a macro level, which James talked about in a previous post. However, it might be tough to trace how a particular operation; for example: how a specific query fared in the system. This week, I'd like to introduce TraceView's latest support for Solr Server, which provides breakdown of each operation, enabling more precise performance monitoring and troubleshooting.

How does Solr work, anyway?

Requests made to Solr Server can be roughly divided into 3 categories: queries, data indexing/update, admin operations (check server health/log, optimize server etc). Requests made to Solr server first go through the SolrDispatchFilter that looks up the corresponding Solr Core (a running instance of index/dataset along with configurations) and handler. The retrieved SolrCore and handler are then used to process the request. For SearchHandler, it is further broken down into smaller SearchComponents (for example, querying, highlighting results, calculating statistics etc).

Take note that since version 4, Solr added distributed capabilities in Solr called SolrCloud, which enable highly available, fault tolerant cluster of Solr servers. Index/dataset is no longer hosted on a single Core/instance. Instead, it is constructed by several shards, each as a disjoint subset of the documents in the index, and each shard can be backed by multiple cores, which are replicas of each other.

For caching, Solr uses several built-in classes (LRUCache, FastLRUCache, LFUCache) for various aspects (field, filter, query result, document etc). They are implemented as in-memory caches to allow quicker operations. The configurations of caches can be found in the Query section of solrconfig.xml.

Indexing and updates can be slow, and certainly compete for resources, but they can be done asynchronously. Admin operations can be slow as well, but they are typically one-time events, or not performance sensitive. Queries are typically the most important operation, as they live in the critical path for the user. TraceView monitor activities on each of the classes that correspond to these actions and their components to extract run-time information on each of the modules described. Let's take a look at an example.

Tracing Queries
A trace follows the path of a request through the entire application. For example, a trace below shows the breakdowns of a query operation:

solr3-request

There's a lot more going on that just searching the index for the given phrase!

Solr has a lot more functionality than just returning a list of documents that match a query. In particular, this query's time is actually dominated by "highlight.process" (highlight matching phrases in the query result) and "ResponseWriter.write" (write data to the response stream). We could speed up this request by nearly 2x by just disabling the highlighter in config!

The other interesting information here is what's not visible. This query didn't hit a cache of any sort. Even for highly variable search terms, the cache can help with users paging through longer lists or sharing links. In this case, TraceView found several cache miss events on "documentCache" that contribute to longer load time.

solr3-cache-info

Pre-warming the cache could help with this, or if this is a common term, it's possible that it's being evicted due to small cache size. We could verify this by checking the JMX statistics for cache eviction rates and adjusting the cache size in our config, if necessary.

Beyond Solr
Instrumentation on Solr Server does not only give an isolated view of how Solr performs, it also draws a complete picture of all the interactions with other applications/systems. For example, below is trace of a query request issued a typical Drupal search page backed by Solr service runs on a different host:

solr3-request-full

We can trace the request handling all the way through the Apache server, Drupal modules, and Solr handling all in one trace. In particular, this tracks requests even when the app and Solr are running on different machines, which allows you to track down issues where the application calls the Solr server incorrectly.

More Stories By Patson Luk

A Java developer who has spent the better part of the last decade working on financial services applications with companies from HSBC to Mobilearth and Parasoft, Patson is experienced in various aspects of computer systems, from large scale enterprise banking system to lightweight mobile payment solutions. He now leads Java instrumentation and tool development for the TraceView product at AppNeta. Patson's focus is on using java bytecode manipulation technologies to gain greater visibility into the full spectrum of Java based technologies. This includes higher level application frameworks from Spring and Struts to Webflow, AppServers from TomCat to JBoss, and Databases from MySQL to Oracle. He's also writes frequently on the AppNeta blog - www.appneta.com/blog

@ThingsExpo Stories
For basic one-to-one voice or video calling solutions, WebRTC has proven to be a very powerful technology. Although WebRTC’s core functionality is to provide secure, real-time p2p media streaming, leveraging native platform features and server-side components brings up new communication capabilities for web and native mobile applications, allowing for advanced multi-user use cases such as video broadcasting, conferencing, and media recording.
IoT generates lots of temporal data. But how do you unlock its value? You need to discover patterns that are repeatable in vast quantities of data, understand their meaning, and implement scalable monitoring across multiple data streams in order to monetize the discoveries and insights. Motif discovery and deep learning platforms are emerging to visualize sensor data, to search for patterns and to build application that can monitor real time streams efficiently. In his session at @ThingsExpo, ...
There will be new vendors providing applications, middleware, and connected devices to support the thriving IoT ecosystem. This essentially means that electronic device manufacturers will also be in the software business. Many will be new to building embedded software or robust software. This creates an increased importance on software quality, particularly within the Industrial Internet of Things where business-critical applications are becoming dependent on products controlled by software. Qua...
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2016 Silicon Valley. The 19th Cloud Expo and 6th @ThingsExpo will take place on November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Interne...
Machine Learning helps make complex systems more efficient. By applying advanced Machine Learning techniques such as Cognitive Fingerprinting, wind project operators can utilize these tools to learn from collected data, detect regular patterns, and optimize their own operations. In his session at 18th Cloud Expo, Stuart Gillen, Director of Business Development at SparkCognition, discussed how research has demonstrated the value of Machine Learning in delivering next generation analytics to imp...
In addition to all the benefits, IoT is also bringing new kind of customer experience challenges - cars that unlock themselves, thermostats turning houses into saunas and baby video monitors broadcasting over the internet. This list can only increase because while IoT services should be intuitive and simple to use, the delivery ecosystem is a myriad of potential problems as IoT explodes complexity. So finding a performance issue is like finding the proverbial needle in the haystack.
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform. In his session at @ThingsExpo, Craig Sproule, CEO of Metavine, demonstrated how to move beyond today's coding paradigm and shared the must-have mindsets for removing complexity from the develo...
SYS-CON Events announced today that MangoApps will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. MangoApps provides modern company intranets and team collaboration software, allowing workers to stay connected and productive from anywhere in the world and from any device.
The IETF draft standard for M2M certificates is a security solution specifically designed for the demanding needs of IoT/M2M applications. In his session at @ThingsExpo, Brian Romansky, VP of Strategic Technology at TrustPoint Innovation, explained how M2M certificates can efficiently enable confidentiality, integrity, and authenticity on highly constrained devices.
SYS-CON Events announced today that LeaseWeb USA, a cloud Infrastructure-as-a-Service (IaaS) provider, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. LeaseWeb is one of the world's largest hosting brands. The company helps customers define, develop and deploy IT infrastructure tailored to their exact business needs, by combining various kinds cloud solutions.
Early adopters of IoT viewed it mainly as a different term for machine-to-machine connectivity or M2M. This is understandable since a prerequisite for any IoT solution is the ability to collect and aggregate device data, which is most often presented in a dashboard. The problem is that viewing data in a dashboard requires a human to interpret the results and take manual action, which doesn’t scale to the needs of IoT.
Internet of @ThingsExpo has announced today that Chris Matthieu has been named tech chair of Internet of @ThingsExpo 2016 Silicon Valley. The 6thInternet of @ThingsExpo will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
When people aren’t talking about VMs and containers, they’re talking about serverless architecture. Serverless is about no maintenance. It means you are not worried about low-level infrastructural and operational details. An event-driven serverless platform is a great use case for IoT. In his session at @ThingsExpo, Animesh Singh, an STSM and Lead for IBM Cloud Platform and Infrastructure, will detail how to build a distributed serverless, polyglot, microservices framework using open source tec...
“delaPlex Software provides software outsourcing services. We have a hybrid model where we have onshore developers and project managers that we can place anywhere in the U.S. or in Europe,” explained Manish Sachdeva, CEO at delaPlex Software, in this SYS-CON.tv interview at @ThingsExpo, held June 7-9, 2016, at the Javits Center in New York City, NY.
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
A critical component of any IoT project is what to do with all the data being generated. This data needs to be captured, processed, structured, and stored in a way to facilitate different kinds of queries. Traditional data warehouse and analytical systems are mature technologies that can be used to handle certain kinds of queries, but they are not always well suited to many problems, particularly when there is a need for real-time insights.
The 19th International Cloud Expo has announced that its Call for Papers is open. Cloud Expo, to be held November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA, brings together Cloud Computing, Big Data, Internet of Things, DevOps, Digital Transformation, Microservices and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding business opportuni...
IoT is rapidly changing the way enterprises are using data to improve business decision-making. In order to derive business value, organizations must unlock insights from the data gathered and then act on these. In their session at @ThingsExpo, Eric Hoffman, Vice President at EastBanc Technologies, and Peter Shashkin, Head of Development Department at EastBanc Technologies, discussed how one organization leveraged IoT, cloud technology and data analysis to improve customer experiences and effi...
The Internet of Things will challenge the status quo of how IT and development organizations operate. Or will it? Certainly the fog layer of IoT requires special insights about data ontology, security and transactional integrity. But the developmental challenges are the same: People, Process and Platform and how we integrate our thinking to solve complicated problems. In his session at 19th Cloud Expo, Craig Sproule, CEO of Metavine, will demonstrate how to move beyond today's coding paradigm ...
Big Data engines are powering a lot of service businesses right now. Data is collected from users from wearable technologies, web behaviors, purchase behavior as well as several arbitrary data points we’d never think of. The demand for faster and bigger engines to crunch and serve up the data to services is growing exponentially. You see a LOT of correlation between “Cloud” and “Big Data” but on Big Data and “Hybrid,” where hybrid hosting is the sanest approach to the Big Data Infrastructure pro...