Welcome!

Java IoT Authors: Elizabeth White, Yeshim Deniz, Tim Hinds, Douglas Lyon, Stackify Blog

Related Topics: @DevOpsSummit, Java IoT, Linux Containers, @CloudExpo, Apache

@DevOpsSummit: Blog Feed Post

Solr vs. Elasticsearch — How to Decide? By @Sematext | @DevOpsSummit [#DevOps]

Which one is better, Solr or Elasticsearch? Which one is faster? Which one scales better?

Solr vs. Elasticsearch — How to Decide?

By Otis Gospodnetić

[Otis is a Lucene, Solr, and Elasticsearch expert and co-author of “Lucene in Action” (1st and 2nd editions).  He is also the founder and CEO of Sematext. See full bio below.]

“Solr or Elasticsearch?”…well, at least that is the common question I hear from Sematext’s consulting services clients and prospects.  Which one is better, Solr or Elasticsearch?  Which one is faster?  Which one scales better?  Which one can do X, and Y, and Z?  Which one is easier to manage?  Which one should we use?  Which one do you recommend? etc., etc.

These are all great questions, though not always with clear and definite, universally applicable answers. So which one do we recommend you use? How do you choose in the end?  Well, let me share how I see Solr and Elasticsearch past, present, and future, let’s do a bit of comparing and contrasting, and hopefully help you make the right choice for your particular needs.

Early Days: Youth vs. Experience

Apache Solr is a mature project with a large and active development and user community behind it, as well as the Apache brand.  First released to open-source in 2006, Solr has long dominated the search engine space and was the go-to engine for anyone needing search functionality.  Its maturity translates to rich functionality beyond vanilla text indexing and searching; such as faceting, grouping (aka field collapsing), powerful filtering, pluggable document processing, pluggable search chain components, language detection, etc.

Solr dominated the search scene for several years.  Then, around 2010, Elasticsearch appeared as another option on the market.  Back then it was nowhere near as stable as Solr, did not have Solr’s feature depth, did not have the mindshare, brand, and so on.  But it had a few other things going for it: Elasticsearch was young and built on more modern principles, aimed at more modern use cases, and was built to make handling of large indices and high query rates easier.  Moreover, because it was so young and without a community to work with, it had the freedom to move forward in leaps and bounds, without requiring any sort of consensus or cooperation with others (users or developers), backwards compatibility, or anything else that more mature software typically has to handle.  As such it exposed certain highly sought-after functionality (e.g., Near Real-Time Search) before Solr did.  Technically speaking, the ability to have NRT Search really came from Lucene, the underlying search library to both Solr and Elasticsearch use.  The irony is that because Elasticsearch exposed NRT Search first, people associated NRT Search with Elasticsearch, even though Solr and Lucene are both part of the same Apache project and, as such, one would expect Solr to have such highly demanded functionality first.

Elasticsearch, being more modern, appealed to several groups of people and organizations:

  • those who didn’t yet have a search engine and hadn’t invested a lot of time, money, and energy in its adoption, integration, etc.
  • those who had to deal with large volumes of data and needed to more easily shard and replicate data (search indices) and shrink or grow their search cluster

Of course, let’s admit it, there will always be those who like jumping on new shiny objects, too.

Evening the Search Playing Field

Fast forward to 2014 and now 2015.  Elasticsearch is no longer new, but it’s still shiny.  It closed the feature gap with Solr and, in some cases, surpassed it.  It certainly has more buzz around it.  At this point both projects are very mature.  Both have lots of features.  Both are stable.  I have to say though, that I do see more Elasticsearch clusters with issues, but I think that is primarily because of a few reasons:

  • Elasticsearch, traditionally being easier to get started with, made it possible for anyone to start using it out of the box, without too much understanding of how things work.  That’s great to get started, but dangerous when data/cluster grows.
  • Elasticsearch, lending itself to easier scaling, attracts use cases demanding larger clusters with more data and more nodes.
  • Elasticsearch is more dynamic – data can easily move around the cluster as its nodes come and go, and this can impact stability and performance of the cluster.
  • While Solr has traditionally been more geared toward text search, Elasticsearch is aiming to handle analytical types of queries, too, and such queries come at a price.

Although this may sound scary, let me put it this way — Elasticsearch exposes a ton of control knobs one can play with to control the beast.  Of course, the key bit is that one has to be aware of all possible knobs, know what they do, and make use of that.  For example, despite what you just read about Elasticsearch, we rely on it in our organization for several different products, even though we know Solr just as well as we know Elasticsearch.

Solr: Not Totally Eclipsed

What about Solr?  Solr hasn’t exactly stood still.  The appearance of Elasticsearch was actually great for Solr and its community of developers and users.  Despite being almost 10 years old, Solr development is going faster than ever.  It, too, has a friendly API now.  It, too, has the ability to more easily grow and shrink clusters, create indices more dynamically, shard them on the fly, route documents and queries, etc., etc. Note: when people refer to SolrCloud they specifically mean this form of very distributed, Elasticsearch-like Solr deployment.

I recently attended a Lucene/Solr Revolution conference in Washington, DC and was pleasantly surprised by what I saw: a strong community, healthy project, lots of big name companies not only using Solr, but investing in it through adoption, contribution through development/engineering time, etc.  If you follow just the news you’d be led to believe Solr is dead and everyone is just flocking to Elasticsearch.  That is actually not the case.  Elasticsearch being newer, is naturally more interesting to write about.  Solr was news 5+ years ago.  And of course there were some people going from Solr to Elasticsearch when Elasticsearch appeared — in the beginning there were simply no Elasticsearch users.

So which is better?  Which one should you use?  Where do Solr and Elasticsearch differ?  What does the future hold?

Here are some other things you should keep in mind:

  • Both are released under the Apache Software License
  • Solr is truly open-source — community over code.  Anyone can contribute to Solr and new Solr developers (aka committers) are elected based on merit.  Elasticsearch is technically open-source, but less so in spirit.  Anyone can see the source, anyone can change it and offer a contribution, but only employees of Elasticsearch can actually make changes to Elasticsearch.
  • Solr contributors and committers come from a number of different organizations, while Elasticsearch committers are from a single company.
  • A number of organizations have chosen Solr over Elasticsearch as their horses in the search race (e.g. Cloudera, Hortonworks, MapR, etc.) even though they’ve also partnered with Elasticsearch.
  • Both Solr and Elasticsearch have lively user and developer communities and are rapidly being developed.
  • If you need to add certain missing functionality to either Solr or Elasticsearch, you may have more luck with Solr.  True, there are ancient Solr JIRA issues that are still open, but at least they are still open and not closed.  In Solr world the community has a bit more say even though at the end of the day it’s one of the Solr developers who has to accept and handle the contribution.
  • Both have good commercial support (consulting, production support, integration, etc.)
  • Both have good operational tools around it, although Elasticsearch has, because of its easier-to-work-with API, attracted the DevOps crowd a lot more, thus enabling a livelier ecosystem of tools around it.
  • Elasticsearch dominates the open-source log management use case — lots of organizations index their logs in Elasticsearch to make them searchable.  While Solr can now be used for this, too (see Solr for Indexing and Searching Logs and Tuning Solr for Logs), it just missed the mindshare boat on this one.
  • Solr is still much more text-search-oriented.  On the other hand, Elasticsearch is often for filtering and grouping – the analytical query workload – and not necessarily text search.  Elasticsearch developers are putting a lot of effort into making such queries more efficient (lowering of the memory footprint and CPU usage) at both Lucene and Elasticsearch level.  As such, at this point in time, Elasticsearch is a better choice for applications that need to do not just text search, but also complex search-time aggregations.
  • Elasticsearch is a bit easier to get started – a single download and a single command to get everything started.  Solr has traditionally required a bit more work and knowledge, but Solr has recently made great strides to eliminate this and now just has to work on changing its reputation.
  • Performance-wise, they are roughly the same.  I say “roughly”, because nobody has ever done comprehensive and non-biased benchmarks.  For 95% of use cases either choice will be just fine in terms of performance, and the remaining 5% need to test both solutions with their particular data and their particular access patterns.
  • Operationally speaking, Elasticsearch is a bit simpler to work with – it has just a single process.  Solr, in its Elasticsearch-like fully distributed deployment mode known as SolrCloud, depends on Apache ZooKeeper.  ZooKeeper is super mature, super widely used, etc. etc., but it’s still another moving part.  That said, if you are using Hadoop, HBase, Spark, Kafka, or a number of other newer distributed software, you are likely already running ZooKeeper somewhere in your organization.
  • While Elasticsearch has built-in ZooKeeper-like component called Xen, ZooKeeper is better at preventing the dreaded split-brain problem sometimes seen in Elasticsearch clusters.  To be fair, Elasticsearch developers are aware of this problem and are working on improving this aspect of Elasticsearch.
  • If you love monitoring and metrics, with Elasticsearch you’ll be in heaven.  The thing has more metrics than people you can squeeze in Times Square on New Year’s Eve!  Solr exposes the key metrics, but nowhere near as many as Elasticsearch.  Regardless, having comprehensive monitoring and centralized logging tools like Sematext’s SPM Performance Monitoring and Logsene Log Management and Analytics — especially when they work seamlessly together like these two do — is essential if you want to have a handle on metrics and other operational data.

Here are a few charts to demonstrate what I mean:

Elasticsearch user mailing list traffic: 36,127 (source: Search-Lucene)

Solr user mailing list traffic is about two thirds of the Elasticsearch mailing list traffic.  Of course, this could be because there are more Elasticsearch users, or because there are more problems with Elasticsearch and users are in need of more help, or perhaps they are just a chattier bunch.

Elasticsearch vs. Solr Contributors (source: Open Hub)   click to enlarge

As you can see, Elasticsearch numbers are trending sharply upward, and now more than double Solr with regard to Commit activity.  This is not a very precise or absolutely correct way to compare open-source projects, but it gives us some data points.  For example, Elasticsearch is developed on Github, which makes it very easy to merge others’ Pull Requests, while Solr contributors tend to create patches, upload them to JIRA, where they get reviewed by Solr committers before being applied — a less streamlined process.  Moreover, Elasticsearch repository contains documentation, not just code, while Solr keeps its documentation in a Wiki.  This contributes to higher numbers for both commits and contributors for Elasticsearch.

Boil It Down For Me

In conclusion, here are the bits that I think make the most difference for anyone having to make a choice:

  • If you’ve already invested a lot of time in Solr, stick with it, unless there are specific use cases that it just doesn’t handle well.  If you think that is the case, speak to somebody close to both Solr and Elasticsearch projects to save you time, guessing, research, and avoid mistakes.
  • If you are a strong believer in true open-source, Solr is closer to that than Elasticsearch, and having one company control Elasticsearch may be a turn-off.
  • If you need a data store that can handle analytical queries in addition to text searching, Elasticsearch is a better choice for that today.

If you expected a single definitive winner, I’m sorry to disappoint.  We don’t have one here.  However, I hope this quick comparison of the two leading open-source search engines provides enough information and guidance to help you make the right choice for your organization

About the author: in addition to being a Lucene, Solr, and Elasticsearch expert and author, Otis Gospodnetić is the founder and CEO of Sematext. Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection of Solr, Elasticsearch, Hadoop, Spark and many other applications (SPM), log management and analytics (Logsene), site search analytics (SSA), and search enhancement. The company also provides Search and Big Data consulting services and offers 24/7 production support for Solr and Elasticsearch to clients worldwide.

[Note: the original version of this article appeared at Datanami.com

Read the original blog entry...

More Stories By Sematext Blog

Sematext is a globally distributed organization that builds innovative Cloud and On Premises solutions for performance monitoring, alerting and anomaly detection (SPM), log management and analytics (Logsene), and search analytics (SSA). We also provide Search and Big Data consulting services and offer 24/7 production support for Solr and Elasticsearch.

@ThingsExpo Stories
Leading companies, from the Global Fortune 500 to the smallest companies, are adopting hybrid cloud as the path to business advantage. Hybrid cloud depends on cloud services and on-premises infrastructure working in unison. Successful implementations require new levels of data mobility, enabled by an automated and seamless flow across on-premises and cloud resources. In his general session at 21st Cloud Expo, Greg Tevis, an IBM Storage Software Technical Strategist and Customer Solution Architec...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
Product connectivity goes hand and hand these days with increased use of personal data. New IoT devices are becoming more personalized than ever before. In his session at 22nd Cloud Expo | DXWorld Expo, Nicolas Fierro, CEO of MIMIR Blockchain Solutions, will discuss how in order to protect your data and privacy, IoT applications need to embrace Blockchain technology for a new level of product security never before seen - or needed.
Imagine if you will, a retail floor so densely packed with sensors that they can pick up the movements of insects scurrying across a store aisle. Or a component of a piece of factory equipment so well-instrumented that its digital twin provides resolution down to the micrometer.
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settle...
BnkToTheFuture.com is the largest online investment platform for investing in FinTech, Bitcoin and Blockchain companies. We believe the future of finance looks very different from the past and we aim to invest and provide trading opportunities for qualifying investors that want to build a portfolio in the sector in compliance with international financial regulations.
No hype cycles or predictions of a gazillion things here. IoT is here. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, an Associate Partner of Analytics, IoT & Cybersecurity at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He also discussed the evaluation of communication standards and IoT messaging protocols, data...
Coca-Cola’s Google powered digital signage system lays the groundwork for a more valuable connection between Coke and its customers. Digital signs pair software with high-resolution displays so that a message can be changed instantly based on what the operator wants to communicate or sell. In their Day 3 Keynote at 21st Cloud Expo, Greg Chambers, Global Group Director, Digital Innovation, Coca-Cola, and Vidya Nagarajan, a Senior Product Manager at Google, discussed how from store operations and ...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, whic...
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
We are given a desktop platform with Java 8 or Java 9 installed and seek to find a way to deploy high-performance Java applications that use Java 3D and/or Jogl without having to run an installer. We are subject to the constraint that the applications be signed and deployed so that they can be run in a trusted environment (i.e., outside of the sandbox). Further, we seek to do this in a way that does not depend on bundling a JRE with our applications, as this makes downloads and installations rat...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
DX World EXPO, LLC, a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
In this strange new world where more and more power is drawn from business technology, companies are effectively straddling two paths on the road to innovation and transformation into digital enterprises. The first path is the heritage trail – with “legacy” technology forming the background. Here, extant technologies are transformed by core IT teams to provide more API-driven approaches. Legacy systems can restrict companies that are transitioning into digital enterprises. To truly become a lead...
Digital Transformation (DX) is not a "one-size-fits all" strategy. Each organization needs to develop its own unique, long-term DX plan. It must do so by realizing that we now live in a data-driven age, and that technologies such as Cloud Computing, Big Data, the IoT, Cognitive Computing, and Blockchain are only tools. In her general session at 21st Cloud Expo, Rebecca Wanta explained how the strategy must focus on DX and include a commitment from top management to create great IT jobs, monitor ...
"Cloud Academy is an enterprise training platform for the cloud, specifically public clouds. We offer guided learning experiences on AWS, Azure, Google Cloud and all the surrounding methodologies and technologies that you need to know and your teams need to know in order to leverage the full benefits of the cloud," explained Alex Brower, VP of Marketing at Cloud Academy, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clar...
The IoT Will Grow: In what might be the most obvious prediction of the decade, the IoT will continue to expand next year, with more and more devices coming online every single day. What isn’t so obvious about this prediction: where that growth will occur. The retail, healthcare, and industrial/supply chain industries will likely see the greatest growth. Forrester Research has predicted the IoT will become “the backbone” of customer value as it continues to grow. It is no surprise that retail is ...