Welcome!

Java Authors: Trevor Parsons, Esmeralda Swartz, Sandi Mappic, Carmen Gonzalez, Pat Romanski

Related Topics: Java

Java: Article

Reader Feedback Special: Java vs C++ Benchmark Brouhaha

Reader Feedback Special: Java vs C++ Benchmark Brouhaha

  • Java vs C++ "Shootout" Revisited
  • Read/Join the Feedback to the original article 

    On Monday we published an article about Keith Lea's one-man attempt to take the benchmark code for C++ and Java from the now outdated Great Computer Language Shootout conducted by Doug Bagley in Fall 2001 and run the tests himself. Within 12 hours 40,000 people had read the piece and by the end of the day Bagley's "Shootout" had even been revived as an ongoing Debian Alioth project.

    But while the team behind that project cheerfully says on its Web site "Contributions are welcomed for revised implementations, bug fixes, or ideas for new benchmarks," not everyone agrees with this kind of activity.

    When Bagley did the original shootout, he said it was "To learn and to have fun," adding that he would continue continue "as long as the fun holds out." This notion of enjoyment has been revived by the Debian Alioth update: 

    "The project goals have not changed substantially since Doug's original project. This work is continuing so that we all can learn about new languages, compare them in various (possibly meaningless) ways, and most importantly, have some fun!"
    But many of those who troubled to record their views in the course of the past few days did not recognize benchmarking/microbenchmarking as anything worth doing at all.

    Some took issue with Lea's choice of performance speed as the parameter to compare Java and C++. One, Lachlan Stuart, wrote:

    "What you must realize is that C++ and Java have much different targets, it's rare that you will encounter a situation where both are equally suitable. No matter what, unless you forget to turn debug compile off, C++ will beat Java in real world situations on the performance front..."

    "Try praising something else about Java...like its security, or ease of programming," Stuart continued.

    GooberToo concurred. "The more I read the code, it sure is starting to look more and more like an apples and oranges comparison; which is usually what happens when people do Java benchmarks." And Iain too was unhappy: "Aside from perhaps demonstrating that you know Java far better than you know C++, I don't think these tests were a particularly useful exercise."

    Others too felt that Lea's relative unfamiliarity with C++ compromised the exercise. "This article is completely useless and only shows the lack of understanding of C++ by the author," wrote Dr Valentin Samko. "This reminds me of a person who clamed that qbasic is faster than C++ because his program which outputs a few million blank lines works faster in qbasic than in C++."

    Glen Low queried the shootout's methodology on a number of counts. Here is one:

    "The small size of the programs are a definite advantage for Java's runtime optimizer. A run-time optimizer can't spend forever optimizing, but it has access to actual run-time data, AND can specialize its optimization for the specific processor. (G++, and most other C++ compilers, have flags to allow this as well.) Given the small size, a complete analysis of the program is possible in the limited time available, and given the other additional data, I would find it rather worrisome if Java didn't do better. I'm more sceptical with regards to real programs, however. If there really is one small critical loop, Java should win; that's not the case for most of my programs, however."
    Scott Ellsworth disagreed with those who dismissed Lea's shootout, though. "The point is that many C++ users feel that Java is too slow and uses too much memory. Microbenchmarks like this show that this off the cuff response is as silly as claiming that exceptions cost too much just because cfront compilers had that cost," he wrote. 

    Ellsworth continued:

    "I spent seven years doing C++ in a shrink wrap environment, and five doing Java. In my experience with real world apps, Java code gets done faster, works better out of the box, and is easier to maintain. Tricks like custom memory allocators and carefully crafted templates can make faster C++ code, but I reserve those for time critical parts of the program. Where I do not spend some programmer cycles profiling and optimizing, the two environments are pretty close.

    Thus - if Java is producing code reasonably close or ahead in fairly straightforward usage, and the code gets written faster and more portably, then Java is a win for my work. I can always choose to use C++, assembly, or FORTRAN for the time critical routines, and then link in."
    Patrik Beno issued a challenge. "I suggest that now it's time for C++ experts to optimize C++ code and compilation, run the very same tests again and post the results."

    "Don't blame Lea, he did his best, now it's your turn, C++ experts," he added.

    Perhaps SWagner would be someone who will rise to the challenge in due course. S/he wasn't at all happy with Lea, writing:

    "As an experienced C++ developer (and Java as well) I only see that the C++ code is optimized for minimum performance. Especially the methodcall where the class has virtual constructors or destructors which is seldom useful and significantly hurts performance. Just because you can't turn this off in Java does not mean that you have to make it slow in C++."
    "In my practical experience," continued Wagner, "normal Java code is 1000 times slower [compared] to OPTIMIZED C++. However if you make severe design flaws you can make any program slow. And if you base on a bad benchmark and make it even worse you should better stop any benchmarking you planned for the future."

    Somewhat less disgruntled was m, who wrote: "The benchmarks are interesting, but frankly with real applications, I have never experienced Java performance equal to or better than that of C++."

    "I have done a number of home-grown benchmarks for XML parsing, and C/C++ (think expat) are 3-9 times faster," m continued. "The difference there is that the server JVM does indeed optimize over a series of many runs. Java certainly allows us to develop faster, and quite often time to market performance is more important than the performance differences in these two (great in my opinion) platforms. They each have specific advantages."

    This, not surprisingly, was the essence of quite a few of the comments. "Java is faster for some applications than C++, and for others C++ is faster. It depends on what you want to do with it," said RebelCool

    But Iain's objection was to the choice of tools to compare, to the entire premise of the shootout in other words: 
    "The comparison [Lea is] trying to make between C++ and Java is not particularly meaningful. At the end of the day, if a VM is faster, then there is nothing to stop you implementing a VM in C++ and running the 'C++ program' within that framework. What times would then be valid? The tools you are comparing are very different.
    In addition, the C++ code is seriously sub-optimal. To take one example, the 'matrix.cpp' is definitely not written for speed. A general purpose C function is being used to repeatedly allocate rows, when a single large block of size 'mn' could have been used. For that matter, why is malloc( ) being used at all? (A custom memory manager optimised for small blocks could have been used instead - malloc( ) is slow.)

    If you insist on writing simple C, then at least use memset( ) for the zeromatrix( ) function. Fast template matrix libraries exist for C++ - these could have been used instead - the result would have been shorter, clearer and considerably faster. Aside from perhaps demonstrating that you know Java far better than you know C++, I don't think these tests were a particularly useful exercise."

    NotZed disagreed with the premise too, writing: "Java works well as a COBOL replacement, a backend application language for boring buisiness apps. That's what it should be benchmarked against. Comparing it to C is like comparing apples and oranges."

    Chris countered: "Surely the benchmark is no less meaningful than when it was originally used to show C++ was faster than Java."  

    And Christopher White commented:

    "Thanks for the comparison Keith. I enjoyed reading your commentary.

    Many people have stated in response to this article that Java, as everyone knows, is slow... This is certainly not the case. I am team leader of a Java development team. We have six applications in production on a certain open source application server. We have 1200 users of these applications. The applications are load-balanced over two machines (each machine : HP DL380 2*2.8GHz 2Gb RAM). We have on average 2% Cpu utilisation. Whilst this is of course just a subjective view of Java performance, I can say that we certainly do not have the impression that Java is at all slow.

    I accept that for memory deallocation C++ can be tailored to the needs of the application at hand so it will be difficult to beat on this score, however one must remember that the just in time compiler compiles code into native machine language and that this compiler is very much optimized on a per platform basis so I don't see why C++ should produce naturally faster compiled code.

    Whichever is faster, Java is easily fast enough for most applications and I strongly believe that the overhead in development time incurred by using C++ is rarely justified except in specialized applications such as driver or game coding."

    For the moment let's give the last word to SB:

    "Java and C++ are different languages, with different advantages/drawbacks, and different philosophies. My experience, from a large company that produces tons of software, is that performance always comes down to the quality of the developers, not the language itself. And we also found out that in today's world, most developers are only average, and this class of developers yields better results with Java than C++."
    The debate, as they say, will doubtless continue...

  • More Stories By Java News Desk

    JDJ News Desk monitors the world of Java to present IT professionals with updates on technology advances, business trends, new products and standards in the Java and i-technology space.

    Comments (3)

    Share your thoughts on this story.

    Add your comment
    You must be signed in to add a comment. Sign-in | Register

    In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


    @ThingsExpo Stories
    Samsung VP Jacopo Lenzi, who headed the company's recent SmartThings acquisition under the auspices of Samsung's Open Innovaction Center (OIC), answered a few questions we had about the deal. This interview was in conjunction with our interview with SmartThings CEO Alex Hawkinson. IoT Journal: SmartThings was developed in an open, standards-agnostic platform, and will now be part of Samsung's Open Innovation Center. Can you elaborate on your commitment to keep the platform open? Jacopo Lenzi: Samsung recognizes that true, accelerated innovation cannot be driven from one source, but requires a...
    SYS-CON Events announced today that Red Hat, the world's leading provider of open source solutions, will exhibit at Internet of @ThingsExpo, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Red Hat is the world's leading provider of open source software solutions, using a community-powered approach to reliable and high-performing cloud, Linux, middleware, storage and virtualization technologies. Red Hat also offers award-winning support, training, and consulting services. As the connective hub in a global network of enterprises, partners, a...
    P2P RTC will impact the landscape of communications, shifting from traditional telephony style communications models to OTT (Over-The-Top) cloud assisted & PaaS (Platform as a Service) communication services. The P2P shift will impact many areas of our lives, from mobile communication, human interactive web services, RTC and telephony infrastructure, user federation, security and privacy implications, business costs, and scalability. In his session at Internet of @ThingsExpo, Robin Raymond, Chief Architect at Hookflash Inc., will walk through the shifting landscape of traditional telephone a...
    SYS-CON Events announced today that Matrix.org has been named “Silver Sponsor” of Internet of @ThingsExpo, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Matrix is an ambitious new open standard for open, distributed, real-time communication over IP. It defines a new approach for interoperable Instant Messaging and VoIP based on pragmatic HTTP APIs and WebRTC, and provides open source reference implementations to showcase and bootstrap the new standard. Our focus is on simplicity, security, and supporting the fullest feature set.
    BSQUARE is a global leader of embedded software solutions. We enable smart connected systems at the device level and beyond that millions use every day and provide actionable data solutions for the growing Internet of Things (IoT) market. We empower our world-class customers with our products, services and solutions to achieve innovation and success. For more information, visit www.bsquare.com.
    How do APIs and IoT relate? The answer is not as simple as merely adding an API on top of a dumb device, but rather about understanding the architectural patterns for implementing an IoT fabric. There are typically two or three trends: Exposing the device to a management framework Exposing that management framework to a business centric logic • Exposing that business layer and data to end users. This last trend is the IoT stack, which involves a new shift in the separation of what stuff happens, where data lives and where the interface lies. For instance, it’s a mix of architectural style...
    SYS-CON Events announced today that SOA Software, an API management leader, will exhibit at SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. SOA Software is a leading provider of API Management and SOA Governance products that equip business to deliver APIs and SOA together to drive their company to meet its business strategy quickly and effectively. SOA Software’s technology helps businesses to accelerate their digital channels with APIs, drive partner adoption, monetize their assets, and achieve a...
    From a software development perspective IoT is about programming "things," about connecting them with each other or integrating them with existing applications. In his session at @ThingsExpo, Yakov Fain, co-founder of Farata Systems and SuranceBay, will show you how small IoT-enabled devices from multiple manufacturers can be integrated into the workflow of an enterprise application. This is a practical demo of building a framework and components in HTML/Java/Mobile technologies to serve as a platform that can integrate new devices as they become available on the market.
    SYS-CON Events announced today that Utimaco will exhibit at SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Utimaco is a leading manufacturer of hardware based security solutions that provide the root of trust to keep cryptographic keys safe, secure critical digital infrastructures and protect high value data assets. Only Utimaco delivers a general-purpose hardware security module (HSM) as a customizable platform to easily integrate into existing software solutions, embed business logic and build s...
    Connected devices are changing the way we go about our everyday life, from wearables to driverless cars, to smart grids and entire industries revolutionizing business opportunities through smart objects, capable of two-way communication. But what happens when objects are given an IP-address, and we rely on that connection, sometimes with our lives? How do we secure those vast data infrastructures and safe-keep the privacy of sensitive information? This session will outline how each and every connected device can uphold a core root of trust via a unique cryptographic signature – a “bir...
    Internet of @ThingsExpo Silicon Valley announced on Thursday its first 12 all-star speakers and sessions for its upcoming event, which will take place November 4-6, 2014, at the Santa Clara Convention Center in California. @ThingsExpo, the first and largest IoT event in the world, debuted at the Javits Center in New York City in June 10-12, 2014 with over 6,000 delegates attending the conference. Among the first 12 announced world class speakers, IBM will present two highly popular IoT sessions, which will take place November 4-6, 2014 at the Santa Clara Convention Center in Santa Clara, Calif...
    Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
    WebRTC defines no default signaling protocol, causing fragmentation between WebRTC silos. SIP and XMPP provide possibilities, but come with considerable complexity and are not designed for use in a web environment. In his session at Internet of @ThingsExpo, Matthew Hodgson, technical co-founder of the Matrix.org, will discuss how Matrix is a new non-profit Open Source Project that defines both a new HTTP-based standard for VoIP & IM signaling and provides reference implementations.

    SUNNYVALE, Calif., Oct. 20, 2014 /PRNewswire/ -- Spansion Inc. (NYSE: CODE), a global leader in embedded systems, today added 96 new products to the Spansion® FM4 Family of flexible microcontrollers (MCUs). Based on the ARM® Cortex®-M4F core, the new MCUs boast a 200 MHz operating frequency and support a diverse set of on-chip peripherals for enhanced human machine interfaces (HMIs) and machine-to-machine (M2M) communications. The rich set of periphera...

    SYS-CON Events announced today that Aria Systems, the recurring revenue expert, has been named "Bronze Sponsor" of SYS-CON's 15th International Cloud Expo®, which will take place on November 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Aria Systems helps leading businesses connect their customers with the products and services they love. Industry leaders like Pitney Bowes, Experian, AAA NCNU, VMware, HootSuite and many others choose Aria to power their recurring revenue business and deliver exceptional experiences to their customers.
    The Internet of Things (IoT) is going to require a new way of thinking and of developing software for speed, security and innovation. This requires IT leaders to balance business as usual while anticipating for the next market and technology trends. Cloud provides the right IT asset portfolio to help today’s IT leaders manage the old and prepare for the new. Today the cloud conversation is evolving from private and public to hybrid. This session will provide use cases and insights to reinforce the value of the network in helping organizations to maximize their company’s cloud experience.
    The Internet of Things (IoT) is making everything it touches smarter – smart devices, smart cars and smart cities. And lucky us, we’re just beginning to reap the benefits as we work toward a networked society. However, this technology-driven innovation is impacting more than just individuals. The IoT has an environmental impact as well, which brings us to the theme of this month’s #IoTuesday Twitter chat. The ability to remove inefficiencies through connected objects is driving change throughout every sector, including waste management. BigBelly Solar, located just outside of Boston, is trans...
    SYS-CON Events announced today that Matrix.org has been named “Silver Sponsor” of Internet of @ThingsExpo, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Matrix is an ambitious new open standard for open, distributed, real-time communication over IP. It defines a new approach for interoperable Instant Messaging and VoIP based on pragmatic HTTP APIs and WebRTC, and provides open source reference implementations to showcase and bootstrap the new standard. Our focus is on simplicity, security, and supporting the fullest feature set.
    Predicted by Gartner to add $1.9 trillion to the global economy by 2020, the Internet of Everything (IoE) is based on the idea that devices, systems and services will connect in simple, transparent ways, enabling seamless interactions among devices across brands and sectors. As this vision unfolds, it is clear that no single company can accomplish the level of interoperability required to support the horizontal aspects of the IoE. The AllSeen Alliance, announced in December 2013, was formed with the goal to advance IoE adoption and innovation in the connected home, healthcare, education, aut...
    SYS-CON Events announced today that Red Hat, the world's leading provider of open source solutions, will exhibit at Internet of @ThingsExpo, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Red Hat is the world's leading provider of open source software solutions, using a community-powered approach to reliable and high-performing cloud, Linux, middleware, storage and virtualization technologies. Red Hat also offers award-winning support, training, and consulting services. As the connective hub in a global network of enterprises, partners, a...