Welcome!

Java Authors: Unitiv Blog, Liz McMillan, Dana Gardner, Roger Strukhoff, Kevin Benedict

Related Topics: Java, Websphere, Weblogic

Java: Article

Java vs C++ "Shootout" Revisited

Java vs C++ "Shootout" Revisited

Keith Lea writes of the benchmark, on his results page, "I was sick of hearing people say Java was slow, when I know it's pretty fast, so I took the benchmark code for C++ and Java from the now outdated Great Computer Language Shootout and ran the tests myself."

Lea used G++ (GCC) 3.3.1 20030930 (with glibc 2.3.2-98) for the C++, with the -O2 flag (for both i386 and i686). He compiled the Java code normally with the Sun Java 1.4.2_01 compiler, and ran it with the Sun 1.4.2_01 JVM. He ran the tests on Red Hat Linux 9 / Fedora Test1 with the 2.4.20-20.9 kernel on a T30 laptop. The laptop "has a Pentium 4 mobile chip, 512MB of memory, a sort of slow disk," he notes.

The results he got were that Java is significantly faster than optimized C++ in many cases.

"They also show that no one should ever run the client JVM when given the choice," Lea adds. ("Everyone has the choice," he says. To run the server VM, see instructions in the Using the Server JVM section below.)

JDJ has agreed to post online anyone else's results as long as they use Java 1.4.2 or higher and any version of GCC that produces faster or equivalent code than the 3.3.1 I used. We encourage you to download the source and/or the binaries and perform the tests yourself, with your favorite compiler and on your favorite platform.


Lea's Data and Results

JVM startup time was included in these results. "That means even with JVM startup time, Java is still faster than C++ in many of these tests," says Lea.

Some of the C++ tests would not compile. "I've never been very good at decoding GCC's error messages," he admits, "so if I couldn't fix a test with a trivial modification, I didn't include it in my benchmarks."

Lea also modified one of the tests, the string concatenation test for Java.

"The test was creating a new StringBuffer in each iteration of the loop, which was just silly," he explains. "I updated the code to use a single StringBuffer and appending to it inside the loop."

(The updated tests at the original shootout use this new method.)

"Java lost this benchmark even with the modifications," Lea declares. "So if anyone wants to accuse me of biasing the results, they're going to have to try harder."

Several versions of some of the C++ tests (like matrix) were present in the original shootout source, he continues. 

"I used the versions without numbers in them, like matrix.g++ instead of matrix.g++2. I don't know which of these were used in the original benchmarks, but from my quick experimenting, the numberless ones generally ran faster than their numbered counterparts."

"Looking at them again," Lea says, "matrix.g++3 runs faster than the matrix.g++ that I use. However, it still runs slower than the Java version, so I don't plan to modify the graph/data unless someone asks me to, since getting that graph in the first place was sort of a pain.)"

He continues: "I've been told that the C++ code for the Method Call benchmark returns by value while the Java code returns by reference, and that modifying the C++ code to pass a pointer makes that benchmark faster. However, even with the modification, the C++ version still runs slower than the Java version."

Lea ran th Java and the C++ tests to "warm up" (both the Java and C++ tests got faster after he ran them a few times).

"I've been told that these tests are invalid because they were run with GCC," he concedes, adding: "I have seen both benchmarks that show GCC producing faster code than Visual Studio's VC++ compiler, and benchmarks showing the opposite. If I update the benchmarks with another compiler added, it will be the Intel C++ Compiler, which I'm pretty sure produces faster code than VC++."

Lea says he's been accused of biasing the results by using the -O2 option for GCC, "supposedly because -O2 optimizes for space, thus slowing down the benchmark," he explains.

But this is not what -O2 does, he points out, referring to the GCC -O documentation:

JVM startup time was included in these results. "That means even with JVM startup time, Java is still faster than C++ in many of these tests," says Lea.

Some of the C++ tests would not compile. "I've never been very good at decoding GCC's error messages," he admits, "so if I couldn't fix a test with a trivial modification, I didn't include it in my benchmarks."

Lea also modified one of the tests, the string concatenation test for Java.

"The test was creating a new StringBuffer in each iteration of the loop, which was just silly," he explains. "I updated the code to use a single StringBuffer and appending to it inside the loop."

(The updated tests at the original shootout use this new method.)

"Java lost this benchmark even with the modifications," Lea declares. "So if anyone wants to accuse me of biasing the results, they're going to have to try harder."

Several versions of some of the C++ tests (like matrix) were present in the original shootout source, he continues. 

"I used the versions without numbers in them, like matrix.g++ instead of matrix.g++2. I don't know which of these were used in the original benchmarks, but from my quick experimenting, the numberless ones generally ran faster than their numbered counterparts."

"Looking at them again," Lea says, "matrix.g++3 runs faster than the matrix.g++ that I use. However, it still runs slower than the Java version, so I don't plan to modify the graph/data unless someone asks me to, since getting that graph in the first place was sort of a pain.)"

He continues: "I've been told that the C++ code for the Method Call benchmark returns by value while the Java code returns by reference, and that modifying the C++ code to pass a pointer makes that benchmark faster. However, even with the modification, the C++ version still runs slower than the Java version."

Lea ran the tests many times before running the "official" recorded set of tests, so there was plenty of time for both Java and the C++ tests to "warm up" (both the Java and C++ tests got faster after he ran them a few times).

"I've been told that these tests are invalid because they were run with GCC," he concedes, adding: "I have seen both benchmarks that show GCC producing faster code than Visual Studio's VC++ compiler, and benchmarks showing the opposite. If I update the benchmarks with another compiler added, it will be the Intel C++ Compiler, which I'm pretty sure produces faster code than VC++."

Lea says he's been accused of biasing the results by using the -O2 option for GCC, "supposedly because -O2 optimizes for space, thus slowing down the benchmark," he explains.

But this is not what -O2 does, he points out, referring to the GCC -O documentation:

-O2: Optimize even more. GCC performs nearly all supported optimizations that do not involve a space-speed tradeoff. The compiler does not perform loop unrolling or function inlining when you specify -O2. As compared to -O, this option increases both compilation time and the performance of the generated code.

"On the other hand, -O3 performs space-speed tradeoffs, and -O performs fewer optimizations. Thus, for these tests, I think O2 was the best choice," Lea concludes.

 

"I don't have an automated means of building and benchmarking these things (and the scripts that came with the original shootout didn't run for me)," he continues. "I really do want people to test it on their own machines, but it's going to take some work, I guess."

Lea compiled the C++ code with:

g++ [test].cpp -O2 -march=i386 -o [test]-386

g++ [test].cpp -O2 -march=i686 -o [test]-686

and the Java code with:

javac [test].java

To see how he ran the binaries, see the run log. You can download the source code he used in either .bz2 or .zip format.

Using the Server JVM

Every form of Sun's Java runtime comes with both the "client VM" and the "server VM."

"Unfortunately, Java applications and applets run by default in the client VM," Lea observes. "The Server VM is much faster than the Client VM, but it has the downside of taking around 10% longer to start up, and it uses more memory."

Lea explains the two ways to run Java applications with the server VM as follows

  1. When launching a Java application from the command line, use java -server [arguments...] instead of java [arguments...]. For example, use java -server -jar beanshell.jar.
  2. Modify the jvm.cfg file in your Java installation. (It's a text file, so you can use Notepad or Emacs to edit it.) This is located in C:\Program Files\Java\j2reXXX\lib\i386\ on Windows, /usr/java/j2reXXX/lib/i386/ on Linux. You will see two lines:
    -client KNOWN
    -server KNOWN
    You should change them to:
    -server KNOWN
    -client KNOWN
    This change will cause the server VM to be run for all applications, unless they are run with the -client argument.

He can be contacted at

Every form of Sun's Java runtime comes with both the "client VM" and the "server VM."

"Unfortunately, Java applications and applets run by default in the client VM," Lea observes. "The Server VM is much faster than the Client VM, but it has the downside of taking around 10% longer to start up, and it uses more memory."

Lea explains the two ways to run Java applications with the server VM as follows

  1. When launching a Java application from the command line, use java -server [arguments...] instead of java [arguments...]. For example, use java -server -jar beanshell.jar.
  2. Modify the jvm.cfg file in your Java installation. (It's a text file, so you can use Notepad or Emacs to edit it.) This is located in C:\Program Files\Java\j2reXXX\lib\i386\ on Windows, /usr/java/j2reXXX/lib/i386/ on Linux. You will see two lines:
    -client KNOWN
    -server KNOWN
    You should change them to:
    -server KNOWN
    -client KNOWN
    This change will cause the server VM to be run for all applications, unless they are run with the -client argument.

He can be contacted at [email protected].

Links

More Stories By Jeremy Geelan

Jeremy Geelan is Chairman & CEO of the 21st Century Internet Group, Inc. and an Executive Academy Member of the International Academy of Digital Arts & Sciences. Formerly he was President & COO at Cloud Expo, Inc. and Conference Chair of the worldwide Cloud Expo series. He appears regularly at conferences and trade shows, speaking to technology audiences across six continents. You can follow him on twitter: @jg21.

Comments (152) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Dave Anderson 06/20/04 01:15:39 PM EDT

Keith Lea just shoed everyone his total lack of knowledge of C++ and clear biased.

After reviewing the hash, fairly simple and mostly a test of memory allocation speed it clearly shows the C++ is inefficient and the Java is efficient. I tried this on Solaris. With a few comparable adjustments I got the C++ version running more than 8 times faster.

He did manage to prove one thing, C++ runs inefficient code faster than Java runs optimized. But it does take away from this sources credibility.

MamanE 06/20/04 12:50:23 PM EDT

I was referring to jos. Yes, it''s way from mature to the point that it isn''t really useful yet but that''s not the point. The point is that you can do it.
Didn''t know about a lego java os :-) but it seems interesting.

Regarding your point about a gc for C, implementing it in the benchmarks would still make for a fairer comparison regardless of whether a GC is used in C much or not. If not, deleting the objects in the C sources is the only correct thing to do. In these particular benchmarks this may give java an unfair advantage (and BTW that updated C++ benchmark where objects aren''t deleted is completely unfair too), which tells me this whole microbenchmark is almost useless. It''s just ''Yet Another Useless Microbenchmark''.

jose 06/20/04 06:44:26 AM EDT

as far as im aware, there isn''t... Beos wasnt written in java, jos & and jnode are crap, sun java desktop is linux under the hood, Solaris is most definately not Java, as it licences lots of Unix code, however Java OS for Lego looks kind cool...

what OS are you referring to??? :\

anyways, back to what other things you were saying, there are actually quite a lot of C/C++ garbage collectors out there written by different people, but its not used because it, to an extent like the gc in java, hogs resources you would otherwise want to use on something else.

i miss having the option of accessing pointers directly, rather than being forbidden

MamanE 06/20/04 06:28:25 AM EDT

" A Java VM will be faster than a C++ VM !!!"

You make no sense. It''s not the language that is (supposedly) faster, it''s run time compilation that can be faster than ahead of time compilation.

MamanE 06/20/04 06:23:15 AM EDT

" There are lots of things you still can''''t do with Java and lots of things Java can do, but it would be too awkward. Operating system code, drivers, video games, earth simulators, etc etc..."

Well, there are things C++ are more *suitable* for, but I''d like to point out that there *is* an OS written in java, complete with drivers and there are plenty of great games written in java. For example I''ve seen a quake3 clone which runs >400fps on my laptop (which is not even very suitable for games in the first place).

That said, I think the benchmark doesn''t prove much for real life programs. In the end of the day, it''s yet another microbenchmark, and an apples-oranges comparison.
I''ve seen people complain about that the C++ programs should not delete objects because the java one doesn''t too, well, maybe the C programs should have implemented a C garbage collector to make the comparison fairer than. But that wouldn''t make any sense in these little programs that basically do nothing, would it?

My personal experience is that I can get the job done with java far more quicker than I can with C or C++ with less bugs. Especially since I have to develop for multiple platforms a lot. I never had any performance issues either that were JVM related and not errors on my part, so I''ll stick with java for a while.

TheSpider 06/19/04 05:01:03 PM EDT

Larry,

When speaking of architectural issues you cover an immense amount of ground. Would you please give some examples of these that have created serious performance issues? It would also be great if you would refer to some good material that highlights the better architectural concepts that you encourage use of.

Daniele Paccaloni 06/19/04 12:51:11 PM EDT

If Sun really believes that Java is faster than C++, then I expect that the next version of the VM/Hotspot to be entirely developed in Java apart from the assembly optimizations (because yet Sun has not claimed Java to be faster than Asm.. but some of their benchmarks may clearly demonstrate that it is !).
This will allow for even faster Java performance ! A Java VM will be faster than a C++ VM !!!
Also, I expect to see no more JNI calls in the VM apart from the OS interface, you don''t want the VM to call slower C++ code.
It is also clear that the new Java-coded VM/Hotspot could even be run on a Server-VM for overkill performance !
I can afford to buy an extra gigabyte of RAM to run at light-speed on a Server-VM, so be it !

...but then I wake up, go to work, and run my C++ compiler.

Zandro 06/19/04 12:17:41 AM EDT

One more note....O2 is a STUPID choice for this test. The hilarious part is the doc is quoted to support the decision. You should have arrived at exactly the opporsite decision. Turn up the optimizer and let the inlining and unrolling occur. This is exactly what happens in the JIT/HotSpot compiler in the JVM.

larry 06/19/04 12:16:28 AM EDT

Zandro, there is no guarantee that compiled java code would be faster. I covered why earlier on in the thread. Read back about 10 messages.

larry 06/19/04 12:14:09 AM EDT

Spider

In some of the work I and others have done in consulting, we found that a lot of performance issues stemmed from architectural difficulties. The larger the architecture, the more careful one needed to be in managing the design. I found that following established architecural patterns helped.

Zandro 06/19/04 12:11:10 AM EDT

Wow, were you out of material to fill this issue?

Java is a good and usable language. For many benchmarks and real world applications, it compares well to C++. However, this ''test'' only proves this is not a benchmark and GCC is not the best optimizing compiler for C++. If you ran these same silly codes with compiled Java, you would see addition ''gains'' for the compiled Java code.

If there is going to be a benchmark, at least pick some standard tests and none trial code examples.

larry 06/19/04 12:06:24 AM EDT

Jose, you miss a lot in java if your comparison goal is comparing the procedural nature of it to the procedural nature of C and to a lesser extent C++. Where Java gets a lot of expressive power is in it OO expression. Take a look at the design patterns movement and the various idioms there. You can express very complex things simply.

Duff''s Device is actually mentioned in the java Language specification. But the position of the do while will cause a syntax error in java. You can''t express pointers directly either, But you can do this sort of thing with the new I/IO (java.nio) package.

There are some discussions point mentioning the enfant terrible school of prgramming practice at: http://www.math.grin.edu/~kensler/terriblec.html

There is also a note on whether the time for Duff''s device is still valid:
http://groups.yahoo.com/group/advanced-java/message/40772

Scott Ellsworth 06/18/04 10:51:41 PM EDT

Jose:

Gah! Just because you can do something does not mean you should. I would far rather focus on writing better programs, that work faster and do more without an increased maintenance burden than little hacks that I cannot use again. Put another way - if a left shift is substantially faster than a multiply, then I would rather the compiler put that in, as then my code is a very readable x *= 2 rather than the slightly more cryptic x<<2.

Languages should, in my opinion, make easy things easy and hard things possible. Thus far, Java does that better than any other language I have used (Basic, Pascal, C, C++, Lisp, Perl, Python, Bash and tcsh scripting, to name just languages I got paid to program in). I do still write shell scripts, I do write the odd C++ program, and I did spend six months implementing a very complicated biotech app in Perl, but the usual tool for a new problem is a Java program, unless it is particularly well suited to another language or tool.

Put another way, there is always room to improve a language, and your own command of it. I would rather learn to do things better in a way that can be maintained, than better in a way that just makes them cryptic. That was just as true when I was doing C++ as it is now that I am doing Java, and I expect I will maintain that point of view with whatever I am using five years from now. That is not likely to be Java, but we shall see.

Scott

José 06/18/04 09:51:54 PM EDT

meh... I mean, here''s another thing. Java is pretty nice how everything is structured, no-nonsense and simple once you get the basics, but it really doesn''t leave much room for imagination in many ways. Now before you start planning to rebuke this comment, just give it a thought. Is it possible to implement anything like Duff''s Device in Java?

In the specific case of Duff''s Device, it isn''t... and most guru''s would call it a dirty trick (some Java programmers even mock the fact that it''s possible in C++ and that Java is more proper that it doesn''t allow such things), but it made me respect C++ a lot. That standards were so constructed that you could even play dirty tricks. I like to feel that when I''m programming, there''s still a lot of space to improve... even to invent something that perhaps no one had thought of before. What not. I hope you get what I mean. What do you think?

SB 06/18/04 05:53:55 PM EDT

You can get a benchmark to say whatever you want, which is another way of saying that you start with a conclusion and find the benchmark that supports your claims. Hence the futility of this discussion.
Java and C++ are different languages, with different advantages/drawbacks, and different philosophies. My experience, from a large company that produces tons of software, is that performance always comes down to the quality of the developers, not the language itself. And we also found out that in todays world, most developers are only average, and this class of developers yields better results with Java than C++. The tails of the distribution are statistically irrelevant.
Go back to your keyboards and work !!!

appsman 06/18/04 05:45:53 PM EDT

The user interface side of Java just plain sucks. It''s slow, bad looking, longer to program than everything but assembler and unpredictable. What''s not to like. Even the best libraries out there are awful and it takes pages and pages of code to do simple things like synchronized, multi-column on-screen arrays. I''ve yet to see a real world Java app with a human usable face.

TheSpider 06/18/04 05:30:27 PM EDT

Larry,

Your comments are an excellent counterweight to the rabid rants of those who look down their knowses at Java. You would think they could balance their justified criticism of the tested C++ code examples with some reasoning as to the rather excellent attributes of Java. Java hasn''t become an essential part of systems level application programming for no reason. It''s just plain better for server centric environments (in it''s current form). This is the sort of world where the code has to actually work and getting it both conceptually right and effectively implemented in Java is 10-100 times easier than in C++.

The main issue I have with Java is how the really "nice" things that would allow us to do things at higher levels of abstraction just kill the performance. There appear to be so many things you should NOT do in Java if you want to retain reasonable performance that it makes it much harder to choose this language/environment vs. a Delphi, VB or other more "predictable" toolset.

If you have the time and inclination, I can give some examples of what I''d like to be able to do in Java that we tried once but eventually dropped (5 years ago). It''s in the boring field of business apps and I''m just itching to convert my 1 million plus lines of 4GL code to something modern, portable and extensible.

José 06/18/04 02:55:16 PM EDT

And I say, fair enough. I''ve worked on projects where Java was most suited. In many cases, because it was the most supported environment, but in others because it was the better option. However, there are a few key areas that I''m unimpressed with, as I have obviously expressed and thats why I''m moaning. Speed being one of them. I do also have some good things to say about Java and at the end of the day, what ever pays is what I''ll end up working with. Thats all I have left to say.

Have fun.

larry 06/18/04 02:38:23 PM EDT

Jose, if you feel comfortable in C++, then good for you. But you have offered your opinion which is your opinion based on your experiences.

I work both in Java and C/C++. I have been a hardcore programmer and architect for 18 years in a large variety of industries.

However I have built real time VM''s (in C++), voice recognition systems, mobility architectures, massively parallel grid based architectures, all on Java. I never felt limited by the language or it''s performance. If I found a performance issue, it was largely my fault.

I have yet to find an application domain where i can''t use it.

Sun is not the only company building VMs out ther. IBM and JRocket build excellent fast VMs. If sun was to disappear tomorrow, then Java would continue to evolve in whatever it evolves.

And that is my opinion also, take it an alternate view. ;)

José 06/18/04 02:17:45 PM EDT

Scott Cotsman, you don''t need to be so anal. In reality, C++ is still better for most things and thus generally preferred. There are lots of things you still can''t do with Java and lots of things Java can do, but it would be too awkward. Operating system code, drivers, video games, earth simulators, etc etc... At the end of the day, people who write software are geeks and what the geeks choose to develop with isn''t some idealogical reason, but for practical reasons and I suspect that most geeks want to keep some level of closeness to the underlying system and historical value. Take the success of Linux, for example...

With all the benefits that Java provides, most geeks already are skilled enough to have their own methods and libraries for RAD. And on a level of eliteness, Assembly is an example of really elite, C++ sits comfortably high, and Java is considered quite good and useful, but geeks do tend to avoid it. Geeks still rule the IT world and C++ is not archaic, that''s what Java guru''s have been trying to say since god-knows-when. And a poorly argued article, which aims at attacking C++ should be put in its place. After all, if the article was about how VB is faster than Java, wouldn''t you think its fair to dispute?

Personally, I wonder how well Java will fare in the next 3-4 years, for example when Longhorn comes around and whether Sun will be able to sustain itself on a good business model as opposed to this moment. And I wonder about .Net once MS get it sorted out. IMO, if C++ really is getting old, then Java won''t be the successor, but something else. Or maybe it''ll always be as fragmented as it is right now. Who knows. I, for one, prefer C++ over Java on all but my laziest days. Maybe the best thing Sun could ever do is to GPL Java?

larry 06/18/04 01:55:32 PM EDT

Eric, it isn''t P-code. I think you are confusing much older VM technologies with the current state of the art.

Bytecode is converted by the interpreted into an machine code equivalent for each bytecode present. This code is run on the processor. This is still considered interpreted mode. There is no translation on the fly as each bytecode operand is executed.

What the compiler does is scan through this machine code based on the profiling result and optimize it. It retains the original bytecode elements that were present in case it needs to go back to the "intepreted" mode (i.e. still machine but expanded as a bytecode set). So basically, machine code is always executed. The interpreted version will be not be optimal but it can become so based on where the hotspot is. That is becuase the VM internal profiler does know what is slow (flagged for compiling) and what is not (does not need compiling).

Jean-Simon LaRochelle 06/18/04 01:40:59 PM EDT

It is fascinating to see how comparing programming languages always turns emotional. Most of the comments that I read (I didn?t read all of them) were very subjective and sometimes contains incorrect information. It will be difficult for anyone to get a clear feeling about Java performance from such article (specially if you read the comments). The best way to handle this is to check if Java does the job for the kind of thing your company is doing. For example, in our case one key function that a programming language has to perform very quickly is an FFT. Before switching to Java as our main language we did a bunch of benchmark and one thing we did was to compare our highly optimized assembly language version (still the fastest at the time but in the process of being phased out of production code because of possible maintenance problem), our C version of the FFT (Borland C++ version 5 at the time) and a new Java implementation.
We found the Java version to be quite good and decided that switching to Java was a good decision for us (based on other factors such as maintenance cost, portability, etc?). Of course the Java FFT avoided such obvious mistake as using multidimensional Java array for matrix (a matrix is stored in a dimension 1 array: float[]).

Eric Ma 06/18/04 01:40:49 PM EDT

Do you believe this guy? Java P-code has to be compiled to machine code, do you think it''s more efficient than compiled C code(looks like this guy though C as C++, so let''s use C as example), which is almost as efficient as ASM.

Java runtime compiles p-code into machine code and cache, if cache always hit, then it''s almost equal to C code; but if cache not hit, then what?

Take the "method call" benchmark as an example, java inlines all virtual calls(per jvm spec), therefore it''s better than a naive C++ virtual call. I haven''t read this guy''s code, but if it''s just C code, that willnot be understandable that the perf is do different.

Scott Cotsman 06/18/04 12:24:12 PM EDT

I can''t believe all these geeks here defending the honor of their beloved C++. I didn''t know people still used that archaic language. Most of the time it''s speed of development that''s the prime concern, and Java wins hands down in that battle.

larry 06/18/04 12:22:34 PM EDT

To be fair on the other side, no one would ever suggest more than a few instances of a JVM on a single machine. If someone suggested that they were running 100 VMs on a server I would laugh (for 2 hours at least) and suggest they probably do not know what they are doing.

The only reason to have more than one instance is to ensure the redundancy of a server side application. I and many others have written component based systems where each app is a loadable component in a container. While J2EE is in wide usage, there are other containers that are available that do not have many of the madening limitations that J2EE has present. Many of these systems have used one or 2 instances of a JVM and have run very high performing systems.

There is no point is writing java like C/C++ when that only leads to poorer performance in the java based app. This is the singular aspect of poorly performing java programs that I come across.

The only way to do a true bakeoff is to have a spec that two teams write too. A group that knows C++ and a group that knows Java. And have a 3rd group that analyzes both parties to ensure that that cheating is minimalized.

ArtyChokes 06/18/04 11:40:31 AM EDT

Sorry, all you''ve done is write C++ code like a Java programmer would. NO self-respecting C++ programmer would write an inherited class structure like you did in your methcall comparison when you could have written a static (global) counter and kept an individual counter inside each base class instance, no multiple constructor/destructor calls needed at all. This one is apples and oranges -- I stop looking right here, assuming the rest of the C++ benchmarks are written by the equally clueless.

And besides, who''s running JRE 1.4.2 anyway? Most machines I''m stuck on is at 1.4.1 and lower. Furthermore, the current g++ release is 3.4.0. To be fair, you should probably have tested JRE 1.4.1 vs g++ 3.3.1.

Get with it. This is no benchmark -- it''s a hatchet job worthy of Redmond that sets out to prove the physically impossible - that you can load a server-side JVM **and** run its application faster than a ** well-written ** ELF binary. Give it up. You might fool some of the people some of the time, but you''re not fooling me. If my server could tolerate 100 concurrent instances of a 16 MB JVM environment in JRE, I might fall for this, too, but it cannot, and hence I cannot, either -- and really, neither can the rest of the world; hence client side JVM is *really* what needs to be tested.

I don''t mind comparitive benchmarks: just be FAIR and real-world.

Java (as a language) has many strengths and weaknesses. I like Java for some purposes and wouldn''t recommend it for others. C++ as a language has many strengths and weaknesses. I would recommend it for some purposes and not others. Not every problem is most efficiently approached with an OO design/implementation; ruling out BOTH Java and C++ in many instances.

But I can almost never -- unless on the client side -- recommend that a standard JVM be run as a platform to interpret your bytecode when performance matters; and only then because it was delivered as bytecode for portability''s sake -- I see JVMs pegging the CPU on the server side all the time, never mind the extra 8-10 MB penalty you pay for the JVM alone _per_instance_. Compiled/linked Java is a beast of an entirely different color -- if you take out the JVM, **NOW** you''ve got a performance shootout. To do anything else isn''t fair to Java (the language). To use the C++ code you did isn''t fair to C++; unless you want to preface your remarks with "benchmarks assuming novice-level stupidity".

Sorry. I have real users to look after.

Dale Cosgro 06/18/04 11:18:11 AM EDT

If you are looking for faster Java code execution - you might try BEA''s JRockit. I was surprised to see that even Oracle (BEA''s competitor) used JRockit in SPECjAppServer benchmark results...very impressive.

larry 06/18/04 09:42:48 AM EDT

JavaWatcher,

I think comaprisons are good. If java fares poorly, at least there is place to improve. Each new release of Java has improved to the point where 1.4.2 is probably 500+ times faster than 1.0.2. There are new runtime optimizers on the way that should accelerate this even further.

larry 06/18/04 09:39:08 AM EDT

DV,

Yes, I do anyway (and was involved in comparisons), but there have been significant attempts to render bytecode into native code through the usage of TowerJ and other others.

In some cases the the performance of the native code is better but in others the VM still gets better numbers. This is primarily because of the runime optimizer. There are cases where a runtime optimizer can achieve better performance because it knows where the code hotspots are. A static optimization system cannot (unless clairvoyance is used) know where these will be. Even a usage change can exercise and stress different areas. While some hotspots will remain the same, more complex code structures will have many hotspots and there will not all be exercised at once.

Jim,

If you are doing J2ME, the compiler mechanisms for this form of java are still relatively unused in commercial applications. I would not expect comparisons to yield much surprise in this area.

If, however, you are doing J2SE or EE, then you definitely need to higher better java people, or a tuning expert like some of my colleagues, because you aren''t getting good value.

DV 06/18/04 09:19:52 AM EDT

Did you know that gcc can compile Java code? It can output byte code, OR an executable that does not require a JVM to run! I''d be interested to see a gcc-based comparison of c++ vs Java that''s compiled all the way down.

Daniele Paccaloni 06/18/04 09:05:45 AM EDT

Hi all,

1) The original shootout did include memory usage ratings, but I can''t see any mem ratings in this one.
I guess this is where Java is *quite* a lot behind C++. I believe that mem usage should be included in order to provide a complete evaluation.

2) All tests run only for a few seconds and the Server JVM could be so fast because of the "lazy" garbage collection policy. I think the tests should last at least a few minutes in order to provide the real GC overhead.

3) The hash map test is quite deceiving. Here you are comparing two hash map implementations, not the same code compiled in Java or C++. As you know, there are many hash implementations faster than the one in stdlib (and of course faster than the one in Java :).

4) I cannot see any real "numeric intensive" benchmarks. Also, I wonder what will be the heapsort benchmark result without the trick of "adjusting" the NumberFormat.

Please note that I love Java. It is just that I don''t get why people struggle for which is the best language: they''re just different and meant for different kind of applications.
Take it easy.

JavaWatcher 06/18/04 07:01:21 AM EDT

Funnily enough though this discussion thread has already been widely replicated and expanded, both at Slashdot and TSS as well as on the comp.lang.c++.moderated Google group - and I believe a group of developers has even decided to revive Doug Bagley''s famous Great Computer Language Shootout on which this latest Jave vs C++ exercise was based and update it with the latest versions of each compiler and interpreter available on the Debian distribution.

A wiki has been set up so that people can help improve the shootout, discuss the implementations of the programs, and suggest optimizations.

The group says:

The project goals have not changed substantially since Doug''s original project. This work is continuing so that we all can learn about new languages, compare them in various (possibly meaningless) ways, and most importantly, have some fun!

What''s so bad about that?

K 06/18/04 06:51:30 AM EDT

The author of this article has really embarrassed himself and sys-con.com. What drivel!

David Clavey 06/18/04 06:45:07 AM EDT

I had to check the date on the article to see if it was April the 1st. I hear that Java is also faster than Machine code assembler!

My company develop applications for hand held computers and I am continually evaluating the performance of Java, VB and C++. C++ wins every time.

I have high hopes for Java, but it always needs a hardware upgrade to get to the performance of C++.

Jim 06/18/04 04:37:55 AM EDT

Wow. Isn''t it funny how some people seem to be really threatened by this kind of benchmark? Get over it, Java has many qualities that are light years ahead of what C++ offers. And after a while, a new language/platform will replace Java and C++ as the most productive one. It''s evolution, don''t expect to manage from graduation until retirement without updating your skills.

Naturally there''s a market for C++ programming still (we won''t see Doom3 implemented on Java, won''t we?) but the market share will continue to diminish. Just like programming assembler has, for example.

ClosedMinds 06/18/04 01:57:30 AM EDT

Great URL, An00n, thanks - here's what those USC authors, J.P.Lewis and Ulrich Neumann, conclude in the paper:

it is possible that no amount of data will alter people's beliefs, and that in actuality these "speed beliefs" probably have little to do with java, garbage collection, or the otherwise stated subject. Our answer probably lies somewhere in sociology or psychology. Programmers, despite their professed appreciation of logical thought, are not immune to a kind of mythology, though these particular "myths" are arbitrary and relatively harmless.

An00n 06/18/04 01:54:01 AM EDT

This article by USC graphics researchers - "Performance of Java versus C++ " - surveys a number of good (mostly numeric) benchmarks and then explains the theory of why maybe java should be faster than C++.

It also raises the (unanswered) question of why geeks (ostensibly intelligent and scientifically-minded people) continue to believe some ideas (for example, ''garbage collection is slow'') despite strong evidence to the contrary that has been available for many years.

larry 06/18/04 01:27:00 AM EDT

Scott,

That is right, the primary purpose for the eden gc space is that allocation and collection cost is very small. So for short lived objects, gc is relatively inexpensive. The cost of longer lived objects is more expensive since then you are collecting the old generation (the greater percentage of the heap). One of the older recommendations for improving gc performance was extensive object pooling. With the newer generation of gc spaces and algorithms, this is not necessary since it tends to make you application slower than the under collections methods. Pooling is still recommended for those objects with expensive tocreate native OS resources such as threads, tcp and database connections.

If you ever watched the gc behaviour, you would see that incremental gsc are far more frequent than full gc operations.

larry 06/18/04 01:20:18 AM EDT

Hi Dilip,

By safety, do you mean determinacy. That is part of my definition but my includes a variety of other things also. By and large, Java has those other things. Determinacy, however, java does not have.

However, JSR-1 the realtime spec for java does take this into account. While most of the j2me versions of RSTJ (real time specification for java) are poor performers, they are absolutely deterministic (if you know what you are doing).

I worked on creating a real time version of J2SE 1.4 which allowed you to have your cake AND eat it also (determinacy and high performance through aggressive server vm compiler). This was due to having realtime threads which ran independently of the gc cycles allowing time/safety critical regions of your application code to have excellent performance.

Real time VM''s like the one described are a new generation that I expect that you will be seeing soon enough.

larry 06/18/04 01:07:26 AM EDT

Jaka Bac,

Yes, the server VM does indeed profile code.

There are three parts to the VM, the interpreter, the client compiler and the server compiler.

When I was making changes to the VM, I would write the interpreted VM change portion first and debug it and then the compiler part later. The compiler is very complex comparatively.

The client and servcer VM are intended for completely different purposes. The client VM is intended for high user interactivity, short run times, smaller footprint. The server VM is intended for long running server type applications where performance is all.

The application is continually profiled while running

On each incremental and full gc, the profiler results are used by a compiler thread which runs in parallel to the gc threads. A server VM makes the most aggressive optimizations.

Now, you can screw around with your program so that you can generate a worse case performance but that is sort of completely outside the point, is it not?

If you let the application run longer you would fine that the compiler would figure out a way around your monkey wrench unless it as intractable.

Seeing that no one read my gc explanation,I would suggest that you investigate the various gc command line switches. You will find that you can increase your gc performance by factors of 5.

Further to that, there are a multitude of gc algorithms available to select. The system tends to use general purpose ones that have better than average behaviour over a wide range of problem domains. Witness that client VM mode being the default setting. You can choose between low pause collectors, agressive collectors, concurrent collectors and choose different ones for each generation type.

If you take a look at jvmstat (with is in the new jdk 1.5 tool set) on the cool stuff page, you have an excellent tool for analyzing gc behaviour. You can use this to trial different GC switches and see exacting what the VM is doing. I use this all the time to figure out my performance issues.

Rather than throw up your hands and complain, at least try it out, kick the tires, figure out where the nitro switches are. ;)

Google Groups 06/17/04 05:51:15 PM EDT

Here's the Google Groups thread on this benchmarking attempt.

[ See http://www.gotw.ca/resources/clcm.htm for info about comp.lang.c++.moderated. First time posters: Do this! ]

Scott Ellsworth 06/17/04 03:15:05 PM EDT

Those getting hung up on GC pauses might want to note that the GC has gotten more than a few options in recent releases, and looks to be getting a few more in 1.5. Further, the cost of object allocations has dropped dramatically with recent JDKs, making costs pretty close to on-stack for objects that would have been appropriate stack variables in C++

Sun putting effort into this is not surprising, given that this was one of the biggest performance bottlenecks for programs that allocate a number of temp objects, do a few things, and then let them fade. By having a generation that exists to hold things like this, you get very similar performance and relatively few objects choking the GC.

Scott

Dilip Ranganathan 06/17/04 02:49:10 PM EDT

This is one of the most balanced analysis of the shootout I''ve read so far. Since Google groups has not yet started displaying this thread over at comp.lang.c++.moderated (I was able to get it out of my news reader), I am copying it verbatim here:

[email protected] (Glen Low) wrote in message
news:<[email protected]>...
> Another Java vs C++ performance shootout:

> Anyone care to comment about the appropriateness of his C++ code and
> compiler settings?

When I''m benchmarking for speed, I use -O3 with g++, not -O2. That''s
definitly an error on his part.

I glanced at a few of the tests. For the most part, they are pretty low
level; it''s hard to say what significance they have with regards to real
code.

The hash code tests actually test how good the hashing function is. The
default algorithm used by g++ is definitly bad for large numbers of
short strings (it will cluster near the front of the table), but I''ve
not done enough analysis to be able to say exactly what is too large or
too short.

It would be interesting to see what happens with more complex data
structures, like std::vector (and push_back) vs. java.util.Vector. Java
has a decided advantage in array handling, because of the absense of
aliasing. As soon as you use any data structure other than an Array of a
built-in type, however, you need to dynamically allocate each element in
the structure. Relocating garbage collection generally results in much
faster allocation than you can get with operator new in C++, but even
the fastest allocation can''t compete with no allocation.

Note too that while garbage collection allocation is typically much
faster than manual allocation (perhaps an order of magnitude or so),
this is partitially offset by the time you spend collecting. His
programs allocate little enough that he probably never needs to collect.

The small size of the programs are a definite advantage for Java''s
runtime optimizer. A run-time optimizer can''t spend forever optimizing,
but it has access to actual run-time data, AND can specialize its
optimization for the specific processor. (G++, and most other C++
compilers, have flags to allow this as well.) Given the small size, a
complete analysis of the program is possible in the limited time
available, and given the other additional data, I would find it rather
worrisome if Java didn''t do better. I''m more sceptical with regards to
real programs, however. If there really is one small critical loop,
Java should win; that''s not the case for most of my programs, however.

And of course, g++ is far (very far) from state of the art optimizing,
especially (so I''ve been told) for Intel architectures.

All that said, however, for many applications, there will be little or
no difference in speed between Java and C++. Java has a number of
advantages which make life easier for the optimizer, and for most
applications, garbage collection will be faster than manual memory
management, but the fact that everything has to be dynamically allocated
can cost a lot.

In the end, of course, performance isn''t necessarily the key argument.
For most of the work I do, safety is. And that is a criteria where C++
is a clear winner; it is very, very difficult to write correct programs
of any significant size in Java. (It''s not really that easy in C++, but
at least the language never actively prevents you from doing the right
thing.)

--
James Kanze GABI Software
Conseils en informatique orient�e objet/
Beratung in objektorientierter Datenverarbeitung
9 place S�mard, 78210 St.-Cyr-l''�cole, France, +33 (0)1 30 23 00 34

[ See http://www.gotw.ca/resources/clcm.htm for info about comp.lang.c++.moderated. First time posters: Do this! ]

Jaka Bac 06/17/04 12:28:34 PM EDT

Like some people said here already "My gun is bigger than yours" comparisons are silly and pointless. But instead of saying angry things to each other let's discover why java is so fast in these cases.

Since i found it hard to believe, i tried the infamous methcall test myself and i was trully unable to make c++ version faster than java. (I got close but it was still slower)
Then i decided to check why java's version is so fast. So i compared disassembly of c++ program, server JVM JITted class and client JVM JITted class. And here is what i found out.

1.C++ disassembly showed it threw all optimizations it could at the program. From loop unrolling, instruction scheduling to method inlining. And when i checked CPU preformance counters i could see that it used about 16e9 cycles and 16e9 instructions to compute tests iterations.

2.client JVM
It jitted the methods, but disassembly was ugly and of course slow, nothing spectacular except it looked like jvm is re-jitting the class now and then (making several versions) And yeah, it used about 22e9 instructions in 33e9 cycles.

3. server JVM
Again dissembly showed that JIT compiler made several versions, but funny thing was that it took only 6e9 instructions in 3e9 cycles to compute the results. Which means that it simplified the algorithm somehow, skipping loop cycles. This was also confirmed by examining disassembly. So my hunch was, that server JVM is profiling the code and trying to simplify it if it can do it.
To prove my thesis i tried to reduce number of test iterations and guess what, java's performance was worse than that of c++.
Then i suspected that because compiler has runtime information it can do the simplification in this case, so i changed the code not to just toggle one boolean value, but to increment an int. Doing so it would need previous result for every invocation of activate and performance of java dropped drasticly, but c++ version stayed practicly same.

So for all these tests it is safe to say, that they run faster on server JVM, because they are ran over 100000000 times and jvm has chance to do some kinds of runtime optimizations. But this is hardly the generic case. Otherwise all programs would start to fly when -server was used, but my java ide refused to take off into the sky. But i admit that whoever made the server JVM JIT, had pretty slick tricks under their sleeves.

So for the end i think we have learned that comparisons like this are silly, that it is most important that we write good algorithms in the first place and not let compilers do our dirty work and that each tool is designed for a purpose, i think people who made java never meant make C or C++ obsolete. Do you think JVM itself is written in java?

m 06/17/04 12:03:02 PM EDT

The benchmarks are interesting, but frankly with real applications, I have never experienced Java performance equal to or better than that of C++. I have done a number of home grown benchmarks for XML parsing, and C/C++ (think expat) are 3-9 times faster. The difference there is that the server jvm does indeed optimize over a series of many runs.
Java certainly allows us to develop faster, and quite often time to market performance is more important than the performance differences in these two (great in my opionion) platforms. They each have specific advantages.

However let me make one more point. If I were going to do "cross platform" development, and development time was not an issue, I would choose good old-fashioned C.
It is available everywhere, it is well known and mature. It is the basis of almost every commercial "platform". I know it isn''t fashionable, and writing it well is almost an art. But there you have it.

Bob Riddle 06/17/04 10:15:42 AM EDT

In the spirit of discussion I do quibble with whether such tests have any relevance from a business respective, though.

I like Java when used appropriately and work with it every day. But it is only one tool on my belt and I do not belong to the "Java: one tool to rule them all" religious cult. I also help maintain a compiled code legacy system that supports 40,000 active users and manages 10 ms. average response times at the server boundary at an 800 TPS demand level with only about 2,400 MIPS.

In the real world, the issues are not how fast the base Java compiler and JVM implementations can do Math, matrix manipulation, or string concatenation. I?ve routinely been able to get good performance for the basic actual Java code execution of comparable lines of code.

From what we?ve seen benchmarking real-world corporate apps, the real world issues of using Java revolve around:

Non-responsiveness during Garbage Collection ? (example: a Java port of our high-volume XML message broker on average performed on a par with a compiled version in between GC cycles but response times tanked by nearly 1000% during a full GC causing uncomplimentary 90th percentile and max response times during the tests). GC cycles often seem to have a negative effect far beyond the actual time it takes to do a GC due to queue depths building up during the GC because of steady request arrival rates, TCP/IP network retries caused while the machine is in the GC loops, etc.

Resource requirements to start up a complete virtual environment for each individually launched program. (example: just firing up a multi-threaded Java services test harness added over 40MB to my workstation?s current memory commit while starting up my similar multi-threaded Delphi tester added just over 2 MB; it also takes many times as long for the standalone program to start up, paint its initial screen, and become usable).

General poor performance of many widely-used, widely-accepted Java technologies and libraries (example: Xerces, Cocoon, etc). Often it seems that these technologies were written more in line with ?the Java way? and academic OOP purity than with a critical eye on performance and transient object creation under high load and busy CPU conditions).

Poor performance and, often, poor functionality when interacting with non-Java technologies (example: JNI threading issues, interface complexities when Java is the called rather than the calling technology). Much of the ?Java way? of doing things falls apart when mixed language environments are required (example: using third-party libraries, use of scalar and bitmapped common System fonts, security ACL exchanges, propagation of exceptions during interactions with 3rd party Windows apps, etc.).

Maddening quirks and lengthy, ugly code required to do a good GUI interface that matches what users expect from Windows and Macs (example: three pages of code to instantiate a non-mutable complex grid vs. a single line of code in Delphi, C++Builder, etc. to create a grid to which I can add a column, change column header font/color, etc. with only one more single line of code, lousy mouse wheel support, repaint issues, difficulty in creating tight layouts in limited screen space).

Versioning problems between JDK requirements which are as bad as any ?DLL Hell? requirements from Windows (example: IBM JVM required to talk to WebSphere, obsolete JVM required for Fidelity Extensity e-Expense app, inability to run various containers under new JDK?s for months after Sun releases them, etc.). This is particularly true now that .Net allows a program installation with odd versions of DLLs and objects solely by a single XCOPY of a directory.

None of this means that Java is bad; just that it is not always the best tool for every job as some religious zealots claim. It also means that Sun needs to kick it in gear and address these types of real world issues.

Chris White 06/17/04 09:41:54 AM EDT

Thanks for the comparison Keith. I enjoyed reading your commentary.

Many people have stated in response to this article that Java, as everyone knows, is slow... This is certainly not the case. I am team leader of a Java development team. We have six applications in production on a certain open source application server. We have 1200 users of these applications. The applications are load-balanced over two machines (each machine : HP DL380 2*2.8GHz 2Gb RAM). We have on average 2% Cpu utilisation. Whilst this is of course just a subjective view of Java performance, I can say that we certainly do not have the impression that Java is at all slow.

I accept that for memory deallocation C++ can be tailored to the needs of the application at hand so it will be difficult to beat on this score, however one must remember that the just in time compiler compiles code into native machine language and that this compiler is very much optimised on a per platform basis so I don''t see why C++ should produce naturally faster compiled code.

Whichever is faster, Java is easily fast enough for most applications and I strongly believe that the overhead in development time incurred by using C++ is rarely justified except in specialised applications such as driver or game coding.

Christopher White

GPSnoopy 06/17/04 09:09:42 AM EDT

I agree that most of the benchmarks given here are useless; however there are a couple that can be interesting (e.g. the sieve/prime finder).

If you want to see what I consider to be a more correct prime finder in C++, I suggest you check http://users.pandora.be/tfautre/softdev/stuff/Prime.zip (if you don''t know a lot about C++, I strongly recommend you look at the code, just to understand that the aforementioned C++ benchmarks are everything but representative C++ code).

I did some early tests, and this implementation outperforms the given sieve.cpp.
I modified sieve.cpp so that it would look up, only once, for all the primes between 0 and 33554433 (8192 * 4096 + 1). On a P4 2.6 C 1024 MB, compiled with VC7.1 (target P4), it took 3844 msecs with his implementation, and 375 msecs with the second one.

If anyone wants to convert it to Java, be my guest. I'd love to see the results, and moreover these results sent back to the author of those awful benchmarks.

Chris 06/17/04 02:03:02 AM EDT

I wonder how many anecdotal language war assertions and personal preferences I can fit into a comment prefixed by "this benchmark is meaningless"?

... and then perhaps I can assume my specific programming domain demonstrates "real world" scenarios for everyone and cap the whole comment off by a confused understanding of what a programming language is (as opposed to an API or an implementation of compiler or runtime).

Surely the benchmark is no less meaningful than when it was originally used to show C++ was faster than Java.

For a bit more better-informed discussion:

http://c2.com/cgi/wiki?GreatComputerLanguageShootout

Girod 06/16/04 07:17:21 PM EDT

For those who, even now, stick to the old anthem "Java is much slower than C++" :

- it is know clear that it is no longer the case, and that Java is often faster and cleaner than equivalent C/C++ code (apart for some types of apps, when C++ / C is faster, for example when using JNI extensively), and even in real world apps !!
- it is also true that java tend to use a lot more memory the equivalent C/C++ program, but I don''t know the odds when using extensively things like STL in C++, for example...
- and if you want to have fun and experiment the fastness of a C code straightforward port to Java, try Jake2 (a port of Quake2), at the following URL : http://www.bytonic.de/html/jake2.html, the Java code is very similar to the C code. Java is slower in this case, but not too much slower (test it yourself), and :
- there was no optimization yet (will be done later by the team who did this excellent work)
- lot of JNI used for the OpenGL wrapper (jogl)
- direct C structure transfered to Java