Welcome!

Java Authors: Elizabeth White, Carmen Gonzalez, Dana Gardner, Ivan Antsipau, Esmeralda Swartz

Related Topics: Java, Websphere, Weblogic

Java: Article

Java vs C++ "Shootout" Revisited

Java vs C++ "Shootout" Revisited

Keith Lea writes of the benchmark, on his results page, "I was sick of hearing people say Java was slow, when I know it's pretty fast, so I took the benchmark code for C++ and Java from the now outdated Great Computer Language Shootout and ran the tests myself."

Lea used G++ (GCC) 3.3.1 20030930 (with glibc 2.3.2-98) for the C++, with the -O2 flag (for both i386 and i686). He compiled the Java code normally with the Sun Java 1.4.2_01 compiler, and ran it with the Sun 1.4.2_01 JVM. He ran the tests on Red Hat Linux 9 / Fedora Test1 with the 2.4.20-20.9 kernel on a T30 laptop. The laptop "has a Pentium 4 mobile chip, 512MB of memory, a sort of slow disk," he notes.

The results he got were that Java is significantly faster than optimized C++ in many cases.

"They also show that no one should ever run the client JVM when given the choice," Lea adds. ("Everyone has the choice," he says. To run the server VM, see instructions in the Using the Server JVM section below.)

JDJ has agreed to post online anyone else's results as long as they use Java 1.4.2 or higher and any version of GCC that produces faster or equivalent code than the 3.3.1 I used. We encourage you to download the source and/or the binaries and perform the tests yourself, with your favorite compiler and on your favorite platform.


Lea's Data and Results

JVM startup time was included in these results. "That means even with JVM startup time, Java is still faster than C++ in many of these tests," says Lea.

Some of the C++ tests would not compile. "I've never been very good at decoding GCC's error messages," he admits, "so if I couldn't fix a test with a trivial modification, I didn't include it in my benchmarks."

Lea also modified one of the tests, the string concatenation test for Java.

"The test was creating a new StringBuffer in each iteration of the loop, which was just silly," he explains. "I updated the code to use a single StringBuffer and appending to it inside the loop."

(The updated tests at the original shootout use this new method.)

"Java lost this benchmark even with the modifications," Lea declares. "So if anyone wants to accuse me of biasing the results, they're going to have to try harder."

Several versions of some of the C++ tests (like matrix) were present in the original shootout source, he continues. 

"I used the versions without numbers in them, like matrix.g++ instead of matrix.g++2. I don't know which of these were used in the original benchmarks, but from my quick experimenting, the numberless ones generally ran faster than their numbered counterparts."

"Looking at them again," Lea says, "matrix.g++3 runs faster than the matrix.g++ that I use. However, it still runs slower than the Java version, so I don't plan to modify the graph/data unless someone asks me to, since getting that graph in the first place was sort of a pain.)"

He continues: "I've been told that the C++ code for the Method Call benchmark returns by value while the Java code returns by reference, and that modifying the C++ code to pass a pointer makes that benchmark faster. However, even with the modification, the C++ version still runs slower than the Java version."

Lea ran th Java and the C++ tests to "warm up" (both the Java and C++ tests got faster after he ran them a few times).

"I've been told that these tests are invalid because they were run with GCC," he concedes, adding: "I have seen both benchmarks that show GCC producing faster code than Visual Studio's VC++ compiler, and benchmarks showing the opposite. If I update the benchmarks with another compiler added, it will be the Intel C++ Compiler, which I'm pretty sure produces faster code than VC++."

Lea says he's been accused of biasing the results by using the -O2 option for GCC, "supposedly because -O2 optimizes for space, thus slowing down the benchmark," he explains.

But this is not what -O2 does, he points out, referring to the GCC -O documentation:

JVM startup time was included in these results. "That means even with JVM startup time, Java is still faster than C++ in many of these tests," says Lea.

Some of the C++ tests would not compile. "I've never been very good at decoding GCC's error messages," he admits, "so if I couldn't fix a test with a trivial modification, I didn't include it in my benchmarks."

Lea also modified one of the tests, the string concatenation test for Java.

"The test was creating a new StringBuffer in each iteration of the loop, which was just silly," he explains. "I updated the code to use a single StringBuffer and appending to it inside the loop."

(The updated tests at the original shootout use this new method.)

"Java lost this benchmark even with the modifications," Lea declares. "So if anyone wants to accuse me of biasing the results, they're going to have to try harder."

Several versions of some of the C++ tests (like matrix) were present in the original shootout source, he continues. 

"I used the versions without numbers in them, like matrix.g++ instead of matrix.g++2. I don't know which of these were used in the original benchmarks, but from my quick experimenting, the numberless ones generally ran faster than their numbered counterparts."

"Looking at them again," Lea says, "matrix.g++3 runs faster than the matrix.g++ that I use. However, it still runs slower than the Java version, so I don't plan to modify the graph/data unless someone asks me to, since getting that graph in the first place was sort of a pain.)"

He continues: "I've been told that the C++ code for the Method Call benchmark returns by value while the Java code returns by reference, and that modifying the C++ code to pass a pointer makes that benchmark faster. However, even with the modification, the C++ version still runs slower than the Java version."

Lea ran the tests many times before running the "official" recorded set of tests, so there was plenty of time for both Java and the C++ tests to "warm up" (both the Java and C++ tests got faster after he ran them a few times).

"I've been told that these tests are invalid because they were run with GCC," he concedes, adding: "I have seen both benchmarks that show GCC producing faster code than Visual Studio's VC++ compiler, and benchmarks showing the opposite. If I update the benchmarks with another compiler added, it will be the Intel C++ Compiler, which I'm pretty sure produces faster code than VC++."

Lea says he's been accused of biasing the results by using the -O2 option for GCC, "supposedly because -O2 optimizes for space, thus slowing down the benchmark," he explains.

But this is not what -O2 does, he points out, referring to the GCC -O documentation:

-O2: Optimize even more. GCC performs nearly all supported optimizations that do not involve a space-speed tradeoff. The compiler does not perform loop unrolling or function inlining when you specify -O2. As compared to -O, this option increases both compilation time and the performance of the generated code.

"On the other hand, -O3 performs space-speed tradeoffs, and -O performs fewer optimizations. Thus, for these tests, I think O2 was the best choice," Lea concludes.

 

"I don't have an automated means of building and benchmarking these things (and the scripts that came with the original shootout didn't run for me)," he continues. "I really do want people to test it on their own machines, but it's going to take some work, I guess."

Lea compiled the C++ code with:

g++ [test].cpp -O2 -march=i386 -o [test]-386

g++ [test].cpp -O2 -march=i686 -o [test]-686

and the Java code with:

javac [test].java

To see how he ran the binaries, see the run log. You can download the source code he used in either .bz2 or .zip format.

Using the Server JVM

Every form of Sun's Java runtime comes with both the "client VM" and the "server VM."

"Unfortunately, Java applications and applets run by default in the client VM," Lea observes. "The Server VM is much faster than the Client VM, but it has the downside of taking around 10% longer to start up, and it uses more memory."

Lea explains the two ways to run Java applications with the server VM as follows

  1. When launching a Java application from the command line, use java -server [arguments...] instead of java [arguments...]. For example, use java -server -jar beanshell.jar.
  2. Modify the jvm.cfg file in your Java installation. (It's a text file, so you can use Notepad or Emacs to edit it.) This is located in C:\Program Files\Java\j2reXXX\lib\i386\ on Windows, /usr/java/j2reXXX/lib/i386/ on Linux. You will see two lines:
    -client KNOWN
    -server KNOWN
    You should change them to:
    -server KNOWN
    -client KNOWN
    This change will cause the server VM to be run for all applications, unless they are run with the -client argument.

He can be contacted at

Every form of Sun's Java runtime comes with both the "client VM" and the "server VM."

"Unfortunately, Java applications and applets run by default in the client VM," Lea observes. "The Server VM is much faster than the Client VM, but it has the downside of taking around 10% longer to start up, and it uses more memory."

Lea explains the two ways to run Java applications with the server VM as follows

  1. When launching a Java application from the command line, use java -server [arguments...] instead of java [arguments...]. For example, use java -server -jar beanshell.jar.
  2. Modify the jvm.cfg file in your Java installation. (It's a text file, so you can use Notepad or Emacs to edit it.) This is located in C:\Program Files\Java\j2reXXX\lib\i386\ on Windows, /usr/java/j2reXXX/lib/i386/ on Linux. You will see two lines:
    -client KNOWN
    -server KNOWN
    You should change them to:
    -server KNOWN
    -client KNOWN
    This change will cause the server VM to be run for all applications, unless they are run with the -client argument.

He can be contacted at [email protected].

Links

More Stories By Jeremy Geelan

Jeremy Geelan is Chairman & CEO of the 21st Century Internet Group, Inc. and an Executive Academy Member of the International Academy of Digital Arts & Sciences. Formerly he was President & COO at Cloud Expo, Inc. and Conference Chair of the worldwide Cloud Expo series. He appears regularly at conferences and trade shows, speaking to technology audiences across six continents. You can follow him on twitter: @jg21.

Comments (152) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
Scott Ellsworth 06/16/04 12:18:33 PM EDT

The point is that many C++ users feel that java is too slow and uses too much memory. Microbenchmarks like this show that this off the cuff response is as silly as claiming that exceptions cost too much just because cfront compilers had that cost.

I spent seven years doing C++ in a shrink wrap environment, and five doing Java. In my experience with real world apps, Java code gets done faster, works better out of the box, and is easier to maintain. Tricks like custom memory allocators and carefully crafted templates can make faster C++ code, but I reserve those for time critical parts of the program. Where I do not spend some programmer cycles profiling and optimizing, the two environments are pretty close.

Thus - if Java is producing code reasonably close or ahead in fairly straightforward usage, and the code gets written faster and more portably, then Java is a win for my work. I can always choose to use C++, assembly, or FORTRAN for the time critical routines, and then link in.

Scott

Larry 06/16/04 12:17:20 PM EDT

Calling System.gc() marks someone as a rank neophyte. Garbage collection does not run in paralllel to the main execution. It runs when the main app is not running Sorry if this is insulting, but there is no real reason to ever call it. This was left for backwards compatibility with older vm''s

As someone who has developed a jvm, one of the worse things for performance is to manually call System.gc(). This invokes a full gc only which is the most intensive gc possible. Like killing a rat with an atom bomb. Effective but definitely causes collateral issues.

Garbage collection since 1.2, 1.3, 1.4 and currently in 1.4.2 have leapt in technology capabilities tremendously.

A lot of people seem to be confused about what is done in garbage collection.

There are several types of garbage collection in a VM. These can be categorized as incremental or full. Incremental involves sweeping through a smaller space to remove objects making space. It is very fast, primarily due to the nature of the space it works on. As said before, full is the full monty, quite time consuming and linaerily proportional to the heap size.

There are several differently type of spaces where objects are kept in the heap. These are called generations. These are new generation, old generation and permanent generation. Incremental gc is done mostly on the new generations

All objects created go into the new gen which is composed of an eden (as in garden of ...) where they are first created and two survivor spaces called 1 and 2 where they are moved on each inc gc if they are still needed. These spaces use very efficient algorithms for allocation, movement, and deletion.

Once an object has been around for a while, it is transferred to the old generation where it is eventually tenured.

The permanent gen is used for interned string (a quoted string in your code), all classes and methods, etc.

full gc compacts the old gen and this can be expensive in time. There may be 100 incremental gcs for every full gc but this depends on your code. Gc is triggered whenever certain way points are reached in the different spaces and generations. Every time eden is filled then an inc gc occurred. Since eden is a smaller fraction of the whole heap, this can be quite fast.

There are many reasons why an object is collected but it comes down to the type of reference attadched to it of which there are 4 types. Strong references are never collected.

I generally run the application and look at the gc behaviour and further tune this with command line parameters to select optimal algorithms based on what I see. I can generally increase gc performance by a factor of 5 because I know what I am doing.

Don''t forget that while a the vm is doing gc it is also compiling the byte code based on performance values captured in the last active phase of the app. Each time the app is running, perfomance measurement and benchmarks are run. When gc happens, those portions of the app that are under the heaviest load are optimized. This sort of capability is what gives a runtime optimizer the teeth to do what a static optimization system like those present in C++ can''t.

Oh and the sun vm is written in C++ mostly. There is a fair amount of inline asm.

Swen Tromm 06/16/04 11:33:38 AM EDT

José: I think this shall addressed on an other place: But very briefly: What would I like to see is a programming language may be called "C++ strict" which follows C++, but has a stricter type handling (e.g. ending the muddeling of size_t and integer, the weak implemention of bool etc.) and the end of keyword like "export".

SWagner 06/16/04 11:33:26 AM EDT

As a expirenced C++ developer (and Java as well) I only see that the C++ code is optimized for minimum performance.

Especially the methodcall where the class hast virtual constructors or destructors which is seldom useful and significantly hurts performance. Only because you cant turn this off in Java does not mean that you have to make it slow in C++.

And that is only one example for all the flaws of this test. In my practical experience normal Java code is 1000 times slower to OPTIMIZED C++. However if you make severe design flaws you can make any program slow.

And if you base on a bad benchmark and make it even worse you should better stop any benchmarking you planned for the future.

José 06/16/04 11:33:00 AM EDT

And lets not get into Solaris. I have no problems with it, other than Linux or FreeBSD serves me well, because I haven''t got an enterprise to run.

José 06/16/04 11:29:41 AM EDT

I dont mean to be patronizing, it''s just that you stated clearly that I didnt know anything about Java, so people shouldn''t listen to what I have to say.

I know very well about Java''s advantages. I agree, C++ doesn''t do cross-platform, but nor was it designed to be. I think it''s great that Sun likes the whole WORA concept, but is it what people really want? Considering the disadvantages? I much prefer diversity. But it''s certainly admirable. I think maybe its a case of being way ahead of their time. You could argue that if speed is all that matters, then one could program in assembly rather than C/C++, for example, but really there was a huge leap between C/C++ and the older languages and the advantages were clear. It was not seen as an appendage of existing technology or just an alternative. This is what I''m trying to say. Java suffers because it hasn''t got that extra oomph to make it as desirable as it wants to be.

Where C++ lacks, you always have alternative technologies. See, I don''t believe in encapsulating every technology into one box. Where technologies win is where it does what it does and it does it very well.

But I digress. Java will always lose out because of the history and implications of Microsoft/UNIX/Linux. Sun doesn''t have an operating system or application to truly show off with, and maybe thats important, because I don''t think many serious industry types really trust Sun with the future of Java. It has too much control over it. And many people think that Java is the last peg Sun has to stand on. What happens when Sun goes belly up? There are more issues than just the technical side of things.

Very Funny 06/16/04 11:16:55 AM EDT

The great thing about these comments is that this article brought all two dozen C++ coders out of the wood-work.

Hehehehehehehehehehehehehehe.

That''s a great comment Cameron

Cameron 06/16/04 11:07:44 AM EDT

The great thing about these comments is that this article brought all two dozen C++ coders out of the wood-work.

Lloyd 06/16/04 10:52:40 AM EDT

Jose said "Also, as I mentioned, this feedback forum is about the article stating that Java is faster than C++, which it isn''''t nor was it designed to be. " AFTER ranting about its various IDEs and its utility in serious projects.

yes I''ve know of System.gc() (but thanks for being patronising) but honestly I have never used it in serious code. Relying on the JVM''s GC algorithm has always worked out fine thus far for me (though I can understand that it would be needed a lot more in a mobile phone environment).

I think the argument I was trying to make was that whilst java undoubtedly doesn''t run as fast as C++ (assuming both programs are written perfectly) it can and has been used for many ''serious'' applications. Or are you discounting Java''s use in web based apps some companies have used it there in very serious ways?

You can complain about applet performance but how well does an applet run thats written in C++? whats that? oh it doesn''t run because there is no cross platform client based support for it. I''d say not being possible counts as infinitely slow.

sorry, I realise that post was emotive and I know C++ is lovely and all that just that it is not the be all and end all and don''t ya forget that.

José 06/16/04 10:29:01 AM EDT

Well, some people misunderstand the use and reason for Export, which is one of its main problems.

http://anubis.dkuug.dk/jtc1/sc22/wg21/docs/papers/2003/n1426.pdf

Grzegorz Grasza 06/16/04 10:16:34 AM EDT

Try using printf instead of cin/cout... (it''s significantly faster in GCC, at least version 3.3)

Swen Tromm 06/16/04 09:37:57 AM EDT

José wrote:"The value of a language consists >not in how fast it runs, but in >what it enables the programmer to do.Which means a win-win situation to C++ in any case, ...". This is just not true. The strengh of a programming language lies also in what it does not provides. C++ weaknesses are more to see in key words like "export" which are not needed for a well planned programme, but exist. C++ would be a much better language if such ideas would be ereased.

José 06/16/04 09:20:14 AM EDT

Lloyd, thanks for the personal comments, but dont be sad if people will read my comments. I do know what Im talking about and my opinions are not mine only.

Namely, the garbage collector. It''s meant to run as a thread parallel to your code, but in reality, in freeing the programmer from the responsibility of recycling memory, clever algorithms are needed and therefore the garbage collector is only as good as the algorithm (plus the overhead on the CPU). As I stated, my opinions are formed of empirical evidence from my own experience. I can even give you snapshots of memory graphs from my own development experiences. Also, this makes it very inconvenient for critical and/or optimized applications that rely on speed and load balancing. Also, does this line ( System.gc() ) not look familiar? Alas, I''ve seen many times the momery usage go up to hundred or so megabytes and then suddenly drop to 15 for example, and not only on my own programs.

As for Java hanging and being slow, trust me... It was a fresh install of Windows. Netbeans, leave it running for a while, maybe an hour or so and notice the slow down... maximize on the desktop and Netbeans slows down to a crawl. In particular when using JavaDocs auto-completion, such that I can make some tea and come back in time to see the combo box appear with accompanied JavaDoc definition. Another example is the Wireless Toolkit for J2ME devices, over time compiling the program and running it in an emulator gets really slow. What you can do with the Wireless Toolkit is tell it to output every line of code possible and you get your screen flooding with code that... well... I think the effect would be stronger if you took a look yourself. And we are just talking about a cut down version of J2SE.

If you think I''m trying to say that Java is completely rubbish, I apologize but that''s not what I mean to say. However, I just don''t think Java is as great as everyone makes it out to be, especially Java gurus and I think it encourages messy programming, despite what everyone tries to make me think. Which is why I place it between C++ and Visual Basic, for example. But I don''t use it for any serious work nor would I recommend it for serious projects. But it works quite well for some things, for example, I think its brilliant for mobile phone technology. Unfortunately, most programming still requires optimized code these days. And thats really what I''m trying to say.

Also, as I mentioned, this feedback forum is about the article stating that Java is faster than C++, which it isn''t nor was it designed to be.

Jim Slade 06/16/04 09:05:35 AM EDT

Reminds me of the benchmarks M$ paid for showing that Visual Basic was faster than Delphi.

Lloyd 06/16/04 08:18:54 AM EDT

it makes me sad how many people will listen to Jose and think he knows whats hes talking about with regards to java.

yes, java is sandboxed _When_Running_In_A_Client_JRE_In_A_Browser but for good security reasons and therefore offers a security model not (as far as I know though I may be wrong) provided in C++.

I really don''t know what you were going on about the Java GC needing to be manually called for. I''ve been developing real-time java (and C++ to improve performance in key areas) apps for a couple of years and haven''t seen any GC calls in the code. maybe you were hanging on to object references, I don''t know.

I will totally agree that C++ is faster to run like for like programs than Java. I won''t agree with statements saying it always produces painfully slow code and programs that hang (mine don''t : ) .... well, maybe sometimes).

C++ can live together quite well with Java providing, in my opinion, an environment where I can get some very complex high level things developed rapidly and providing cross platform support whilst certain system aspects can be left to native C++ implementations that are accessed via JNI (Java Native Interface). I haven''t seen any problems with this approach to rapid development of robust real-time apps.

....and relax.

José 06/16/04 07:52:53 AM EDT

>The value of a language consists
>not in how fast it runs, but in
>what it enables the programmer to
>do.

Which means a win-win situation to C++ in any case, because Java is no revolution in programming terms really, which is why it is and always has been the vague point in the computer industry and why its not really being adopted that much, despite at its matured stage. And clearly, speed is absolutely a serious consideration when it comes to HCI.

In any case, people here are not arguing whether or not Java is useful or not or more useful in some contexts. In case you have forgotten, this article was specifically about Java being faster than C++, which is completely wrong.

Trance 9 06/16/04 07:38:09 AM EDT

The value of a language consists not in how fast it runs, but in what it enables the programmer to do.

Trance 9 06/16/04 07:36:25 AM EDT

A good test of a compiler is whether it can take ordinary code and get good performance out of it. It's much less interesting to take optimized code and see how much more the compiler can optimize it.

What you're basically saying is if I want to hand optimize every line of code I write, rather than concentrating on the task at hand, then I should use C++, but if I want the compiler to do the heavy lifting then I should use Java?

OK--I''ve made my choice!

Realistically each language has its place. The JVM is written in C because C is a good systems programming language. For a similar reason the kernel is written in C, and so forth. However that doesn't mean you want to write business logic of any kind in C--you don't.

The question is then whether you turn to C++ or to Java or to some other higher level language.

Patrik Beno 06/16/04 07:35:36 AM EDT

This benchmark was made by Java guy who has only a little expertise in C++.

I suggest that now it''s time for C++ expert to optimize C++ code and compilation, run the very same tests again and post the results.

I would do that but I''m Java guy too :-)

So don''t blame Lea, he did his best, now its your turn, C++ experts :-)

Troy Baer 06/16/04 07:34:44 AM EDT

The author doesn''t really explain why he didn''t compile with -O3 aside from a very slight amount of hand-waving about space-speed tradeoffs, which quite frankly I don''t buy. If you''re benchmarking, why wouldn''t you optimize as heavily as possible? If he was really interested in benchmarking this stuff objectively, he could''ve at least shown that there wasn''t much different between -O2 and -O3. Not to mention the question of whether g++ generates good binary code on a given platform...

This didn''t exactly fill me with optimism either:

I don''t have an automated means of building and benchmarking these things (and the scripts that came with the original shootout didn''t run for me). I really do want you to test it on your own machine, but it''s going to take some work, I guess.

This would seem to imply that the author does not know much about either shell scripting or Makefiles. I''m not sure I''m willing to trust benchmarks from somebody who can''t figure out an automated way to build and run them.

miyako 06/16/04 07:32:39 AM EDT

I wish people would just get over this programming language pissing contest that seems to have been going on, and promote each language based on its own merits, and not try to say that any given language is the end all be all of programming.

In my experience, Java is fast to write, and C++ is fast to run. C++ can be hard to debug, and Java can make it hard to eliminate overhead.

If one day Java is faster than C++, Then I think it will have come at the expense at many of the things that java gets right.

TheLastUser 06/16/04 06:48:19 AM EDT

Its not impossible that Java would be faster at some things than C++, merely because the VM is written in C++. VM''s have runtime advantages over languages like C++. A VM can optimize code based on how it is currently being used and where it is currently running, whereas C++ optimizations are a strictly compile-time exercise. If I create an app running on Linux/x86 and have it send an object to a Solaris/sparc machine, the object will optimize itself properly on both machines. If my application starts to use a particular segment of code heavilly then the VM can re-optimize for that execution pattern.

Java takes a hit by requiring a VM, but, as this article shows, the hit isn''t that big and the payoffs are probably worth it.

rebelcool 06/16/04 06:44:56 AM EDT

Java is faster for some applications than C++, and for others C++ is faster. It depends on what you want to do with it.

I think java continues to improve its performance and feature set though, with every new version that comes out every couple of years. C++ is more or less into its mature, sunsetting years and not likely to have any revolutionary, or even major evolutionary, improvements.

One Problem 06/16/04 06:43:35 AM EDT

Java and C++ are language. Languages aren''t "faster" or "slower", but compilers for them might be.

Omnifarious 06/16/04 06:38:41 AM EDT

I think Java is highly overrated. I would prefer that a better C++ (a C-like memory model, powerful generic programming, inheritance, and polymorphism) that lacked C++''s current nightmare of strangely interacting features and syntax.

I use Python when I don''t need C++s speed or low-level memory model, and I''m happier for it. It''s more flexible than Java, much quicker to develop in, and faster for running small programs. Java doesn''t play well with others, and it was seemingly designed not to.

Besides, I suspect that someone who knew and like C++ really well could tweak his benchmarks to make C++ come out faster again anyway. That''s something I''ve noticed about several benchmarks that compare languages in various ways.

MyName 06/16/04 05:44:36 AM EDT

The server VM is faster but consumes more memory??? But high memory comsumption is one of the main problems with Java! And takes even more time to start?
This is plain dumb. Also, i agree with those who said that C++ should be doing no deletes to compare with Java!

Dr. Valentin Samko 06/16/04 05:36:50 AM EDT

methcall example, win32, g++ 3.3.3 (mingw special), compiled with -O3, results are quite different from what you have.

> time java -client methcall_java 1000000000
0.01user 0.01system 0:19.39elapsed
> time methcall_cpp 1000000000
0.03user 0.00system 0:12.03elapsed

José 06/16/04 05:34:27 AM EDT

>i too, am sick of hearing about how slow java is.
>i think this claim had a lot more validity back in
>the day (''''98 or so) but now it''''s just echoed
>amongst the idiots. one thing they''''ve beaten into
>us in computer science courses, is that optimization
>is really only needed in a few critical areas, and
>the rest of the program wont depend that much on
>performance.

>with today''''s absurdly fast computers, just-in-time
>compiling and runtime optimizations, semi-native
>environments (OSX''''s java, GCJ, JNI, or projects
>combining multiple languages) -- why must this
>issue constantly be touted every time java is
>mentioned?

For 2 reasons:
1. Because Java is still __really annoyingly slow__. Java applications still run abysmally on my home development PC, on an Athlon 1800+ with 512 mb ram with Windows XP & Java software from SUN. No, the difference in speed is not irrelevant _yet_. Java applications still have the annoying habit to often (often being the word) suddenly freeze and then you just don''t know what it''s doing anymore, whether its crashed or not. Often, Java widgets and UI elements are ugly and really should atleast be on par with Microsoft''s basic style (this is more a quipe than anything).

2. Java''s WORA concept is nice but with heavy flaws. For example, Java apparently interprets floating points differently depending on the machine it was written for. People still joke about using applets for a good reason. Because Java has no native OS, it ultimately depends on how well someone can wrap a layer onto the underlying OS or "if" it is possible to do it well at all. Ultimately, Java is best suited for small applications where speed matters little and memory limits are defined (such as mobile phones). Ironic, considering that Java originated from set-top tv boxes.

Java, although finding its niche in some markets, is still struggling to find its purpose as being something between "cool, proven and well-liked as C/C++" and "simple to write for and controlled by one monopolistic company Visual Basic". Yeah, I hear lots of Java gurus go one about how Java is totally object orientated and interfaces and bla-dee-bla, but really, that''s all it is, bla-dee-bla...

FYI, the Garbage Collector is terrible. Invoking it directly is not recommended by Sun, but you have to, otherwise you watch your memory fill up wondering how the hell your small application could possibly use so much. Then once it reaches some peak, then the GC decides to do something about it, and suddenly you are back to pretty small memory usage and the cycle continues. Good idea in principle, but its a bit like relying on an elephant to clean up your room after you. I prefer to do that job myself.

Jukian Morrison 06/16/04 05:26:26 AM EDT

C++ is lame, so, for the race to be fair, Java has to have one leg strapped behind its back? Rubbish. Java CG is supposed to defer memory stuff into efficient rare operations rather than faffing about with allocation and deallocation in the antique manual style of C.

GooberToo 06/16/04 05:24:34 AM EDT

The more I read the code, it sure is starting to look more and more like an apples and oranges comparison; which is usually what happens when people do java benchmarks.

As typically observed, I''m seeing the benchmarks take serious advantage of java''s GC mechanism, whereby, they never pay the piper. With C++, the piper is constantly being paid. All of the benchmarks which allocate objects and then delete them, are therefore, invalid. To be fair, I think, you either need to add a System.gc() line in the java code where C++ is doing its deletes or you need to implement your own new and delete operators to function more in line with what java is doing. Until you do either one of those things, the comparisons where objects are being allocated and deallocated are invalid. And frankly, I''m still not sure adding System.gc() is even fair, on either side. The reason is, calling System.gc() simply hints that it''s a good idea to collect. There is nothing which requires the collection to take place. So, technically, the call could still be many times faster. On the other hand, I don''t know enough about how they handle their gc-hint logic nor am I aware of exactly how much overhead is involved in the actual collection process. If it occurs too often, the shift in workload may be too unfair. Nonetheless, it''s a point of very serious contention.

Just for kicks, I modified objinst.java with, "if( i%(n/1000) == 0 ) System.gc() ;", on the lines that the C++ code had it''s delete. When I timed it, it was over twice as slow as the C++ code (24+s vs 55+s). Worse, when I ran it with a 1:1 ratio of delete:System.gc(), I simply got tired of waiting, having waited over 5 minutes.

So, basically, I''m not nearly as impressed as I first was. Simplistically, it''s starting to look like a serious apples and oranges comparison. Elsewhere, you can find other examples of just plain bad code. Where again, with correct C++ code, came in about twice as fast at the Java code, whereby, more optimizations were still possible with the C++ code.

So, it looks like we''re seeing a combination of things here. Looks like a combination of bad code, ideal corner cases for java''s hot spot, and invalid comparisons with memory allocation.

Sadly, I''m once again seriously disappointed in java.

anon 06/16/04 05:21:15 AM EDT

I looked at the Ackermann and Fibonacci implementations. If you used better algorithms for these computations you could beat both the C++ and Java versions he has using just about any implementation language you chose.

Both algorithms are practically textbook examples of how a runtime optimizer can beat compile time only optimization of the same algorithm. In both of these situations you compute the exact same value over and over a ridiculous number of times. If you switch to more intelligent algorithms there becomes less runtime optimization opportunities and you have to rely on the optimization of your compiler. Furthermore, you get overhead from the runtime system which will hurt your performance.

I did a brief test of just the computation of Fibonacci(45). Java with -server took 21 seconds, GCC 3.5.0 built from CVS head today with good optimization switches took 15 seconds. Using a better algorithm and implementing in Python, the computation took 0.031 seconds(Hello World in Python took 0.029 seconds).

kcpeppe 06/16/04 05:19:07 AM EDT

Though benchmarks are about the "real world", they themselves are not real world. They must ask a question and then answer it. They cannot take into account all questions, only the question that has been asked. If you are evaluating the validity of a benchmark, you must ask, what was the question that was asked and does the benchmark answer it. The usefulness of a benchmark is a much more subjective measure. If you are not doing any EJB work, then EJB benchmarks would not be very useful to you.

When we say one language is faster than another, what is the question we are asking. We are saying that the compiler can take equivalent source code and compile it to a more reduced/efficient set of machine codes. Languages are expressions or a notation convenience to express your thoughts about how to solve a particular problem. Some languages are more expressive in a particular problem domain. So, how do we define "equivalent" source code. An even more fundamental question is, which problem domains to we consider. We can''t do the all and we really can''t create equivalent pieces of source code, there really can be no definitive answer to this question. One last point, doesn''t faster hardware make this point moot? The problem is that we still cannot develop applications at the rate desired by those sponsoring projects. Certainly the largest gains to be had come from being able to express ourselves efficiently with our most important tool, language.

Kirk Pepperdine
www.javaperformancetuning.com

unwesen 06/16/04 05:16:15 AM EDT

c++ relies heavily on the stl, which, being templates, relies heavily on inlining. gcc switches on inlining of methods with the -O3 option, so -O2 is not necessarily realistic in the first place.

NotZed 06/16/04 04:55:59 AM EDT

Java == modern COBOL. Java works well as a COBOL replacement, a backend application language for boring buisiness apps.

Thats what it should be benchmarked against. Comparing it to C is like comparing apples and oranges.

No thank you 06/16/04 04:55:35 AM EDT

But what is the point in a benchmark of code that no one would ever use in a production environment. Surely knowing the best a language can do is more useful than knowing what kind of performance a naive implementation produces?

It would be interesting to see this shootout re-run with code submitted by people who know the languages well.

TheLittleJetson 06/16/04 04:54:52 AM EDT

However, C++ is faster to LEARN than java - at least if you already know C.

TheLittleJetson 06/16/04 04:51:29 AM EDT

i too, am sick of hearing about how slow java is. i think this claim had a lot more validity back in the day (''98 or so) but now it''s just echoed amongst the idiots. one thing they''ve beaten into us in computer science courses, is that optimization is really only needed in a few critical areas, and the rest of the program wont depend that much on performance.

with today''s absurdly fast computers, just-in-time compiling and runtime optimizations, semi-native environments (OSX''s java, GCJ, JNI, or projects combining multiple languages) -- why must this issue constantly be touted every time java is mentioned?

Thank You 06/16/04 04:49:27 AM EDT

To run benchmarks well a person must be diligent, methodical, and objective. Being a hacker is not a prerequisite. These benchmarks look a lot better than many I''ve seen.

Staf Verhaegen 06/16/04 04:41:41 AM EDT

I read your comment about -O2 but I disagree. I certainly think -O3 should be
added to the pictures:
- When doing a benckmark for C++ against Java one should not try to minimize
the memory footprint for C++ program because Java hapilly takes as much memory
it wants.
- The HotSpot technique does heavy loop enrolling and function inlining at run
time. Comparing this to a compile optimization option that disallows loop
unrolling and function inlining at compile time is not fair IMO.

greets,
Staf.

Swen Tromm 06/16/04 04:07:34 AM EDT

I only had a look into the matrix handling. The handling of the matrix is the typical example of "fine-in-a-text-book-but-do-not-do-so": It is well known that a the handling of pointer to pointer is in respect of performace a bad idea. Any experience coder would build a one-dimensional array and would calulate the pointer "manually"; Soemthing like:

int Matrix::CacululatePos (int iPos_x, int iPos_y)
{
return (iPos_x + iPos_y * iCount_x),
}

chocolatei 06/16/04 04:03:24 AM EDT

Oh dear. The -server switch seems to work nice though.

Dr. Valentin Samko 06/16/04 03:54:48 AM EDT

This article is completely useless and only shows the lack of understanding of C++ by the author. This reminds me of a person who clamed that qbasic is faster than C++ because his program which outputs a few million blank lines works faster in qbasic than in C++.

Why in half of your files you use char* instead of std::string? You still call that C++?

Or take strcat.cpp for example, how usefull is that? std::stringstream, or std::deque would be much more suitable there.

Also, consider heapsort.cpp, do you really think, this is C++? This is pure C. This benchmark has nothing to do with C++. Claiming that Java is faster than C is stupid at least, because JVM was written in C.

matrix.cpp - allocating memory for each matrix row separately only shows you incompetence. This is NOT how it is done by any math library. And again, what does this code have to do with C++, except from std::cout<< call ? This is C code, you really have to change the title of your article to C vs Java, then just remove this article completely, because saying that C is faster than Java is plain stupid (see above).

boxed 06/16/04 03:49:12 AM EDT

Greetings all commentors. Welcome to 1995. Have a nice day.

Iain 06/16/04 02:13:28 AM EDT

The comparison you are trying to make between C++ and Java is not particularly meaningful. At the end of the day, if a VM is faster, then there is nothing to stop you implementing a VM in C++ and running the "C++ program" within that framework. What times would then be valid? The tools you are comparing are very different.

In addition, the C++ code is seriously sub-optimal. To take one example, the "matrix.cpp" is definitely not written for speed. A general purpose C function is being used to repeatedly allocate rows, when a single large block of size "mn" could have been used. For that matter, why is malloc( ) being used at all? (A custom memory manager optimised for small blocks could have been used instead - malloc( ) is slow). If you insist on writing simple C, then at least use memset( ) for the zeromatrix( ) function. Fast template matrix libraries exist for C++ - these could have been used instead - the result would have been shorter, clearer and considerably faster.

Aside from perhaps demonstrating that you know Java far better than you know C++, I don''t think these tests were a particularly useful exercise.

Lachlan Stuart 06/16/04 01:44:33 AM EDT

The simple truth of the matter is that java cannot be faster than C++.
C++ compiles straight to machine code, while Java compiles to bytecode, then is compiled to machine code or interpreted as bytecode by the VM.
There are ways to abuse the languages to make java seem faster, let me list a few:

1. Use useless benchmarks that dont output. Any loop that doesn''t use the output variable for anything will be removed, especially when there is a recompilation/ interpretation stage... see how Via''s mini-ITX computers treat benchmarks, however many C++ compilers let you make useless loops, or only remove useless loops if you inline any involved functions. Most real world apps dont have useless loops, and if they do, they''re inlined enough to be removed.

2. Neglect resource usage. C++ is relatively resource friendly... so nyah :-P
At least give C++ a chance to hog resources by using -O3 after all, java expands loops and inlines functions

3. Isolate unlikely situations.
Even in the 3D games I code, matrix math, hashing, random number generation and string adding only take up a small amount of the processing time.

4. Use external timers.
Applications only start once. If you dont use a timer inside your program, you''re going to time start up and close time, which gives java an advantage, assuming it doesn''t close itself between every startup

Though I could list many more, im quite sure only the first example made a considerable impact.

What you must realize is that C++ and java have much different targets, its rare that you will encounter a situation where both are equally suitable. No matter what, unless you forget to turn debug compile off, C++ will beat java in real world situations on the performance front...Try praising something else about java...like its security, or ease of programming

Tom 06/16/04 01:43:29 AM EDT

First i feel bad for any programmer who''s manager just read this obviously erroneous benchmark. Ignoring the obviously bad C++ code there are many issues with these tests. The Intel and Microsoft compilers are the best out there and run circles around G++. Since the Java code isn''t really deleting memory ( as it''s GC''d ) then none of the C++ tests should try to delete memory. None of these tests look at resource usage... how much CPU time was used? How much memory was used? In both instances you''ll find most C++ compilers superior to any Java compiler.

It''s in the title of the original article... This is nothing, but a Java lovefest.

Howe 06/15/04 10:21:07 PM EDT

This is one of the most ridiculous article I ever read.
Besides being slow, and everybody know that, the JVM is so resource angry that it will make your whole system run slower in order to try to make the the program run "fast". Just observe how much CPU is eaten running a single applet. that does little. Normal users usually have other programs running in their computers and the CPU degradation isn''t acceptable. Oh, you just run the JVM and your applet in it ? You''re lucky then, stay in your little world, thank you.

I won''t even check how you run your benchmarks and methodology: this article was already a big, fat waste of my time. Go study a bit more and don''t ever write again, please.

Jose 06/15/04 08:03:35 PM EDT

You must be joking? I''ve seen Java slow down to a snails pace on the simplest of tasks. It''s fine if you want to continuously "System.out.println("Hello Beans");", but anything that involves immutable objects and the dodgy garbage collector, its a different situation.

I can only say this. I''m judging based on my own experiences. I''ve developed in Java several times and can see it''s good points, but I would never accept it as reliable or optimized. All including its irrational treatment of floating point calculations and even using Netbeans as an IDE was so slow, I ended up using a Delphi written IDE called GEL. Admittedly, using a JVM slows things considerably, but nothing as to be so brave to say its on par with any C++ compiler. And, besides, isn''t Java meant to be sandboxed, hence there is always a form of JVM? Suspicious...

Finally, was this tested taking into account you can use inline assembly in GCC/C++?

M. Jung 06/15/04 08:01:16 PM EDT

Those "benchmark" results make me laught.

The binaries in the archive are not stripped and the C++ code is pretty ineffective. You didn''t proof that java was faster than C++ but you showed that you didn''t invest any time in reading the G++ manpage and know nothing about C++.

This must be a joke - let''s hope nobody believes this nonsense.

Chris 06/15/04 07:37:28 PM EDT

Lets face it, gcc is NOT an overly optimising compiler - so comparing it with a highly optimizing java compiler and interpreter is not a comparison of languages, mearly a comparison of tools.

Try doing your comparison using a vendor C++ compiler for a particular platform (such as SunONE, etc). You will then see C++ code that is orders of magnitude more efficient.