Have you ever compared the static analysis tools Klocwork and Findbugs?

Question

We are using Klocwork as a static analysis tool.

Klocwork is a commercial tool and has many advantages but also has limitations like false-positives.

I wonder who has ever compared Klocwork with other open source tools such as Findbugs.

Generally, commerical tools is known to be more reliable than open source tools.

But I think that Klocwork has also a few reliable issues in specific business domain such as android.

Can you say that Klocwork is superior to other open source tools, especially Findbugs in aspects of false positives and false negatives?

Answer 1

I have done a comparison between commercial and opensource static code analysis tools (SCAT) a few years back. Klocwork was one of them. To make a long story short, the result was that in the java environment the commercial tools didn't provide the (additional) value necessary to justify their price. In the number of bugs found, findbugs was much better than any of those tools. Only coverity found some bugs which none of the other tools found and had the lowest FPR. On the other side coverity didn't detect many bugs findbugs did find - so for us all the tools didn't justify the money involved. And if configured right, findbugs also provides a very low FPR. In a more recent version coverity included the possibility to integrate findbugs into their solution. Mhhmmm so why did they do that? :-)

There are however scenarios e.g. when using other programming languages or inter-programming language scans (e.g. your codebase includes other programming languages (like C, C++, C# ...)) or you need some of the additional functionalities provided by those tools - then it could be worth looking into commercial tools. But you can make up your own mind because every solution can be tested. Go to their webpage and download/request a trial and try for yourself (maybe in the meantime things changed?).

I checked out the following tools:

Commercial:

1. Coverity
2. Klocwork
3. Parasoft
4. Cast

OpenSource:

• Findbugs
• PMD
• Checkstyle (although not a real SCAT but more like a style checker)

So in the end what did we do? We installed the free sonar server which combines many tools like findbugs, pmd, checkstyle, cobertura and the like. With that we got a free solution which is in many aspects better than the commercial tools available. And if I would need to go the commercial route I would probably take a close look at coverity (and never look at CAST again).

[Update] Regarding your question about the performance - findbugs is able to scan huge codebases without a problem (if you experience problems then post it on the findbugs mailing list and they will help you). I remember back then findbugs was also one of the quickest tools. Coverity for instance needed nearly 2h to finish whereas findbugs was done in less then 10minutes. The other tools where in between those numbers.

Regarding the scanning of C and C++ code you may look at Splint and cppcheck. There are also some other helpful threads discussing that topic. But as mentioned since findbugs is only for java and you want to detect inter-programming language problems commercial tools may have an advantage. So in the end since findbugs is for free - just run it on your codebase and see what happens for yourself - your project can only benefit from it! Then fix the bugs found and afterwards do a trial with some of the commercial tools - and depending on the amount of bugs they find and the price you will pay, draw your own conclusion. To give you an example: One of the tools costs 'Lines of code' /divided by 10 (in $). But it may be that in your case you get a discount ;-)

[UPDATE II] Found an interesting master thesis discussing this topic. The main part is about findbugs but it also mentions klockwork and coverity.