How do commercial Java static analysis tools compare with the free ones?
I'm familiar with a handful of the free static analysis tools available for Java, such as FindBugs and PMD. What I'd like to know is how the commercial products such as Klocwork and Coverity stack up against these. What are their strengths and weaknesses?
Answer
We use a suite of open source and commercial static analysis tools. The different tools find different kinds of bugs and some are tuned for lower false positive rates, at the expense of possibly missing some real problems.
In my experience, Findbugs does a good job of finding real problems, especially if you focus on Correctness errors as their team suggests. Recently the developers of Findbugs have added some basic security vulnerability checks as well. Coverity has a low false positive rate especially if you don't turn on their experimental checkers, and Coverity Prevent includes a good tracking database for trend/cluster analysis. I am not convinced yet that their threading checkers (static or dynamic) work - at least they haven't found anything interesting for us. Klocwork Developer for Java returns higher false positives, but we find they have the strongest security checking of these tools. So it depends on whether your priority is quality checking (Findbugs, Coverity) or security vulnerability analysis (Klocwork, or Fortify). Some of our developers also use PMD to support source code reviews, as it helps with general code cleanup.
A recent project conducted with NIST called "SATE: Static Analysis Tool Exposition" reviewed a wide variety of different tools and their underlying approaches. https://samate.nist.gov/index.php/SATE.html and other references to this project such as at OWASP. The general finding is that different tools have different strengths and weaknesses, so use more than one if you want to do a thorough job.