Monday, November 4, 2013

Static Analysis tools: Evaluating PMD and FindBugs

Software developers use a large number of tools to help them in their day to day tasks and coding activities. Static analysis tools are most commonly used tools which help verify the code and find potential bugs.
The two most popular static analysis tools for Java are PMD and FindBugs. According to code quality tools review, 2013, FindBugs is used by 33% of the respondents and PMD is used by 23% of the respondents. It is worthwhile to evaluate their effectiveness and find out the best practices in using them, since these are the most commonly used tools.

Introduction to PMD

Basic features
PMD is a rule based static analysis tool that works on source code files. It comes with a set of rules, grouped into rulesets. Each rule has the following properties:
  • Priority
  • Rule Set
  • Rule implementation class
  • Message

Priority
Rules are also categorized as per their severity into one of five classes, each represented by a default color:
  1. Blocker (red)
  2. Critical (cyan)
  3. Urgent (green)
  4. Important (pink)
  5. Warning (blue)
You can also change the severity (and color) of a particular rule based on user preference.

Rule Set
Currently, PMD supports around 33 rulesets which contain approximately 313 rules such as:
  1. Basic (java): This ruleset checks agains good practices which should be followed for all Java projects.
  2. Basic XML: Contains a collection of good practices for XML.
  3. Migration: Contains rules about migrating from one JDK version to another.
  4. Naming: Checks against conventions for naming variables, methods and identifiers.
  5. Optimization: Checks against optimization best practices.
  6. Security Code Guidelines: These rules check against the security guidelines from Sun.
  7. Strict Exceptions: Check for bugs or bad code in exception handling.

PMD also comes with its default ruleset which includes all the rules in the Basic (java), Import Statements ruleset and Unused Code ruleset which gives a total of 34 rules. A user can configure his own custom ruleset borrowing rules from any or all of these rulesets. In fact, a user can write his own rules too. The rule-ruleset association can also be changed, for example if you believe a particular rule is more appropriate in some other ruleset, you can add it to that ruleset.

Working
PMD is a source code analysis tool. This makes it most appropriate for finding problems with code style, syntax and violations of coding standards and best practices.
  1. User selects the rules or rulesets based on his preferences. User runs PMD on a source file or a package(s). Alternately, PMD can also be configured to run automatically.
  2. PMD parses the source code and generates an abstract syntax tree of the files of the source.
  3. Each of the rulesets traverses the abstract syntax tree.
  4. Each ruleset reports all the violations it found. This data is then consolidated into a report which the user can view. PMD allows you to view this report in different formats such as CSV, XML, HTML, etc.

Use
PMD can be used in different ways based on user's comfort:
  1. From the command line
  2. As a plugin integrated into eclipse (also as a Maven plugin)
  3. PMD comes with a CPD GUI (Copy Paste Detector) which finds repeated blocks of code. The minimum number of lines to consider as a block is configurable.
PMD comes with a lot of configurable settings and features and it is evident that user friendliness and flexibility are a key focus.

Introduction to FindBugs

FindBugs is different since it works on byte code. FindBugs uses BCEL (Byte Code Engineering Library) which helps it analyze java bytecode. BCEL itself comes with a byte code verifier named JustIce. FindBugs can also be used to find problems in compiled classes and jar files. FindBugs finds potential problems matching a bytecode against different bug patterns.
There is a bug detector defined for each bug pattern. Each bug detector has associated with it:
  • Detector Id
  • Pattern
  • Speed
  • Category
Bug categories
Each bug detector finds a particular type of bug which belongs to one of the following categories:
  • Malicious code vulnerability
  • Dodgy code
  • Bad practice
  • Correctness
  • Internationalization
  • Performance
  • Security
  • Multithreaded Correctness
  • Experimental
Bug Rank
Bugs are ranked according to their severity as:
  1. Scariest (red)
  2. Scary (orange)
  3. Troubling (yellow)
  4. Of Concern (blue)
Confidence level
Each bug detector has a confidence level associated with it. This indicates how much confidence the detector has that the bug warnings found by it are truly bugs. You can configure at what confidence level you want the bug detectors to show you the results. This means, if you are nervous about the code and want FindBugs to show all warnings, you can configure the detectors to show bugs at all confidence levels. This configuration is applicable to the selected bug categories only. This helps the user focus on the main aspect of the project for e.g. You can configure FindBugs to find security related bugs at all confidence levels. Confidence level settings can also be used to ignore warnings at lower confidence level so as to decrease the number of false positives reported.
Use
FindBugs can be used as an Eclipse Plugin, GUI tool and from the command line. The warnings detected can be reported in XML format.

What we found most interesting about the two tools

PMD:
  1. PMD supports a very large number rules covering broad range of bug problems.
  2. PMD is highly configurable and flexible. This makes it both good and bad. Good in the sense that the user can use the tool according to his comfort and requirement. However, this becomes overwhelming sometimes. We discovered, what bugs get fixed and how many false positives you get depend highly on the configurations and the users understanding of the configurations and the project.
  3. PMD has a easy to use rule designer interface that allows you to add your own new custom rule.

FindBugs:
  1. FindBugs finds lesser false positives and you can set the confidence level at which the detectors should work. This makes it less overwhelming to go through the bug warnings.
  2. FindBugs has a new feature that connects your bug warning reports to the Cloud.
  3. FindBugs also comes with its own data mining capabilities that can be used to mine bug reports over several project versions and timelines. This makes it very handy tool for fine tuning the configuration of the tool and analyzing developer mistakes over time.


Our experience with the tool

JFreeChart
  • JfreeChart is an open source Java library that helps developers create a large variety of high quality charts. JfreeChart is the most popular chart library.
  • It started as an open source project since 2000. Since it has been in existence for over 13 years, it is rich in version history.
  • Since it is popularly used, the expectation is that most of the relevant bugs are fixed by the developers over several versions of the tool.
  • We found it relatively easy to configure the project.
Approach
  • We ran the tools on two versions of the tool. We selected version 1.0.0 and version 1.0.16 (latest). These versions were sufficiently apart for the major bugs to have been fixed over the versions.
  • We first ran only the tools with default configurations since we believe that the default configuration must be most commonly used by developers. The default setting must find the most relevant bugs.
  • Any bug that got fixed from the older versions in the newer versions, must be a true positive.
  • Only classes that were present in both the versions were considered since additional classes indicate either new features or deprecation.
Running PMD on JfreeChart
For PMD Basic Ruleset, the results were as follows:

JFreeChart v1.0.0
Total warnings : 595
Total distinct warnings : 25

Warning
Count
Avoid unused constructor parameters such as shapesVisible.
1
Avoid unused imports such as org.jfree.chart.renderer.category.LevelRenderer
1
Avoid unused method parameters such as g2.
1
Avoid unused constructor parameters such as outlinePaint.
1
Avoid unused method parameters such as orientation.
1
Unnecessary use of fully qualified name java.util.Date due to existing import java.util.Date
2
Avoid unused constructor parameters such as linesVisible.
1
Avoid unused local variables such as b.
2
Avoid unused imports such as org.jfree.chart.axis.CategoryAxis
1
These nested if statements could be combined
45
Avoid unused local variables such as contentConstraint.
1
Avoid unused private fields such as flag.
1
Avoid unused constructor parameters such as outlineStroke.
1
Avoid modifiers which are implied by the context
492
Unnecessary use of fully qualified name java.util.ArrayList due to existing import java.util.ArrayList
16
Unnecessary use of fully qualified name java.awt.Color.lightGray due to existing import java.awt.Color
1
Unnecessary use of fully qualified name java.text.SimpleDateFormat due to existing import java.text.SimpleDateFormat
1
Avoid unused imports such as org.jfree.chart.renderer.category.LayeredBarRenderer
2
Avoid unused method parameters such as dataArea.
1
Avoid unused imports such as org.jfree.chart.plot.CategoryPlot
1
Avoid unused imports such as org.jfree.data.category.DefaultCategoryDataset
1
Avoid unused imports such as org.jfree.chart.plot.Plot
1
Ensure you override both equals() and hashCode()
18
Avoid unused constructor parameters such as percent.
1
Avoid unused local variables such as image.
1


JFreeChart v1.0.16
Total warnings : 742
Total distinct warnings : 21

Warning
Count
Avoid unused method parameters such as orientation.
2
Avoid unused method parameters such as g2.
1
Avoid unused imports such as org.jfree.data.general.DatasetUtilities
6
Avoid unused local variables such as datasetIdx.
1
These nested if statements could be combined
65
Avoid unused local variables such as section.
1
Avoid unused private fields such as event.
1
Avoid modifiers which are implied by the context
561
Unnecessary use of fully qualified name java.util.ArrayList due to existing import java.util.ArrayList
41
Avoid unused local variables such as rect.
1
Unnecessary use of fully qualified name java.awt.Color.lightGray due to existing import java.awt.Color
1
Avoid unused constructor parameters such as cloneData.
2
Unnecessary use of fully qualified name java.text.SimpleDateFormat due to existing import java.text.SimpleDateFormat
1
Avoid unused constructor parameters such as labelCount.
1
Avoid unused method parameters such as dataArea.
1
Avoid modifying an outer loop incrementer in an inner loop for update expression
2
Ensure you override both equals() and hashCode()
50
Avoid unused local variables such as description.
1
Avoid unused local variables such as count.
1
Avoid unused imports such as java.util.Objects
1
Avoid unused method parameters such as ignoreThisDummyArgument.
1

What we are looking for in this data:
  1. Warnings that are common to both versions, imply either:
  • If their count has decreased from the previous version, the warning is important for this project. Hence, some of them were fixed. These type of warnings are recurring in this project and need to be looked at carefully.
  • If the count has not decreased, it means these warnings are not correct, especially in the context of the project since the developers did not find them useful/important enough to fix them.
  1. Some of the “bugs” detected could be delibrate or for testing purpose, conciously placed by the developer. For example “Avoid unused method parameters such as ignoreThisDummyArgument.” These warnings need not be considered.
  2. Increase in warning count doesn't necessarily mean poorer quality of code. It could mean more features have been added to the code. However, warnings per kloc of code or warnings per method, need to be closely looked at.

Running FindBugs on JFreeChart
Information
v1.0.0
v1.0.16
Total bugs
163
133
By Bug Rank
Scariest
1
0
Scary
38
15
Troubling
117
109
Of concern
7
9
By Category
Bad Practice
107
103
Correctness
40
15
Dodgy Code
8
10
Multithreaded correctness
8
5
By Confidence
High Confidence
112
119
Normal Confidence
21
44

What we are looking for in this data:
  1. It is clearly evident that the count of most of the warnings has gone down over the next version. This implies that the particular bug category is relevant to the project. Hence, FindBugs must always be configured to detect this category of bugs for this project.
  2. FindBugs detector confidence levels have gone up over the versions of the project. This implies more obvious bugs are being introduced into the code.

We plan to consolidate and derive more findings from such data. We would like to suggest a plan for configuring rulesets/detectors which will be most relevant for the project based on previous bug fixes. This way one could derive maximum efficiency from the tool and also have to look at lesser false positives.

Nachiket V. Naik
Computer Science
North Carolina State University
Raleigh, US
nvnaik@ncsu.edu

Preeti Satoskar
Computer Science
North Carolina State University
Raleigh, US
pysatosk@ncsu.edu

References

Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)