Static Analysis tools: Evaluating PMD and FindBugs

Software developers use a large number of tools to help them in their day to day tasks and coding activities. Static analysis tools are most commonly used tools which help verify the code and find potential bugs.

The two most popular static analysis tools for Java are PMD and FindBugs. According to code quality tools review, 2013, FindBugs is used by 33% of the respondents and PMD is used by 23% of the respondents. It is worthwhile to evaluate their effectiveness and find out the best practices in using them, since these are the most commonly used tools.

Introduction to PMD

Basic features

PMD is a rule based static analysis tool that works on source code files. It comes with a set of rules, grouped into rulesets. Each rule has the following properties:

• Priority
• Rule Set
• Rule implementation class
• Message

Priority

Rules are also categorized as per their severity into one of five classes, each represented by a default color:

1. Blocker (red)
2. Critical (cyan)
3. Urgent (green)
4. Important (pink)
5. Warning (blue)

You can also change the severity (and color) of a particular rule based on user preference.

Rule Set

Currently, PMD supports around 33 rulesets which contain approximately 313 rules such as:

1. Basic (java): This ruleset checks agains good practices which should be followed for all Java projects.
2. Basic XML: Contains a collection of good practices for XML.
3. Migration: Contains rules about migrating from one JDK version to another.
4. Naming: Checks against conventions for naming variables, methods and identifiers.
5. Optimization: Checks against optimization best practices.
6. Security Code Guidelines: These rules check against the security guidelines from Sun.
7. Strict Exceptions: Check for bugs or bad code in exception handling.

PMD also comes with its default ruleset which includes all the rules in the Basic (java), Import Statements ruleset and Unused Code ruleset which gives a total of 34 rules. A user can configure his own custom ruleset borrowing rules from any or all of these rulesets. In fact, a user can write his own rules too. The rule-ruleset association can also be changed, for example if you believe a particular rule is more appropriate in some other ruleset, you can add it to that ruleset.

Working

PMD is a source code analysis tool. This makes it most appropriate for finding problems with code style, syntax and violations of coding standards and best practices.

1. User selects the rules or rulesets based on his preferences. User runs PMD on a source file or a package(s). Alternately, PMD can also be configured to run automatically.
2. PMD parses the source code and generates an abstract syntax tree of the files of the source.
3. Each of the rulesets traverses the abstract syntax tree.
4. Each ruleset reports all the violations it found. This data is then consolidated into a report which the user can view. PMD allows you to view this report in different formats such as CSV, XML, HTML, etc.

Use

PMD can be used in different ways based on user's comfort:

1. From the command line
2. As a plugin integrated into eclipse (also as a Maven plugin)
3. PMD comes with a CPD GUI (Copy Paste Detector) which finds repeated blocks of code. The minimum number of lines to consider as a block is configurable.

PMD comes with a lot of configurable settings and features and it is evident that user friendliness and flexibility are a key focus.

Introduction to FindBugs

FindBugs is different since it works on byte code. FindBugs uses BCEL (Byte Code Engineering Library) which helps it analyze java bytecode. BCEL itself comes with a byte code verifier named JustIce. FindBugs can also be used to find problems in compiled classes and jar files. FindBugs finds potential problems matching a bytecode against different bug patterns.

There is a bug detector defined for each bug pattern. Each bug detector has associated with it:

• Detector Id
• Pattern
• Speed
• Category

Bug categories

Each bug detector finds a particular type of bug which belongs to one of the following categories:

• Malicious code vulnerability
• Dodgy code
• Bad practice
• Correctness
• Internationalization
• Performance
• Security
• Multithreaded Correctness
• Experimental

Bug Rank

Bugs are ranked according to their severity as:

1. Scariest (red)
2. Scary (orange)
3. Troubling (yellow)
4. Of Concern (blue)

Confidence level

Each bug detector has a confidence level associated with it. This indicates how much confidence the detector has that the bug warnings found by it are truly bugs. You can configure at what confidence level you want the bug detectors to show you the results. This means, if you are nervous about the code and want FindBugs to show all warnings, you can configure the detectors to show bugs at all confidence levels. This configuration is applicable to the selected bug categories only. This helps the user focus on the main aspect of the project for e.g. You can configure FindBugs to find security related bugs at all confidence levels. Confidence level settings can also be used to ignore warnings at lower confidence level so as to decrease the number of false positives reported.

Use

FindBugs can be used as an Eclipse Plugin, GUI tool and from the command line. The warnings detected can be reported in XML format.

What we found most interesting about the two tools

PMD:

1. PMD supports a very large number rules covering broad range of bug problems.
2. PMD is highly configurable and flexible. This makes it both good and bad. Good in the sense that the user can use the tool according to his comfort and requirement. However, this becomes overwhelming sometimes. We discovered, what bugs get fixed and how many false positives you get depend highly on the configurations and the users understanding of the configurations and the project.
3. PMD has a easy to use rule designer interface that allows you to add your own new custom rule.

FindBugs:

1. FindBugs finds lesser false positives and you can set the confidence level at which the detectors should work. This makes it less overwhelming to go through the bug warnings.
2. FindBugs has a new feature that connects your bug warning reports to the Cloud.
3. FindBugs also comes with its own data mining capabilities that can be used to mine bug reports over several project versions and timelines. This makes it very handy tool for fine tuning the configuration of the tool and analyzing developer mistakes over time.

Our experience with the tool

JFreeChart

• JfreeChart is an open source Java library that helps developers create a large variety of high quality charts. JfreeChart is the most popular chart library.
• It started as an open source project since 2000. Since it has been in existence for over 13 years, it is rich in version history.
• Since it is popularly used, the expectation is that most of the relevant bugs are fixed by the developers over several versions of the tool.
• We found it relatively easy to configure the project.

Approach

• We ran the tools on two versions of the tool. We selected version 1.0.0 and version 1.0.16 (latest). These versions were sufficiently apart for the major bugs to have been fixed over the versions.
• We first ran only the tools with default configurations since we believe that the default configuration must be most commonly used by developers. The default setting must find the most relevant bugs.
• Any bug that got fixed from the older versions in the newer versions, must be a true positive.
• Only classes that were present in both the versions were considered since additional classes indicate either new features or deprecation.

Running PMD on JfreeChart

For PMD Basic Ruleset, the results were as follows:

JFreeChart v1.0.0

Total warnings : 595

Total distinct warnings : 25

Warning	Count
Avoid unused constructor parameters such as shapesVisible.	1
Avoid unused imports such as org.jfree.chart.renderer.category.LevelRenderer	1
Avoid unused method parameters such as g2.	1
Avoid unused constructor parameters such as outlinePaint.	1
Avoid unused method parameters such as orientation.	1
Unnecessary use of fully qualified name java.util.Date due to existing import java.util.Date	2
Avoid unused constructor parameters such as linesVisible.	1
Avoid unused local variables such as b.	2
Avoid unused imports such as org.jfree.chart.axis.CategoryAxis	1
These nested if statements could be combined	45
Avoid unused local variables such as contentConstraint.	1
Avoid unused private fields such as flag.	1
Avoid unused constructor parameters such as outlineStroke.	1
Avoid modifiers which are implied by the context	492
Unnecessary use of fully qualified name java.util.ArrayList due to existing import java.util.ArrayList	16
Unnecessary use of fully qualified name java.awt.Color.lightGray due to existing import java.awt.Color	1
Unnecessary use of fully qualified name java.text.SimpleDateFormat due to existing import java.text.SimpleDateFormat	1
Avoid unused imports such as org.jfree.chart.renderer.category.LayeredBarRenderer	2
Avoid unused method parameters such as dataArea.	1
Avoid unused imports such as org.jfree.chart.plot.CategoryPlot	1
Avoid unused imports such as org.jfree.data.category.DefaultCategoryDataset	1
Avoid unused imports such as org.jfree.chart.plot.Plot	1
Ensure you override both equals() and hashCode()	18
Avoid unused constructor parameters such as percent.	1
Avoid unused local variables such as image.	1

JFreeChart v1.0.16

Total warnings : 742

Total distinct warnings : 21

Warning	Count
Avoid unused method parameters such as orientation.	2
Avoid unused method parameters such as g2.	1
Avoid unused imports such as org.jfree.data.general.DatasetUtilities	6
Avoid unused local variables such as datasetIdx.	1
These nested if statements could be combined	65
Avoid unused local variables such as section.	1
Avoid unused private fields such as event.	1
Avoid modifiers which are implied by the context	561
Unnecessary use of fully qualified name java.util.ArrayList due to existing import java.util.ArrayList	41
Avoid unused local variables such as rect.	1
Unnecessary use of fully qualified name java.awt.Color.lightGray due to existing import java.awt.Color	1
Avoid unused constructor parameters such as cloneData.	2
Unnecessary use of fully qualified name java.text.SimpleDateFormat due to existing import java.text.SimpleDateFormat	1
Avoid unused constructor parameters such as labelCount.	1
Avoid unused method parameters such as dataArea.	1
Avoid modifying an outer loop incrementer in an inner loop for update expression	2
Ensure you override both equals() and hashCode()	50
Avoid unused local variables such as description.	1
Avoid unused local variables such as count.	1
Avoid unused imports such as java.util.Objects	1
Avoid unused method parameters such as ignoreThisDummyArgument.	1

What we are looking for in this data:

1. Warnings that are common to both versions, imply either:

• If their count has decreased from the previous version, the warning is important for this project. Hence, some of them were fixed. These type of warnings are recurring in this project and need to be looked at carefully.
• If the count has not decreased, it means these warnings are not correct, especially in the context of the project since the developers did not find them useful/important enough to fix them.

2. Some of the “bugs” detected could be delibrate or for testing purpose, conciously placed by the developer. For example “Avoid unused method parameters such as ignoreThisDummyArgument.” These warnings need not be considered.
3. Increase in warning count doesn't necessarily mean poorer quality of code. It could mean more features have been added to the code. However, warnings per kloc of code or warnings per method, need to be closely looked at.

Running FindBugs on JFreeChart

Information	v1.0.0	v1.0.16
Total bugs	163	133
By Bug Rank
Scariest	1	0
Scary	38	15
Troubling	117	109
Of concern	7	9
By Category
Bad Practice	107	103
Correctness	40	15
Dodgy Code	8	10
Multithreaded correctness	8	5
By Confidence
High Confidence	112	119
Normal Confidence	21	44

What we are looking for in this data:

1. It is clearly evident that the count of most of the warnings has gone down over the next version. This implies that the particular bug category is relevant to the project. Hence, FindBugs must always be configured to detect this category of bugs for this project.
2. FindBugs detector confidence levels have gone up over the versions of the project. This implies more obvious bugs are being introduced into the code.

We plan to consolidate and derive more findings from such data. We would like to suggest a plan for configuring rulesets/detectors which will be most relevant for the project based on previous bug fixes. This way one could derive maximum efficiency from the tool and also have to look at lesser false positives.

Nachiket V. Naik

Computer Science

North Carolina State University

Raleigh, US

nvnaik@ncsu.edu

Preeti Satoskar

Computer Science

North Carolina State University

Raleigh, US

pysatosk@ncsu.edu

References

• Information about Code Quality Review, 2013: http://zeroturnaround.com/rebellabs/code-quality-tools-review-for-2013-sonar-findbugs-pmd-and-checkstyle/
• PMD rulesets: http://pmd.sourceforge.net/pmd-5.0.5/rules/index.html
• PMD rules index: http://www.ing.iac.es/~docs/external/java/pmd/rules/
• Customizing PMD: http://www.javacodegeeks.com/2012/10/customize-pmd-in-eclipse-with-your-own.html
• FindBugs Fact Sheet: http://findbugs.sourceforge.net/factSheet.html
• BCEL homepage: http://commons.apache.org/proper/commons-bcel/
• FindBugs homepage: http://findbugs.sourceforge.net/

Static Analysis tools - Evaluating their Effectiveness

This blog post is dedicated to discussing about static analysis tools in software and approaches on how to evaluate their effectiveness.

An introduction to static analysis tools, how they work and their pros and cons can be found here:

Show Notes:

• Comparing bug finding tools with reviews and tests
• Finding Bugs is Easy
• Logical Foundations for Static Analysis: application to binary static analysis for security
• A comparison of bug finding tools for Java

Here is a screencast that shows the working of two popular tools, FindBugs and PMD.

Why to evaluate static analysis tools?

Now, there are quite a few static analysis tools available for each language. Each of these tools finds a subset of all the bugs and there is some overlap in the bugs found. No one tool completely subsumes all the other and hence there is no clear winner. In such a situation, it becomes necessary to evaluate the effectiveness of these tools.

A number of metrics have been discussed for evaluating the effectiveness of bug finding tools:

• False Positives (FP): A false positive is a bug warning that isn't really a bug.
• True Positives (TP): A true positive is a bug warning that correctly finds a bug and leads to a fix.
• True Negative (TN): A true negative is any line of code where no bug warning was reported and that line was found to be bug free.
• False Negative (FN): A false negative is a line of code for which no bug warning was reported but bugs were later found.

This leads to the definition of other related metrics such as :

• False Positive Rate (FPR) = FP/ (FP+TP) OR FP/(FP+TN) (an approach that uses this)
• Precision = TP/ (FP+TP) OR Precision = 1 - FPR 

Another approach argues that they do not consider project specific importance of bugs.One approach considers that the bug warning which eventually leads to fixes by the developer are true bugs. Hence we can use this information about the lifetime of the bug warning. The lifetime of the bug warning is the number of fix versions over which the bug warning is not fixed. The intuition behind this is that if a bug is not fixed over several fix versions, then the bug warning is not very relevant. In a case study using this approach, relevance rates superior to 50% were found for FindBugs and 10.1% for PMD.

One discussion argues that these metrics do not consider priority of bugs. This means tools which detect trivial bugs will be rated higher. Some categories of bugs found are of lower significance to the developer:

i. True But Low Impact Defects: Bugs that are easy to find but will not cause big problems. If any of the above metrics are used, the tools that find easy to spot but non-important bugs will be rated higher.

ii. Deliberate errors: Intentionally throwing a runtime exception.

iii. Masked errors: Errors conditions that will never occur.

iv. Already Doomed: Sometimes,an error can occur only in a situation where the computation is already doomed, and throwing a runtime exception is not significantly worse than any other behavior that might result from fixing the defect.

v. Testing code: In testing code, developers will often try things to break the system.

vi. Unimportant case or lower importance functions.

The argument against fixing the above bugs is that it consumes resources such as the developer time in return for very less value.

So we can see that quite a few methods for evaluating the tools have been proposed and this seems to be a very active field of research. Evaluating tools will help in improving tools and selection of tools for specific project needs.

Our screencast here discusses on different approaches to use static analysis tools more effectively.

Nachiket V. Naik

Computer Science

North Carolina State University

Raleigh, US

nvnaik@ncsu.edu

Preeti Satoskar

Computer Science

North Carolina State University

Raleigh, US

pysatosk@ncsu.edu

Top 10 stereotypical people you find in every class

When I look back, thinking about the people I've met, I often find myself thinking mostly about the people I spent most of my time with. I remember the close friends, some of which I am in touch with intermittently. Some of them I have lost contact with. 

But some of the times, I wonder about random people that I didn't know that well. People who were just names and faces in a class, but I never really interacted with them. Sometimes I wonder what happened to them, where did they go and what might they be doing now. You see, wherever we go, we just attach our initial impressions of the people we meet and then ignore the ones we thought aren't someone who are like us. We divide ourselves very easily and form associations with the ones we find closest to our comfort zone. After that, there is rarely any effort to move out of that group and interact with someone else. We just stop caring to know who or what other people are.

Although all people are different, our initial impressions tend to have a pattern of finding stereotypes in people. In fact, I think we like it if we find someone we can put into a stereotype. If we find someone who doesn’t fit into any stereotype or seen pattern, it just messes our mind. 

For example, if you see an amazingly beautiful girl but she turn out to very quiet, approachable and extremely nerdy, it just messes our systems.

Anyways, I present to you the list of stereotypical people we find everywhere in college and work (and what happens to them, generally):

1) The "I'll do it": This is the guy who’ll come and break the ice with everyone himself. This one will take all initiatives for the entire class, organize stuff, be the initial contact for everyone, sign up for everything and let everyone know about things they should know and will be handing off unwanted information. This one overshadows everyone and makes people nervous about his preparedness. A huge show-off, you sometimes wonder whether all he does is only so that he can show off. Oh and he has done all the coolest stuff and knows everything, atleast that is what he says.

What will happen: This guy will only tower for the initial phase, and fade away. In sometime, people will just forget him and he’ll take the backseat. He’ll have a pretty normal life.

2) The “coy beautiful girl”: This girl is the one everyone notices the most on the first day. She is extremely beautiful, and you cant help but stare. She stays mostly with the girls and all boys are eyeing her secretly. Girls are jealous of her and boys spellbound. You have searched her on facebook, the minute you are back from first day class.

What will happen: She is wooed by a lot of guys and that gets to her head. She starts becoming arrogant, dates a few jocks. Later in life she tries her hand at modelling with not much success, does an MBA to keep her parents from looking for guys for her, for a while. Then, she gets married to some successful engineer (good going for him because he couldn’t have dated her if he was the last man on earth), just before she starts putting on weight from all the binge eating.

3) The “compulsive joker”: This guy can’t stop making jokes, forwarding SMSes, posting random stuff, commenting on your facebook updates. He simply can’t. To make things worse, it’s not funny, it’s just cheesy. He can’t go a moment in seriousness; he has to make it funny. He is doing it for the audience only and making everyone laugh is a job he takes way too seriously.

What will happen:  heartbreak. No matter how many girls say they like someone who can make them smile, they look for a lot of other things too. He often finds himself lonely when he needs friends. He vows to keep to himself. Concentrates on his work, does well, makes a nice friend circle at work later on in life. His family loves him.

4) The “small town boy/girl”: The bespectacled boy/girl who doesn’t talk much. She/he wears clothes way beyond his age, mostly formals. Stays on the last bench, is hesitant to speak in English, and finds friendship only with the guys and girls like him/her only. Takes down all notes sincerely and doesn’t miss a day in class. Doesn’t show up for class trips.

What will happen: They really bloom later on. They do well at work and make a group of friends at work that stick and is respected amongst peers. They are first to buy a nice phone, house, car, etc. They pick up some hobby like  photography or trekking.

5) The “activist”: The guy/girl who is always out to make a change. Always protesting, sometimes nobody knows what. They’ll meet you and always talk about some campaign, some blood drive, etc. They’ll spend most of their time in organizing such things. You get bored talking to their complaints, avoid them.

What will happen: Lack of attendance and poor grades. However, they are the only ones who keep in touch with the college later on and become contributing alumni. They take organizing stuff as a talent and do the same at work. Everyone now has their number in their contacts.

6) The “art circle girl”: Always organizing some event. They take joy in blowing ballons, filling up the notice boards, discussing colour, etc. They practically live on the college stairs. At work, they spend a lot of time, decorating their cubicle, organizing events, taking pictures, etc.

What will happen: She’ll get into mass media probably, do an MBA, travel the world and then settle down and have a happy family life. You will probably run into her at some handicrafts expo.

7) The “dude”/ “model”: All they talk about is gym. Gym is their hobby, talent, passion and only interest. And they say “I am good at it”. Life is all about branded clothes and emulating filmstars. They live in their own bubble where they think they are stars. They love making fun of the puny kids and is a bully to everyone. Has an IQ level of a fifth grader, spends more time in front of the mirror and is secretly a homophobe.

What will happen: 30 pushups later, he’ll/she’ll find a job get married with much fanfare and lead an ordinary life.

8) The “girl next door”: She has loads and loads of friends. Easily approachable, prominent and funny, she makes social interactions easy. She can strike a conversation with anyone. She is friends with everyone, nerds and jocks alike. She acts against the snobs and is loved by all.

What will happen: She will later bloom and her average looks will transform to good looking. She’ll do well in life and make a good mother. The society aunties loved her then, they love her now.




9) The “average joe”: This guy is a jack of all trades but master of none. Just good enough in studies, good enough at sports and pretty much ok overall. The guy will be present at all outings, make friends, be late for class, cheat and help cheat in exams and make for a fun person to be with. Your mum adores him.


What will happen: He’ll be the one with the fullest life and go on to be an honest guy. Be friends with him, he’ll go on to do well and will be delivering your farewell speech.



10)  The “nerd”: Reads everything, studies is all he talks about, is sick a lot, knows the teachers’ phone numbers by heart. You will find him discussing the answers of the questions even an hour after the exams, fighting for marks after the tests and cribbing about how he forgot to underline a question. If he tells you his exam went bad and he’ll fail, it means he will be the second highest in that class.

What will happen: He will go on to work, do ok, go for higher studies and study till his head hurts. After having worked hard and earned quite a decent living for himself, his family will get him married to an outgoing girl.

In the end, all these are actually just musing of what I think must be. Sometimes I just like to imagine these about people I am no longer in touch with.