No Homework, No Bugs
October 15, 2007
Two software engineers from the IBM Haifa Research Lab have developed a new way to conduct software reviews that is faster and easier than traditional inspection methods.
Shmuel Ur, an IBM Master Inventor and expert in multi-threaded testing, and Eitan Farchi, manager of the Software Testing, Verification and Review Methodology group in HRL developed their approach based on years of experience in the software testing field. Known as selective homeworkless review, Farchi and Ur's approach enables a virtually painless peer review process of code artifacts that appeals to both developers and management.
That's no small feat. While nearly everyone in the software engineering field agrees that peer reviews are a positive and even significant phase of application development, finding the time to do such reviews is always difficult. Traditional code reviews are often a very intense, time-consuming process. For the last 30 years, the industry standard for peer software reviews has been the formal inspection method, developed by IBM software developer Michael Fagan in the mid-1970s. An immensely thorough approach, formal inspection requires management to allot at least 20% of development time to the review process, while requiring software engineers to conduct in-depth preparations prior to reviewing all of their peers' code.
The Selective Homeworkless Review
That's where Farchi and Ur's method comes in. As design cycles get increasingly shorter, very few people are able to devote huge chunks of time to anything that isn't straight-ahead coding.
"The review process has always had inherent top-down and bottom-up problems", notes Ur. "Managers tied to a tight development timetable can't find the time to devote production time and staff to reviews. At the same time, developers find it hard to adequately prepare for the process."
Several years ago, Farchi and Ur developed a review method for concurrent code called interleaving review technique. While working on the interleaving technique, they realized that many people were simply not reviewing properly. As a result, they began to develop a number of other techniques, some of which were based on testing methods they had previously designed.
"We wanted to create a method people could actually do, without arousing too much opposition," Ur explains. "It's like exercising. Everyone knows it's better for you to do aerobics than play soccer, but soccer is certainly better than no physical activity at all. We tried to find an approach that's easy to do."
As a result, the selective homeworkless review was born. At its core, Farchi and Ur's method is quite similar to formal inspection, but it diverges from that traditional approach in two major ways. Developers can focus on reviewing classes of artifacts with the most potential (hence making it "selective") and they don't need to spend hours "doing their homework" before performing the review.
In the selection stage, reviewers deploy either concern-based artifact selection or test selection. Based on their choice, reviewers then "homeworklessly" use one or more of relevant existing review techniques, such as desk checking or paraphrasing, or any of the tailor-made techniques Farchi and Ur developed, such as cross product, state machine, or interleaving review. All in all, the selective homeworkless review requires development teams to meet for a maximum of 90 minutes per week as part of their regular project schedule.
The new approach also includes an educational stage for review moderators, which not only instructs developers how to lead standard inspection processes, but also how to benefit from correct selection and homeworkless review techniques.
Fast and Effective
But Farchi and Ur's method is not just fast—it's also effective. Selective homeworkless reviews find on average 2.17 code issues per person-hour, about a third of which are bugs. This is an extremely cost effective way to review software code. In addition, the average rate of selective homeworkless reviews essentially matches the rate of classical formal inspection meetings, about 100 lines per hour. As with any review technique, the added awareness that results from peer code reviews produces developers who program better, resulting in less bugs right from the start.
The selective homeworkless review is already being used successfully within IBM. Three years ago, IBM firmware developers asked the two Haifa developers to help them revamp their inspection methodology so they could find bugs early in the development process. Farchi and Ur have worked with IBM Firmware since then, helping them to constantly improve their review process. In recent years, selective homeworkless reviews have been introduced in other IBM divisions and departments, such as Research, Storage, z Series Software, and others. The two Haifa researchers are now preparing to teach the methodology to IBM partners and present it at professional conferences.
According to Farchi, any software development organization can benefit from their method.
"We have convincing evidence that this methodology can be introduced even in organizations that are resistant to formal inspections," he noted. "It leads to significant improvements in quality for anyone who deploys it."