Center for Software Engineering

Examples of the Use of ODC

Topics Center for Software Engineering Home Orthogonal Defect Classification Testing Software Data Analysis Publications University Partnerships Conferences & Workshops Write Us {leftbottom}

Is this picture familiar?

Real-Life Usage Scenarios Sample software defects Sample Classified data Examples of Analysis: ODC vs. Growth Model Process Escapes Base Code Problems Customer Impact Design/Code Completeness Customer Usage Profile

Last Edited on February 1, 2002

Real-Life Usage Scenarios:

The following sample scenarios describe ways in which ODC has proven to be invaluable in achieving specific project goals in the past.

Sample software defects:

1) The code has only 3 choices for the city but the design document lists 4.

2) An "default" path was tested which would have had the result that an error message was printed, but because the default clause had been omitted, no error message was printed, although the data wasn't used.

3) When "save" button was selected, the data was lost. Apparently, the data was being saved to one location and being read from a different one. The developer had to redo how the data was being accessed.

4) When trying to read data from the word processor got a message saying the format was unknown and then the system hung. This error message occurred because the routine that was supposed to convert the data from the word processor's format to the accepted format, had not been added to the code yet.

5) When the tester tried to add one more character than the maximum allowable, an error message was displayed which just showed one word per line.

Typical Data Table containing ODC Information:

This example illustrates the use of ODC vs. growth model based analysis

Shows the cumulative number of development defects for a component of
a high end operating system with time.

We have shown four periods (Period 0-Period 3) to highlight the behavior. After Period 0 when there was not much activity, the number of defects starts increasing rapidly in Period 1 and more than doubles during Period 2. Period 3 shows a saturation. Is this component ready for release? A typical growth model will say "Yes". Assuming the test cases were still being run in Period 3, the apparent behavior could easily be the result of ineffective test cases. When we classified the defect data using ODC, just the analysis by Defect Type portrays a different picture.

The normalized distribution of defects (within each period, the defect types
add to 100%) for the four periods indicated in Figure 1.

Defect types describe the nature and scope of the defect fixes. Typically, assignment and checking errors are simple coding problems while algorithm, function, timing/serialization, and interface problems are more complex and involve both high and low-level design issues. Even though the number of defects may be different in the four periods, the nature of the defect type distribution shows that the percentage of the Functions defects is steadily increasing through the four periods. This indicates a basically functionally unstable product, with a potential for high risk after the software release.

As it turned out, this component had severe problems in the field and ODC analysis could have helped to identify this at least during Period 2. Overall, ODC gives us much more detailed information about our testing and development process than a simple growth curve. We can learn about product, stability, testing effectiveness, as well as exposures and risks in the software.

This example illustrates the use of ODC to cut test cost by avoiding process escapes

Reliability growth models showing cumulative number of defects vs. days of testing for two releases of comparable sizes of a communication software. The scales are identical on the two plots. The red dots are the actual data and blue lines are fits to models. Release 2 has a much higher defect detection rate than Release 1.

In this example, the data came from two phases, T1 and T2. T1 was a combined Unit Test and Function Test performed by the developers and T2 was a System test performed by a different organization in a more expensive environment with the real hardware. There was a phase T0 consisting of Design Review/Code Inspection before T1, for which there were no data available (a practice common in many organizations). The distribution of the defects showed the following trend for the two releases.

It is clear that while Release 2 had a higher defect detection rate, it was at a substantially higher cost. The most expensive Phase T2 found most of the defects. Thus we need to understand how to cut the cost down relative to Release 2, and ODC can help us accomplish that.

When the defects from the two releases are classified using the ODC scheme, ODC Trigger for each defect is identified. Based on the mapping of the Triggers to the Activities performed in the organization, we can understand the distribution of defects by the appropriate Activity to expose them. Now let us understand the difference between Phase Found and Activity. According to process expectations (which may or may not be documented) the organization is expected to accomplish certain specific goals through its Activities in the appropriate Phases. In this example, UT-FT Activity (in T1), was expected to target all the triggers that are defined under Unit Test and Function Test in our generic mapping scheme. System Test (in T2) was expected to target all the triggers listed under System Test under the generic mapping scheme. Thus by identifying triggers for the defects we can clearly separate what was done in (say) T2 phase that was not defined as the System Test Activity.

More than half the number of defects found in T2 could have been found earlier in the cycle. The fraction in T0 identified by the ODC analysis points to the ineffectiveness of T0, even though we do not have the actual data collected from T0. UT-FT activity (supposed to be done in T2) appears to be the best place to find nearly 60% of the recorded defects. Thus the ODC analysis points to the Phases in the process that are weak, resulting in less efficient and more expensive testing.

This example illustrates the use of ODC attribute "Age" to identify the target areas of code for achieving better quality and lower development/maintenance cost.

For significant cost savings in development and maintenance, a major area of focus has to be the Base code.

This example illustrates an analysis of field reported defects using ODC attribute "Impact" to assess and improve direct customer satisfaction:

This example illustrates the use of ODC attribute "Defect Qualifier" to assess the completeness of Design/Code in a component :

This example illustrates an analysis of field reported defects to define a user profile for a high end operating system product component using ODC Triggers:

This chart shows the distribution of System Test activity related triggers by
the quarters after general availability. Different triggers have different usage
profiles and different products have different trigger profiles. For example,
Startup/Restart trigger peaks right away whereas Workload/Stress does not peak
until the second year after release. This information can be used in developing a
service strategy.