Orthogonal Defect Classification (Archival)       


P.  Santhanam photo

Orthogonal Defect Classification (Archival) - FAQ on ODC


A. History:

A-1:How long has ODC been in existence and how widely is it practiced today?

ODC was a concept invented at IBM Research circa 1991. Its initial focus was to help in the in-process feedback of the actual defect data to the developers and testers. The original classification scheme was first piloted in Poughkeepsie to refine the classification scheme, as well as to validate the analysis results. The first paper summarizing the full scheme was published in 1992.

Over the years ODC has been deployed in hundreds of projects and currently it is reaching more than 4000 software professionals in IBM in terms of its impact. There have been other companies such as Motorola, Tandem and NorTel that have also embraced the technology.

A-2: How has ODC evolved over the years?

The original focus of ODC was to lay the foundation for providing analysis and feedback of defect data targeting quality issues in software design and code in a procedural language environment. Explicit inclusion of the terms of the Object Oriented programming in the basic ODC scheme was another such enhancement. The current version is ODC v5.2. As the reality of software development includes other aspects such as Information Development (User Documentation), Build, and National Language Support, we have refined the ODC scheme to accommodate for these needs.

B. Concepts and Models:

B-1. What does "Orthogonal" mean, in Orthogonal Defect Classification?

This question has answers at least at two levels.

At the highest level, the eight ODC attributes of a defect i.e. Activity, Trigger, Target, Defect Type, Defect Qualifier, Source, Impact, and Age, truly capture orthogonal (non-redundant) pieces of information. They are designed to be at the right level of granularity (not too large to be useless and too fine to be exhausting) and arguably sufficient to answer most questions of practical interest on the software

At the level of the individual attribute, the orthogonality (say, Defect Type) relates to the fact that the Defect Type distribution describes the state of a software product much the same way the three (orthogonal) Cartesian coordinates describe the location of an object in a three dimensional space. Notice that the evolution of a software product through a schedule is similar to the motion of an object through space and time.

B-2: Why is ODC not just another classification scheme?

Typical defect classification schemes are just convenient ways to group defects and do not have strict rules related to the Attributes and their values. The taxonomy behind ODC is carefully designed and maintained so that certain properties that are essential to provide the basis for the collection of meaningful data and the subsequent mathematical analysis, are not violated.

Some of the properties that make ODC special are:

  • The clear delineation among the Attributes .
  • Classification is independent of product, process, when and where the defects are found (i.e. it is generic , in the language of the programmer).
  • Distributions represent information rather a single number.
  • Trigger, Defect Type and Defect Qualifier are very semantic. They are more than keywords.
  • Activity refers to what the individual was actually doing when the defect surfaced, no matter where in the schedule the product or component is supposed to be. This distinction provides ODC based analysis with a critical advantage. Since ODC is activity-based, it is process independent, meaning that it can be used equally effectively, regardless of the process model or technology being used.
  • Flexibility in the process is reflected in the choice of Activities. Mapping of the Triggers to each Activity helps with detailed process definitions.
  • ODC Triggers constitute a complete and sufficient set for what people do in the V&V of software, and thus can be used as a check list .
  • Ability to link to the people experience/maturity through Triggers

B-3: How is ODC related to Root Cause Analysis?

Root Cause Analysis (RCA) looks at each individual defect through a magnifying glass. It is very valuable in terms of understanding Cause and Effect relationships in the context of a single defect. Given the typical workload of a developer or tester, there are not enough resources in an organization to perform RCA on all or even just the important defects. In addition, you end up in identifying too many actions as a result, as you are looking at the forest at the tree level. The relative prioritization among the choice of actions becomes a significant part of the process to implement solutions. ODC gives an alternative to provide rapid root cause analysis with prioritization with minimal resources. In most cases analysis exploiting the eight ODC attributes and the other useful information typically captured by an organization, such as Open and Closed dates, Component, Subsystem, Severity , etc. are more than adequate to drive and monitor change.

B-4: Isn't defect analysis focusing too much on quality, and not enough on other aspects such as productivity, efficiency, time to market, etc. ?

Don't think that defects relate only to quality. When you give your doctor a few cc of blood sample, it represents your health and well being. Based on certain measurements of the blood related to the concerns you have expressed, your doctor can diagnose your health in a wide range of areas: systemic conditions, any temporary illness, potential risks, etc. Consequently, useful inferences can be drawn to determine any immediate short or long term treatment or recommendations to modify your life style. Over the past 10 years we have used defect data to represent a software product and the development process, much the same way your blood sample represents your well-being. The diagnosis using defects can target any of the concerns you may have about your product or process.

B-5: What is the Butterfly Model?

This model consists of a methodology and algorithms which link the various aspects of software development through the entire life cycle including servicing the product in the field. It captures customer usage unique to the product, environmental influences, and product complexities. The Butterfly Model includes algorithms which capture behaviors of triggers, relationships between triggers, and relationships between triggers and defect types. This information is used to isolate and identify opportunities and exposures at the technical level so that explicit actions are easily defined and executed. In addition, these algorithms afford the ability to project the impact of potential exposures on specific triggers, and the activities which own responsibility for them. The links between defect types and specific process activities (high level design, detailed design, or coding) which could most effectively be improved in order to prevent the defects from being injected have been clearly established. The insight gained from this analysis has implications to schedule integrity, development costs, and product stability.

The Butterfly Model defines the Product Profile as the set of characteristics uniquely defining a product. Key elements include (but are not limited to):

  • the product function (what does this product do?)
  • complexity or difficulty inherent in the design of the product as a whole and the new work elements being done
  • customer usage
  • environment(s) in which the product runs
  • process (signature)
  • historical, classified Life of Product defect data and analysis

Capturing these characteristics provides the foundation to assess their influence on not only the quality of the product during development and in the field, but on every aspect of software including productivity, customer satisfaction, development and warranty costs, workload projections, and activity effectiveness and efficiency.

B-6: What has "Butterfly" got to do with software?

The Butterfly Model is named after the popular analogy in chaos theory linking a butterfly fluttering its wings in Shanghai to a hurricane in North America. You may believe that most software processes are chaotic and hence appreciate the name. In chaotic systems, small changes in the initial conditions can produce major changes in their evolution. Butterfly model addresses the various ways to use the ODC data at the small team level to produce a major change in the maturity and reliability of software at the product level.

B-7: What is the difference between ODC and Butterfly Model?

ODC scheme is the standard in terms of data definitions whereas the Butterfly Model provides decision support for technical and management teams. In any kind of modeling you need a way to describe the data. That is done by ODC. The relationships among the data and properties of interest to the user are captured in terms of models. This is done by the Butterfly Model for topics related to software development and service.

B-8: How is my process captured in an ODC roll out?

One of the first things an organization must do to implement ODC is to define the activities they perform. Then the existing set of Triggers must be mapped to those Activities. Note that although the organization defines their Activities, they do not define or redefine the triggers. This step along with the inherent relationships between Triggers and Defect Types captures the essence of your process. If the same trigger appears in multiple activities, this points to potential unintentional or intentional overlaps between activities. This will help you focus on what every activity is expected to accomplish.

C. Implementation Issues

C-1: Do I need a mature process to use ODC?

A well defined process is useful, but not essential for a basic and very useful level of analysis, as long as a reasonably complete information on defects is available. Existence of a process will greatly enhance the path to improvement.

C-2: How long does it take to realize the benefits of ODC?

It is good to have a focal point who is well trained in ODC and analysis to be the Champion in an organization. Typically it takes a few minutes (<2 minutes per defects when proficient) to classify each defect. Depending on the scope of issues and data targeted, the analysis results can be obtained and evaluated in a very short time, typically a few weeks.

C-3: What are some useful things to think about in order to implement and roll-out ODC effectively?

1) The organization's goals.
2) Current defect data collection methods and tools.
3) The kind of information contained in the current defect data.
4) Method/Tool to collect new ODC defect data.
5) Granularity of the feedback analysis.

C-4:Must we have defect data on a previous project in order to implement ODC and Butterfly?

A. The more historical data that exists, the better. It will help set some reasonable expectations for the various distributions. That is particularly true for exploiting the Butterfly Model functions. However, there is still value that can be added through a direct analysis of ODC data.

D.Field Defects

D-1: What are the differences in the classification of the defects found in-process and those discovered in the field?

1) When classifying an in-process defect, select the activity first, and then the corresponding trigger. When classifying field defects:

a) Select a trigger first based on what was actually done to expose the defect (or would be needed to recreate it). For example, if the customer entered a single command with no parameters, you would choose Test Coverage, even though the customer was in production mode, rather than test.

b) Select the corresponding Activity for that trigger based on the Activity/Trigger mapping defined in your process. This will result in the activity which was most likely to have found the defect, based on your process definitions. For the above example in (a), the Activity may have been "Function Test". Earlier opportunities (for prevention or removal) will be derived based on ODC attributes.

2) There are triggers which are not normally associated with the field, since they involve extensive knowledge of design and code internals. These include:

a) Design Conformance
b) Understanding Flow
c) Concurrency
d) Simple Path
e) Complex Path

In each of these cases, it would be more typical for the customer to uncover a defect using externals (Black Box). In those cases where it is clear that the customer has uncovered a defect, not based on the execution of externals, but through inspection, or conformance with product announcement information, these triggers are appropriate.

3) Since the customers are not typically trying to emulate a development organization, the trigger Normal Mode (Blocked Test Mode) should not be chosen for field reported defects. Instead, you should choose the Trigger from the entire list that approximates best to what the customer was trying to when the defect was exposed.