Defect Characterization and Analysis in Software Engineering, Study notes of Computer Science

Download Defect Characterization and Analysis in Software Engineering and more Study notes Computer Science in PDF only on Docsity! STUDIES OF DEFECTS DEFECT CHARACTERIZATION Goal Analyze the DOS/VS Operating system release 28 in order to characterize it with respect to the defects, interface defects, types of misunderstanding from the point of view of the organization. Environment: IBM Germany Operating System Release ~ 500 modules affected by the modification average size ~360 LOC (480LOC with comments) 432 faults reported Experimental design: Single project/case study in vivo, no report on experience Endres DEFECT CHARACTERIZATION Defect Distribution by Modules Number of Defects Number of Modules Affected 371 (85%) 1 50 2 6 3 3 4 1 5 1 8 DEFECT CHARACTERIZATION Defect Distribution by Modules Number of Defects per Module Number of Modules Number of Defects/Modules 112 1 36 2 15 3 11 4 8 5 2 6 4 7 5 8 3 9 2 10 1 14 1 15 1 19 1 28 DEFECT CHARACTERIZATION Problem Specific Defects Machine Configuration and Architecture 10 Dynamic Behavior and Communication between Processes 17 Functions Offered 12 Output Listings and Formats 3 Diagnostics 3 Performance 1 46% DEFECT CHARACTERIZATION Conclusions Interface between modules not a major source of defects Approximately half the defects originated in misunderstanding of the problem to be solved or potential solutions (=> not susceptible to improved programming techniques, I.e., a higher order programming language) Consider this as Analyze the defects in order to characterize them with respect to the various classification schemes from the point of view of the knowledge builder.in the context of a single version of an operating system, ... REQUIREMENTS DOCUMENT EVALUATION Goal Analyze the SCR requirements method in order to evaluate it with respect to the ease of modification and quality of the requirements document produced from the point of view of the organization. Environment: Naval Research Laboratory On-board flight program for the A-7 aircraft real-time, interactive, using TC-2 computer 16K 16 bit words Data collected after document baselined Experimental design: Single project/case study in vivo, experience in method, novices in application Basili/Weiss REQUIREMENTS DOCUMENT EVALUATION Goal Analyze the method in order to evaluate it with respect to the effect on product from the point of view of the organization. Analyze the SCR method in order to evaluate it with respect to the ease of modification of the requirements document structuredness of the requirements document ability to minimize consistency and ambiguity faults from the point of view of the organization. Analyze the requirements document in order to characterize it with respect to the ease of modification, structure, and consistency and un-ambiguousness from the point of view of quality assurance. Analyze the use of the requirements document in order to characterize it with respect to the its worthiness to be maintained from the point of view of the organization. REQUIREMENTS DOCUMENT EVALUATION Product Questions Changes/defects: How many changes are there to the document? (88) How many of the changes are errors? (79) What is the distribution of errors in the requirements document by type of misunderstanding (I.e., ambiguity, omission, inconsistency, incorrect fact, wrong section)? Context: How is the document being used? How was the need for change discovered? REQUIREMENTS DOCUMENT EVALUATION Product Questions Quality perspective: Ease of change Cost: What is the distribution of the types of changes and effort to make them? What is the distribution of changes by staff time to make the changes? Is the effort to change the document low? Document well-structuredness: Are most of the changes were confined to one section of the document? Consistency and Unambiguity: How consistent and unambiguous is the document relative to its precision and completeness? Worthiness of being maintained: How is the document being used?Is it used in important and relevant ways? TYPES OF CHANGES 0 10 20 30 40 50 60 70 80 90 85% 6% 2% 7% ORIGINAL ERROR CORRECTION COMPLETE OR CORRECT A PREVIOUS CHANGE REORGANIZE OTHER < 1 HOUR TRIVIAL 1 HOUR - 1 DAY EASY 1 DAY - 1 WEEK MODERATE > 1 MONTH FORMIDABLE 88 CHANES F M E T T E E T E % OF CHANGE NONCLERICAL ERRORS BY TYPE 70 60 50 40 30 20 10 % of non- clerical errors 5 % 31 % 13 % 49 % 2 % Ambiguity Omission Incon- sistency Incorrect Fact Wrong Section ERROR TYPE USE OF DOCUMENT Discovery of Need for Change 70 60 50 40 20 10 % of non- clerical errors 30 23 % 10 % 2 % 45 % 1 % 19 % Review by authors Review by nonauthors As maintenance reference As design reference As coding reference Other REQUIREMENTS DOCUMENT EVALUATION Conclusions DATA COLLECTION METHODOLOGY FEEDBACK Partial data provides useful feedback to developers Data analysis generates new questions of interest A-7 REQUIREMENTS DOCUMENT FEEDBACK The document is relatively more consistent and precise than complete and correct. Most changes are confined to single sections Can we conclude the document is well-structured? Seems to be a small effort to make changes Can we conclude the document is easy to change (maintain)? The document is heavily used as a design reference Can we conclude that it is worth maintaining? BUILDING DEFECT BASELINES Error Origin Classification Questions: Process conformance: What is the life cycle model? How well is it being applied? Domain conformance: How well do the developers understand the application? Quality perspective: What is the distribution of errors by error origin (i.e., according to the misunderstanding that caused them)? 2 12 10 66 1 2 7 SEL1ERROR P E R C E N T OF N O N C L E R I C A L E R R O R S 10 20 30 40 50 60 70 REQUIRE MENTS FUNCTIONAL SPECIFICATION DESIGN/CODE SINGLE COMPONENTS DESIGN/CODE MULTIPLE COMPONENTS ENVIRONMENTLANGUAGE OTHERS SEL2 5 3 10 72 5 1 1 ERROR P E R C E N T OF N O N C L E R I C A L E R R O R S 10 20 30 40 50 60 70 REQUIRE MENTS FUNCTIONAL SPECIFICATION DESIGN/CODE SINGLE COMPONENTS DESIGN/CODE MULTIPLE COMPONENTS ENVIRONMENTLANGUAGE OTHERS BUILDING DEFECT BASELINES SIMULATOR FOR SATELLITE PLANNING STUDIES Goals Analyze the life cycle defects for a particular project in order to characterize them with respect to various error, fault, and failure classes evaluate them with respect to those from other studies characterize them with respect to the relationship between errors and complexity from the point of view of the experience factory Basili/Perricone PROJECT BACKGROUND • General purpose program for satellite planning studies Size: 90K Source lines and 517 Code segments 370 FORTRAN Subroutines, 36 Assembly segments, 111 COMMON modules, BLOCK data, Utility routines Modified modules - Adopted from a previous system (72%) New modules - Developed specifically for this system • Requirements for the system kept growing and changing over the life cycle Defects Two definitions - Faults[Textual](215) and Errors [Conceptual] (155) 49% Defects/faults in modified modules 51% Defects/faults in new modules Corrections vs Modifications 38% of changes were modifications 62% of changes were fault corrections LIFE CYCLE OF ANALYZED SOFTWARE 1975 JANUARY 1976 1977 1978 1979 1980 1981 CHANGE FORMS MAINTENANCE ACCEPTANCE TESTING CODING DESIGN NUMBER OF ERRORS PER MODULE (Faults: 215 Faults) # Modules New Modified Errors / Module 36 17 19 1 26 13 13 2 16 10 6 3 13 7 6 4 4 1** 3* 5 1 1** 7 Effort to Correct Faults in the Most Fault-Prone New Model NUMBER OF ERRORS AVERAGE EFFORT (12 TOTAL) TO CORRECT MISUNDERSTOOD OR INCORRECT REQUIREMENTS 8 32 HOURS INCORRECT DESIGN OR IMPLEMENTATION OF A MODULE 3 0.5 HOURS CLERICAL ERROR 1 0.5 HOURS Effort to Correct Faults in the Most Fault-Prone Modified Model NUMBER OF ERRORS AVERAGE EFFORT (15 TOTAL) TO CORRECT MISUNDERSTOOD OR INCORRECT REQUIREMENTS 8 24 HOURS INCORRECT DESIGN OR IMPLEMENTATION OF A MODULE 5 16 HOURS COMPONENT CLERICAL ERROR 2 4.5 HOURS SOURCES OF ERRORS 0 % OF NONCLER- ICALS 40 15 4 22 0 0 18 Req Fn1 Design Design Lang Env Other Spec M ulti- Comp Single Com p Type of Fault 80 70 60 50 40 30 20 10 SEL2 SOURCES OF NONCLERICAL ERRORS 0 % OF NONCLER- ICALS 35 10 72 8 1 1 Req Fn1 Design Design Lang Env Other Spec Multi- Comp Single Comp Type of Fault 80 70 60 50 40 30 20 10 EFFORT 36 % 19 % 18 % 27 % 21 % 11 % 3 % 15 % 30 % 20 % 10 % 1 2 3 4 NEW MODULES 1 - 1 hr. or less 2 - 1 hr. to 1 day 3 - 1 day to 3 days 4 - more than 3 days 15 % 12 % 15 % 8 % FAULT TYPES RESULT: The largest percent of faults involve interface (39%) Control is more of a problem in new modules Data and initialization are more of a problem in modified modules Small number of omission faults in modified modules POSSIBLE EXPLANATION – The basic algorithms for the modified modules were correct but needed some adjustment with respect to data values and initialization for the application of the old algorithm to the new application FAULTS/1000 EXECUTABLE LINES (INCLUDES ALL MODULES) MODULE SIZE FAULTS/1000 LINES 50 16.0 100 12.6 150 12.4 200 7.6 >200 6.4 POSSIBLE EXPLANATIONS: Interface faults are spread across all modules The majority of modules examined were small, biasing the result The larger modules were coded with more care The faults in smaller modules were more apparent Fault Rate vs. Size Measuring Fault Rate against Size and Complexity Actual Believed Hypothesized Fault Rate Size/Complexity SUMMARY Defect Analysis provides useful information Can see new application with changing requirements Fault profile for new and modified modules different Fault profile for a new application different than that of more mature application Module size an open issues with respect to fault rate evidence that lager modules, within limits, may be less fault prone should not put artificial limits on module size BUILDING DEFECT BASELINES Study Conclusions Different project characteristics/environments produce different patterns of defect origin Projects with new requirements have more defects traceable to the requirement/specification phase Analyze the defects in order to characterize them with respect to the various classification schemes from the point of view of the knowledge builder.in the context of full life cycle at NASA/SEL Classification schemes: omission, commission interface, control flow, data (initialization, use) BUILDING DEFECT BASELINES Inspection Process Fault Classification Goal: Analyze the inspection process in order to characterize it with respect to the fault correction effort from the point of view of the manager Environment: Major mainframe manufacturer Next release of a library tool Development or modification of 40K source lines Total size 100K SLOC PL/1 like language Questions: What was the isolation and fix effort and total error correction effort for errors of omission and commission? Selby/Basili Investigating Influential Factors for Software Process Improvement Goal Analyze the project set in order to evaluate and improve future systems with respect to the defect prevention and detection from the point of view of the organization Environment: Matsushita Communications, four communications software projects Waterfall model development, Defect data collected during acceptance test and operation Experimental design: multi-project study in vivo, experienced subjects Mashiko/Basili Investigating Influential Factors for Software Process Improvement Definitions Defect notation: Dn(d1n,d2n,d3n,d4n;Cdn,SDn,Mdn), where d1 injection phase: when defect entered system d2 detection phase: when defect was found d3 error type: Logic, Communication, Omission, Commission d4 fault type: see table 1 Cd cost of defect: cost to fix Sd severity of defect: see table 2 Md number of modules affected Investigating Influential Factors for Software Process Improvement Table 1. Fault Type Fault type Abbrev iation Definition Global structure Gstr' Fault of relationships among subsystems Data structure Dstr Fault of structure of files, tables or other data, including fault of data size Algorithm Algr Fault of algorithm inside program module Human interface Hitr Fault of human interface External interface Eitr Fault of interface between the product and its external system Internal interface Iitr Fault of interface between modules Initialization Init Omission or Commission of initialization of data entry Constant value Cnst Fault of definition of constant value Knowledge Packaging What have we learned from the set of defect studies so far? There is value in multiple studies for both supporting and not supporting hypotheses Make sure you are comparing like things, e.g., same classification (injection time, detection time, environment, subjects, phase of data collection) Vary the classification to check the effects along the various values There are insights to be gained from the collection and analysis of defects according to different classification schemes, independent of the scheme Hypotheses: Are there classes of omission that affect the cost of fix? Or are there certain applications that cost more to modify? What is the relationship of the architecture to the cost of the modification? Knowledge Packaging What have we learned from the set of defect studies so far? Hypotheses: Defect rate goes down as size, complexity rise (within limits) Modified and new modules have different defect profiles Omission defects are difficult to fix after delivery but easy to fix during design Errors injected in the requirements phase are more expensive to fix than those injected in other phases, when detected at a later phase Consequences: The techniques used to build reusable models should be tailored based upon the anticipated defect classes Iterative development is better suited o minimizing the cost of defect fixing (lower cost of omission faults found early) FUNCTIONAL TEST PLAN EVALUATION Goal Analyze the acceptance test plan in order to evaluate and improve it for future releases with respect to the ability of the acceptance test suite to cover the operational use of the system from the point of view of the test developer Environment: NASA SEL Subset of a large satellite system Experimental design: Single project/case study in vitro, no subjects FUNCTIONAL TEST PLAN EVALUATION Product Questions Quality perspective: Compare the structural coverage of the acceptance tests and operational use of the system. What is the procedure coverage for the acceptance test suite by test and in total? What is the statement coverage for the acceptance test suite by test and in total? What is the % of unique code exercised by each test? What is the procedure coverage for the operational use of the system? What is the statement coverage for the operational use of the system? What is the overlap of the acceptance test and operational use coverage? Is there anything different about the statements executed in operational test but not covered during acceptance test? FUNCTIONAL TEST PLAN EVALUATION Product Questions Feedback: Is there any indication, based upon the coverage representation, to indicate whether reliability models can be applied during acceptance test to predict operational reliability? FUNCTIONAL TEST PLAN EVALUATION Structural Coverage of Acceptance Test Executable Statement Coverage by 10 TestCases Case Procedures Executable % Unique _______________Executed (%) Statements (%) Code t1 50.0 27.5 0.0 t2 50.0 27.2 0.0 t3 48.5 24.4 0.0 t4 60.3 37.9 4.4 t5 69.1 47.1 1.7 t6 67.6 42.7 0.0 t7 66.2 39.0 0.0 t8 66.2 45.6 1.0 t9 66.2 41.0 0.0 t10 66.2 40.2 0.0 Cumulative 75.0 56.0 Intersect 42.6 18.1 __________________________________________________________ 44% of executable statements were not exercised in acceptance test. They may have been executed in system/unit testing