Background and Motivation - A Quantitative Approach to Software - Slides | CMSC 735, Study notes of Computer Science

Download Background and Motivation - A Quantitative Approach to Software - Slides | CMSC 735 and more Study notes Computer Science in PDF only on Docsity! 1 1 735 Experiment Fall 2002 Class Discussion Jeff Carver November 26, 2002 2 Families of Reading Techniques Reading Construction Analysis ReuseMaintenance DefectDetection Traceability Usability Test PlanDesign Code Project Source Code Code Library White Box Framework Black BoxFramework Design Requirements Code User Interface SCR English Screen Shot Scope-based Defect-based Perspective-based Usability -based System Wide Task Oriented Inconsistent IncorrectOmission Ambiguity Tester UserDeveloper Novice ErrorExpert PROBLEM SPACE SOLUTION SPACE Requirements Use-Cases 2 3 Background and Motivation • Study run by Jose Maldonado, et al in Brazil • Performed a replication of a study originally run at NASA comparing the effectiveness of PBR and checklist. • 4 Replications were Performed • Two with less experienced subjects (R1 & R2) • Two with more experienced subjects (R3 & R4) 4 Background and Motivation 24.4%33.9%18.2%22.9%NASA 1994 23.2%35.2%33.3%34.4%NASA 1995 7.5%11.7%15.6%15.3%R1 10.2%12.3%13.5%15.3%R2 ChecklistPBRChecklistPBR ATMPG 12.9%6.6%14.5%17.7%R3 15.6%10.3%17.7%8.7%R4 5 9 Study Design • Background questionnaire to determine experience level (High or Low) • Tester perspective was chosen • 6 high experience subjects • 16 low experience subjects • 4 were in Dr. Memon’s testing class and already learned about CPT • Placed in Group 3 • 12 had never seen CPT • Split evenly between Groups 3 and 4 10 Study Design • Treatments 6106Number of Subjects Low Detail -- Own Model High Detail -- Abstract Model High Detail -- Abstract Model Low Detail -- Own Model Process Detail -- Model LowLowHighHighPerspective Experience Group 4Group 3Group 2Group 1 6 11 GQM: Questions and Metrics • Goal: • To analyze PBR techniques for the purpose of evaluating the level of detail with respect to effectiveness from the point of view of the highly experienced inspector. • Question: • What affect does the level of detail in the PBR techniques have on the high experienced subjects? • Metrics • Percentage of defects found by members of Group 1 • Historical Data from previous 735 class (1997) and two replications from Brazil mentioned earlier 12 GQM: Questions and Metrics • Goal: • To analyze PBR techniques for the purpose of evaluating the level of detail with respect to effectiveness from the point of view of the low experienced inspector. • Question: • What affect does the level of detail in the PBR techniques have on the low experienced subjects? • Metrics • Percentage of defects found by members of Group 3 and 4 • Historical Data from previous 735 class (1999) and two replications from Brazil mentioned earlier 7 13 GQM: Questions and Metrics • Goal: • To analyze PBR techniques for the purpose of evaluating the level of detail with respect to effectiveness from the point of view of the low experienced inspector. • Question: • What affect does previous knowledge of the underlying model in the PBR techniques have on the subjects who use that model? • Metrics • Percentage of defects found by members of Group 3 who were not in Dr. Memon’s class • Percentage of defects found by members of Group 3 who were in Dr. Memon’s class 14 GQM: Questions and Metrics • Goal: • To analyze the Training for the purpose of evaluation with respect to effectiveness from the point of view of the researcher. • Question: • What affect did the training have on the subjects? • Metrics • Pretest answers • Posttest answers • Post-experiment questionnaire 10 19 Initial Results 26.5%20.6%22.1%Defect Rate 6106Number of Subjects Low Detail -- Own Model High Detail -- Abstract Model High Detail -- Abstract Model Low Detail -- Own Model Process Detail -- Model LowLowHighHighPerspective Experience Group 4Group 3Group 2Group 1 20 Initial Results Pe rc en t D ef ec ts F ou nd 0.0 0.1 0.2 0.3 0.4 0.5 Category Partition Testing Pe rc en t D ef ec ts F ou nd 0.0 0.1 0.2 0.3 0.4 0.5 No Model - High Experience No Model - Low Experience Outlier? 11 21 Outlier? • A data point that is either from a different population or a very unusual member of the same population • Computed as follows: • Lower Quartile QL= 25th percentile of data • Upper Quartile QU = 75th percentile of data • Interquartile Range (IQR) = QU – QL • Inner Fence = QH + 1.5(IQR) ; Outer Fence = QH + 3(IQR) • Our data (No model – Low Experience group): • QL = .235; QH = .257; IQR = .257-.235 = .022 • Inner = .257 + 1.5(.022) = .29; Outer = .257 + 3(.022) = .324 22 Initial Results Pe rc en t D ef ec ts F ou nd 0.0 0.1 0.2 0.3 0.4 0.5 Category Partition Testing Pe rc en t D ef ec ts F ou nd 0.0 0.1 0.2 0.3 0.4 0.5 No Model - High Experience No Model - Low Experience Outliers 12 23 Initial Results 21.8%20.6%24.1%Defect Rate 5105Number of Subjects Low Detail -- Own Model High Detail -- Abstract Model High Detail -- Abstract Model Low Detail -- Own Model Process Detail -- Model LowLowHighHighPerspective Experience Group 4Group 3Group 2Group 1 24 Initial Results • Question: • What affect does the level of detail in the PBR technique have on the high experienced subjects? • Low Process Detail • This study – 24.1% • High Process Detail • 735(97) [Equivalence Partitioning] – 27.1% • R3 [Equivalence Partitioning] – 17.7% • R4 [Equivalence Partitioning] – 8.7% • (R3 and R4 results were significant) 15 29 Threats to Validity • Internal • History • Results of later treatments may be attributed to events that occurred between treatments • Maturation • Processes occurring within subjects may change over time • Testing • Results may vary over time as subjects get more comfortable with testing procedures 30 Threats to Validity • Internal • Instrumentation • Results may differ with different measures • Selection • Results may differ because of the type of subjects in different groups • Process Conformance • Results may differ because procedure was not followed. 16 31 Threats to Validity • External • Are the results valid outside of this class? • Professional developers? • Real projects? • Are results valid for other requirements documents? • Different formats? • Different domains? • Different languages? 32 What is left to do? • More analysis on your comments about our defect list • Analyze results looking at: • Individual Defects • Classes of Defects • Make sure the experience levels of this class and the historical data are comparable 17 33 Discussion • How was the Process Conformance?