Effectively Prioritizing Tests in Development Environment | CMSC 838, Study notes of Computer Science

Download Effectively Prioritizing Tests in Development Environment | CMSC 838 and more Study notes Computer Science in PDF only on Docsity! 1 Effectively Prioritizing Tests in Development Environment Amitabh Srivastava Jay Thiagarajan& Cyntrica Eaton November 26, 2002 Motivation ? After initial development, software will frequently change. ?Change management is critical to maintaining software utility. ? Small changes in one part of the program may have subtle, undesired effects in other seemingly unrelated parts of the program. Key Approach ? Effectively Prioritizing Tests….. ? Run the right test at the right time ? Focusing testing efforts on parts of the program affected by change. ? New defects probably result of new modifications ?….in Development Environment ? Technique must be fast, useful, and easily integrated into the development process. ? Estimate program change based on comparisons of binary representations of code. Echelon: Basic Idea Binary Change Analysis Coverage Analysis Test Prioritization Old Binary Coverage Prioritized List of Test Cases List of Blocks not Covered by Testing Old Binary New Binary Remaining Presentation ?Features of Echelon ?Test Prioritization Sequence ?Empirical Evaluation Features of Echelon ? Prioritizing Tests ?Calculating Program Change ?Utilizing Program Binaries ? Providing Test Coverage Information ?General Practicality 2 Prioritizing Tests ?Prioritizes tests into an ordered sequence based on program change. ?Does not permanently discard tests like minimization techniques presented in [4][13][19][30]. ?Allows use of a non-precise algorithm that works well in practice. Calculating Program Change ?Computes changes between programs at a very fine granularity using an accurate binary matching algorithm. ? The paper points out other works in test selection [1][3][4][20][28] and test prioritization [6][8][17][31] that also use program change as a guiding factor. ? Cites how differences in techniques affect the estimation of program change. Calculating Program Change ?Source code differencing ?Data and control flow analysis ?Coarse-grained code entities Calculating Program Change ? Source Code Differencing [6][8][28] ? Simple and fast ? Can be accomplished with commonly available tools like “diff” in Unix ? Erroneously marks a procedure as changed even if variables were only renamed. ? Fails to determine the set of affected procedures when header files that define macros and methods have been modified. Calculating Program Change ?Coarse-grained techniques [5] ? Uses functions, global variables, type definitions, etc. to identify which parts of the program might be affected by the changes. ? According to [11], estimations of program change based on coarse-grained entities [5] may select more tests than statement or control flow based techniques. Calculating Program Change ?Data and control flow analysis [1][20] ? Difficult in a language such as C++/C which contains pointers, casts, and aliasing. ? Flow analysis is expensive in large commercial systems [9][21] and should not be used unless the techniques will be helpful for other analyses[9]. 5 Step 3: Test Prioritization Weights 3 2 4 1 Impacted Blocks Prioritized List T1 T2 T3 T4 T5 Step 3: Test Prioritization Weights 3 2 4 1 Impacted Blocks Prioritized List T1 T2 T3 T4 T5 Step 3: Test Prioritization Weights 1 1 0 0 Impacted Blocks T2 T3 T4 T5 Prioritized List T1 Step 3: Test Prioritization Weights 1 0 0 Impacted Blocks T3 T4 T5 Prioritized List T1 T2 Step 3: Test Prioritization Weights 1 0 0 Impacted Blocks Prioritized List T1 T2 T3 T4 T5 Step 3: Test Prioritization Weights 2 4 1 Impacted Blocks T3 T4 T5 Prioritized List T1 T2 6 Step 3: Test Prioritization Weights 2 1 Impacted Blocks T3 T5 Prioritized List T1 T2 T4 Step 3: Test Prioritization Weights 2 1 Impacted Blocks T3 T5 Prioritized List T1 T2 T4 Impacted Blocks Step 3: Test Prioritization Weights 1 1 T3 T5 Prioritized List T1 T2 T4 T5 Step 3: Test Prioritization Weights 1 Impacted Blocks Prioritized List T1 T2 T4 T3 Step 3: Test Prioritization Weights 1 Impacted Blocks T5 Prioritized List T1 T2 T4 T3 Step 3: Test Prioritization Weights Impacted Blocks Prioritized List T1 T2 T4 T3 T5 7 Step 3: Test Prioritization Weights Impacted Blocks Prioritized List T1 T2 T4 T3 T5 Sequence #1 Sequence #2 Empirical Evaluation ?Performance ?Test sequence characteristics ?Predicted blocked coverage accuracy ?Effectiveness ?Early detection of defects when tests run in prescribed, prioritized order. Performance of Echelon 31283128No. of Tests 8,880,1288,880,128File Size (bytes) 668,274668,068Blocks 31,02631,020Functions January 2001December 2000Date Version2Version 1 Performance of Echelon 1,225Number of sequences 210 secondsTime taken by Echelon 16 TestsTests in first seq. 176 BlocksImpacted Blocks Covered Total Old New 220 378 158 Impacted Blocks Performance Analysis ?How many sequences of tests were formed? ?How many tests are in each sequence? ?How accurate is Echelon? Performance Analysis Number of Tests in Each Sequence First Sequence: 16 tests Cover a common routine