单击此处编辑母版文本样式,,第二级,,第三级,,第四级,,第五级,,*,*,,Software Metrics,Lecture 10,,Software Test Metrics,,清 华 大 学 软 件 学 院,,,2004年5月11日,1,COCOMO,where,a,,,b,,,c,and,d,depend on the,development mode,.,Organic mode,,Semi-detached mode,,Embedded mode,Basic model,,,Intermediate model,,Advanced model,2,,COCOMO: Basic,The Basic COCOMO,Mode,a,b,Organic,2.4,1.05,Semi-detached,3.0,1.12,Embedded,3.6,1.20,development time cannot be predicted at this stage.,3,,COCOMO: Intermediate,The Intermediate COCOMO:,Mode,c,d,Organic,2.5,0.38,Semi-detached,2.5,0.35,Embedded,2.5,0.32,Mode,a,b,Organic,3.2,1.05,Semi-detached,3.0,1.12,Embedded,2.8,1.20,4,,COCOMO II,Application Composition,,E = OP / PROD,,,Early Design model,,,E = 2.45 × KLOC × EAF,,Post-Architecture model,5,,Putnam’s SLIM,6,,Test Concepts,,Estimating Number of Test Case &,Allocating Test Times,,Test Coverage Metrics,,Remaining Defects Metrics,Contents,7,,Section 1,,Test Concepts,8,,防不胜防的软件错误 ——,,,例1：1963年, 美国, 飞往火星的火箭爆炸, 损失$ 10,million.,,原因:,FORTRAN,循环,DO 5 I = 1, 3,,误写为,DO 5 I =,,1,.,3,,软件测试是保证软件质量的关键步骤，是对软件规格说明、设计和编码的最后复审，其工件量约占总工作量,40%,以上（对于人命关天的情况，测试相当于其它部分总成本的,3 — 5倍,）,Roles of Software Testing,9,,What is software testing,,“ Testing is the process of,,executing a program with the,,intent of finding errors.”,,,Glen Myers,,10,,Software Testing Objectives,Needing,minimum effort and time,,Showing that software functions,work according to specification,It cannot show absence of errors or defects. It can only show that errors are present,Uncovering,different cases,of errors,11,,1:5-10,Mainframes,,i.e.,1 tester for every 5 to 10 developers,,2:3,,Microsoft, 1992,,2:1,,Lotus (for 1-2-3 Windows),,1:2,,Average of 4 major companies,1992 Microsoft, Borland, WordPerfect, Novell,Tester-to-developer ratios,12,,What is a test case?,A set of test,inputs,, execution conditions, and,expected results,developed for a particular objective,13,,A test is made up of (many) test cases,,A test procedure is the detailed instructions for setting up, starting, monitoring and restarting a given test case. a,.,k.a,test plan,,a test case may be used in more than one test procedure,,a test case may include many subtests,What is a test?,14,,Test,test case,test case,test case,subtest,subtest,subtest,subtest,What is a test?,(cont.),15,,Testing Methodology,White-box,SELECTED,INPUTS,RESULTANT,OUTPUTS,INTERNAL,BEHAVIOR,DESIRED,OUTPUT,SOFTWARE,DESIGN,Black-box,SELECTED,INPUTS,RESULTANT,OUTPUTS,DESIRED,OUTPUT,16,,Objective:,Cover,the software structure,Structural (White-Box) Test Criteria,Criteria based on,,control flow,,data flow,,Metric: percentage of coverage achieved,,Adequacy based on metric requirements for criteria,17,,Graph representation of control flow and,,data flow relationships,Flow Graphs,Control Flow,,The partial order of statement execution, as defined by the semantics of the language,,Data Flow,,The flow of values from definitions of a variable to its uses,18,,Statement Coverage,,Branch Coverage,,Condition Coverage,,Basis Path Coverage,,All-Edges Coverage,,All-Paths Coverage,White-Box Test,19,,1,,2,,3,,4,,5,,6,,7,,8,,9,,10,,11,,12,,13,,14,,15,function,P,return,INTEGER,is,,begin,,X, Y: INTEGER;,,READ(X); READ(Y);,,while,(X > 10),loop,,X := X – 10;,,exit when,X = 10;,,end loop,;,,if,(Y < 20,and then,X,mod,2 = 0),then,,Y := Y + 20;,,else,,Y := Y – 20;,,end if,;,,return,2*X + Y;,,end,P;,A Sample Program to Test,20,,2,3,4,5,6,9´,10,12,14,T,T,F,F,9,,T,F,7,T,F,9,a,9,b,P’s Control Flow Graph (CFG),21,,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,At least 2 test cases needed,(X = 20, Y = 10),,,,,(X = 20, Y = 30),,,,All-Statements Coverage of P,22,,minimum level coverage using structural testing,,may be impossible to achieve 100% statement coverage,,code only executed in exceptional or dangerous circumstances,,code that is unreachable,Statement Testing: weaknesses,23,,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,At least 2 test cases needed,(X = 20, Y = 10),,,,,(X = 15, Y = 30),,,All-Branches Coverage of P,24,,Branch Testing,：,weaknesses,Branch testing is one level up from statement testing in degree of coverage,,same problem in achieving 100% coverage,25,,2,3,4,5,6,9,b,10,12,14,T,T,F,F,9,a,T,F,7,T,F,At least 3 test cases needed,,(X = 20, Y = 20),,,5T,7T,9a,F,,9bT,,(X = 15, Y = 10),,,5T,7,F,,9aT,9b,F,,(X = 9, Y = 10),,,5,F,,7,F,,9aT,9b,F,Condition Coverage of P,X > 10,,X = 10,,Y < 20,,X mod 2=0,26,,Condition Coverage,Generate test data such that all conditions in a decision take on both outcomes (if possible) at least once,,May not achieve branch coverage:,,consider,while (x 20 ),,x = 0,causes first true, second false,,x = 34,causes first false, second true,27,,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,Basic-paths Coverage of P,cyclomatic complexity:,V(G) = e - n + 2,28,,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,4 test cases needed,(X = 10, Y = 10),,,(X = 10, Y = 20),,(X = 20, Y = 10),,(X = 15, Y = 10),,Basic-paths Coverage of P,,,,,,,,29,,Basis path testing,：,Strengths/weaknesses,Basis set is easy to compute,,the technique is readily applicable to both structured and unstructured programs,,Drawback,: the basis set is not unique; there will be obvious paths omitted,,Drawback,: the number of paths in the basis set can greatly exceed the number necessary to achieve branch coverage,30,,2,3,4,5,6,9,b,10,12,14,T,T,F,F,9,a,T,F,7,T,F,At least 3 test cases needed,(X = 20, Y = 10),,,,(X = 5, Y = 30),,,,(X = 21, Y = 10),,,All-Edges Coverage of P,31,,2,3,4,5,6,9b,10,12,14,T,T,F,F,9a,T,F,7,T,F,Infinitely many test cases needed,(X = 5, Y = 10),,(X = 15, Y = 10),,(X = 25, Y = 10),,…,All-Paths Coverage of P,32,,All-Paths testing,Generate test data to exercise all distinct paths in a program!,,a path that makes,i,iterations through a loop is distinct from a path that makes,i+1,iterations through the loop, even if the same nodes are visited in both iterations.,,Thus, there are an,infinite number,of paths in the string processing program!,33,,All-Paths testing,(cont.),包含的不同执行路径数达,5,20,条，对每一条路径进行测试需要1毫秒，假定一年工作,365,×,24,小时，要想把所有路径测试完，需3170年,34,,2,3,4,5,6,9,b,10,12,14,T,F,9,a,T,F,Y,X,X,Y,Y,X,X,Y,Y,X,X,X,T,F,7,T,F,X,X,P’s Control and Data Flow Graph,35,,How many test cases are needed?,2,3,4,5,6,9,b,10,12,14,T,F,9,a,T,F,Y,X,X,Y,Y,X,X,Y,Y,X,X,X,T,F,7,T,F,X,X,All-Uses Coverage of P,36,,all-statements,all-edges,boundary-interior,,loop testing,min-max,,loop testing,all-paths,all-,defs,all-uses,all-DU-paths,all-p-uses,all-c-uses,C1,C2,subsumes,Structural,Subsumption,Hierarchy,all-branches,37,,Black Box Test,Equivalence Class,,Boundary Analysis,,…,38,,Testing Levels,Testing occurs throughout lifecycle,,Unit,,Integration,,System,,Acceptance,,Regression,39,,Overall Approach,Unit testing (module),,,Integration testing,,,System testing,,,Acceptance testing,White / Black box,,,Black box,,,Black box - requirements,,Black box - performance,40,,Section 2,,,Estimating Number of Test Case & Allocating Test Times,,41,,Estimate No. of Test Cases,,Time,,(available time,,available staff)/,,(average time to prepare a test case),,How many test cases do we need?,,Cost,,,(available budget) /,,(average preparation cost per test case),,Select the minimum number of the two,42,,Example,Development budget:,$4 million,,,%10,of which spent on preparing test cases,,Each test case costs,$250,or,4 hours,,The duration of project is set to be,25 weeks,(of 40 hours each) and,5 staffs,for test cases,,How many test cases should be prepared?,,,From cost,N1 = (4,000,000,,0.1) / 250 = 1,600,,,From time,N2 = (25,,40,,5) / 4 = 1,250,,,,,,,,N = min (N1 , N2),, therefore,,N = 1,250,43,,Allocating Test Time,How to allocate test time among,,system components,(e.g., acquired components, developed components),,test type,(e.g., unit test, integrated test, system test) and,44,,Test Time: Systems /1,Allocate less than 10% of the remaining time to certification test of acquired components,,Allocate time to the developed components based on their significance,OS, System software,Developed,,components,Acquired,,components,Interface to other systems,45,,Test Time: Systems /2,Example:,,Total test time: 340 h,,,,,,,,Test case proportion:,,Associated system components: 64.7%,,Operating system component: 23.5%,Interface to other systems,,(40 h),Acquired,,components,,(20 h),Developed,,components,,(200 h),OS, System software,,(80 h),Associated,,System,,Components,,220 h,46,,Test Case Invocation,In what order the system should be tested?,,Recommended sequence of system test:,,Acquired components (certification test only),,Developed product,,Other systems, OS, etc. (load test only),47,,Section 3,,Test Coverage Metrics,48,,Test Coverage Metrics /1,Coverage of what?,,Statement coverage,,Branch coverage,,Component/Module coverage,,Specification coverage,,GUI coverage,,…,49,,Test Coverage /1,Statement coverage (,CV,s,),,Portion of the statements tested by at least one test case.,50,,Test Coverage /2,Branch coverage (,CV,b,),,Portion of the branches in the program tested by at least one test case.,,51,,Test Coverage /3,Component coverage (,CV,cm,),,Portion of the components in the software covered and tested by at least one test case,52,,Test Coverage /4,GUI coverage (,CV,GUI,),,Portion of the GUI elements (e.g., menus, buttons, multiple selections, text fields, etc.) in the software covered and tested by at least one test case.,53,,Test Pass, Failure & Pending Rate/1,Test Pass Rate (,R,tp,),,Portion of the test cases that were executed successfully (i.e., produced expected output).,54,,Test Pass, Failure & Pending Rate/2,Test Failure Rate (,R,tf,),,Portion of the test cases that were not executed successfully (i.e., produced different output than expected).,55,,Test Pass, Failure & Pending Rate/3,Test Pending Rate (,R,tpend,),,Portion of the test cases that were pending (i.e.,,couldn,’t be executed or correctness of the output,couldn,’t be verified).,56,,Section 4,,Remaining Defects Metrics,57,,Remaining Defects Metrics /1,The idea is to inject (seed) some faults in the program and calculate the remaining bugs based on detecting the seeded faults [Mills 1972],Assuming that the probability of detecting the seeded and non-seeded faults are the same,58,,Seeded,,Detected,Seeded,,Undetected,Remaining,,Detected,Remaining,,Undetected,Total,,,Seeded,Total Remaining,Remaining Defects Metrics /2,59,,Remaining Defects Metrics /3,N,s,is already known;,,n,d,,and,n,s,are measured for a certain period of time,60,,Example,Assume that,61,,Comparative Remaining Defects/1,Two testing teams will be assigned to test the same product,62,,Example,63,,Summary,Estimating Number of Test Case?,Test Coverage Metrics,,Remaining Defects Metrics,64,,Further reading,,1. Software testing,Homework,65,,Thanks for your time and attention!,66,,。
