1.1 OVERVIEW
In this module you will get an overview of the most fundamental principles of testing.
Firstly, we point out that although you will come across different terminology throughout your career, professional testers are all pretty much agreed on these basic ideas that we describe in this this module.
Secondly, we take a look at the need for proper testing look at the cost of getting it wrong and we show why exhaustive testing is neither possible not practical. What are errors and how do they get into the software?
Thirdly, we describe a fundamental test process, base on industry standards, and underline importance of planning) tests and determining expected results in advance of test execution.
We conclude with a look at re-testing and regression testing; why it is so important to go that extra mile and run a suite of tests again, just when you thought it was safe to hit the release date.
1.2 OBJECTIVES
After completing this module you will:
» Understand basic testing terminology.
» Understand why testing is necessary.
» Be able to define error, fault and failure.
» Appreciate why errors occur and how costly they can be.
» Understand that you cannot test everything and that testing is therefore a risk management process.
» Understand the fundamental test process.
» Understand that developers and testers have different mindsets.
» Learn how to communicate effectively with both developers and testers.
» Find out why you cannot test your own work.
» Understand the need for regression testing.
» Understand the importance of specifying your expected results in advance.
» Understand how and why tests should be prioritized.
1.3 TESTING TERMINOLOGY
There is no generally accepted set of testing definitions used by the worldwide testing community. The British Standard for Software Component Testing published in August 1998 provides a new source of testing definitions. When involved in a testing project it is important to ensure everyone understands terminology adopted; you may find different terminology is used in your organization.
Exercise
The British Standard for Software Component Testing is known as BS___________
A useful glossary of terms used in software testing is called BS___________
Although it is a British Standard published by the BSI (British Standards Institute), the Specialist Interest Group in Software Testing (SIGIST) developed it over a period of several years.
ANSWERS:
7925-PART2,
7925-PART1
1.4 WHY IS TESTING NECESSARY?
This section explains why testing is necessary and closely at the cost and consequences of errors in computer software.
1.4.1 Definitions Errors, Faults and Failures
An error is a human action that produces an incorrect result. A fault is a manifestation of an error in software. Faults are also known colloquially as defaults or bugs. A fault, if encountered, may cause a fault, which is a deviation of the software from its existing delivery or service.
We can illustrate these points with the true story Mercury spacecraft. The computer program aboa spacecraft contained the following statement wri1 the FORTRAN programming language.
DO 100 i = 1.10
The programmer's intention was to execute a succeeding statements up to line 100 ten times then creating a loop where the integer variable I was using the loop counter, starting 1 and ending at 10.
Unfortunately, what this code actually does is writing variable i do to decimal value 1.1 and it does that once only. Therefore remaining code is executed once and not 10 times within the loop. As a result spacecraft went off course and mission was abort considerable cost!
The correct syntax for what the programmer intended is
DO 100 i =1,10
Exercise
What do you think was the error, fault and failure in this example?
The error is __________
The fault is ___________
The failure is __________
1.4.2 Reliability
Reliability is the probability that software will not cause the failure of a system for a specified time under specified conditions. Measures of reliability include MTBF (mean time between failure), MTTF (mean time to failure) as well as service level agreements and other mechanisms.
1.4.3 Errors and how they occur
Why do we make errors that cause faults in computer software leading to potential failure of our systems? Well, firstly we are all prone to making simple human errors. This is an unavoidable fact of life. However, this is compounded by the fact that we all operate under real world pressures such as tight deadlines, budget restrictions, conflicting priorities and so on.
1.4.4 Cost of errors
The cost of an error can vary from nothing at all to large amounts of money and even loss of life. The aborted Mercury mission was obviously very costly but surely this is just an isolated example. Or is it? There are hundreds of stories about failures of computer systems that have been attributed to errors in the software. A few examples are shown below:
A nuclear reactor was shut down because a single line of code was coded as X = Y instead of X=ABS (Y) i.e. the absolute value of Y irrespective of whether Y was positive or negative.
Blue Cross of Wisconsin installed a new $200m claims processing system - it sent out $60 million in unwarranted and duplicate payments. Also, when a data entry clerk typed 'none' in the town field the system sent hundreds of checks to the non-existent town of 'NONE' in Wisconsin.
In May 1992, Pepsi fan a promotion in the Philippines. It told customers they could win a million pesos (approx. $40,000) if they bought a bottle of Pepsi and found number 349 stamped on the underside of the bottle cap. Unfortunately, due to a software error, 800,000 bottle caps were produced with number 349 instead of one, which was an equivalent of $42 billion in prize money. It cost the company dearly as some people pursued their claims through the courts and Pepsi paid out millions of dollars in compensation.
Another story was printed in the New York Times on 18th February 1994. Chemical Bank managed to allow $15 million to be withdrawn incorrectly from 100,000 accounts - a single line error in the program caused every ATM on their network to process the transaction twice.
1.4.5 what happened on October 26th & 27th 1992?
The London Ambulance Service (LAS) covers an area of just over 600 square miles and is the largest ambulance service in the world. It covers a resident population of some 6.8 million, but its daytime population is larger, especially in central London. Since 1974 South West Thames Regional Health Authority has managed it.
The LAS carries over 5000 patients every day and receives between 2000 and 2500 calls daily (including between 1300 and 1600 emergency calls i.e. 999 calls). In terms of resources the LAS has some 2700 staff, 750 ambulances and a small number of various other vehicles including 1 helicopter. LAS make almost 2 million patient journeys each year. In 1992/1993 its budgeted income was £69.7 million.
On the 26th and 27th October 1992 the new system failed, ambulances failed to turn up and people lost their lives. Although no Coroners Court ever apportioned blame for any deaths directly to the computer systems failure, it was by any standards a major disaster and made the main evening news bulletins on several occasions.
1.4.6 London Ambulance Service
In summary the problems were:
Computer Aided Dispatch - 1
The system relied on near perfect information to propose optimum resource to be allocated to an emergency. However, there were imperfections in the information and changes in operational procedures made it difficult for staff to correct the system.
This was not a problem when it went live at 7 am on 26th October 1992 as the system load was light; however as the number of emergency calls increased throughout the-day it became increasingly difficult for staff to correct errors; this led to:
• Poor, duplicated and delayed allocations.
• Build-up of exception messages and awaiting attention list.
• Slow-down of system as messages and lists built up.
• Increased number of callbacks and hence delays in telephone answering.
The cost of these errors were ultimately that ambulances didn't turn up and people lost their lives although the official enquiry report did not attribute any fatalities to the system problems. The costs in terms of loss of confidence in the computer system, industrial relations and so on were probably also high.
1.4.7 Exhaustive testing why not test everything?
It is now widely accepted that you cannot test everything. Exhausted testers you will find, but exhaustive testing you will not. Complete testing is neither theoretically, nor practically possible. Consider a 10-character string that has 280 possible input streams and corresponding outputs. If you executed one test per microsecond it would take approx. 4 times the age of the Universe to test this completely. For a survey of the methodology and limitations of formal proofs of program correctness see [Manna 78]
1.4.8 Testing and risk
How much testing would you be willing to perform if the risk of failure were negligible? Alternatively, how much testing would you be willing to perform if a single defect could cost you your life's savings, or, even more significantly, your life? [Hetzel 88].
The amount of testing performed depends on the risks involved. Risk must be used as the basis for allocating the test time that is available and for selecting what to test and where to place emphasis. A priority must be assigned to each test.
Test Managers and Project Managers come up with different prioritization schemes but ht basic principle is that you must focus the testing effort on those areas of the system that are likely to have the most defects. Another key principle is that you must execute the most important test first. Otherwise, if you run out of time, which is likely, you will not have exercised the tests that give you the best payback in terms of faults found.
1.4.9 Testing and quality
Testing identifies faults whose removal increases the software quality by increasing the software's potential reliability. Testing is the measurement of software quality. We measure how closely we have achieved quality by testing the relevant factors such as correctness, reliability, usability, maintainability, reusability, testability etc.
1.4.10 Testing and legal, contractual, regulatory or mandatory requirements
Other factors that may determine the testing performed may be legal, contractual requirements, normally defined in industry specific standards or based on agreed best practice (or more realistically non-negligent practice).
1.4.11 How much testing is enough?
It is difficult to determine how much testing is enough. Testing is always a matter of judging risks against cost of extra testing effort. Planning test effort thoroughly before you begin, and setting completion criteria will go some way towards ensuring the right amount of testing is attempted. Assigning priorities to tests will ensure that the most important tests have been done should you run out of time.
1.5 FUNDAMENTAL TEST PROCESS
1.5.1 Introduction
Testing must be planned. This is one of Bill Hetzel's 6 testing principles [Hetzel 88 p25] and he says we are all agreed on this one. However, he points out that the problem is that most of us do not discipline ourselves to act upon it. Good testing requires thinking out an overall approach, designing tests and establishing expected results' for each of the test cases we choose.
You have seen already that we cannot test everything, we must make a selection, and the planning and care we expend on that selection accounts for much of the difference between good and poor testers.
The quality and effectiveness of software testing are primarily determined by the quality of the test processes used [Kit 95]. This is one of Ed Kit's 6 essentials of software testing. Test groups that operate within organizations that have an immature development process will feel more pain than those that do not. However, the test group should strive to improve its own internal testing processes. This section of the course shows a fundamental test process, based on the BS7925-2 standard for software component testing.
The fundamental test process comprises planning, specification, execution, recording and checking for completion. You will find organizations that have slightly different names for each stage of the process and you may find some processes that have just 4 stages, where 4 & 5 are combined, for example. However, you will find that all good test processes adhere to this fundamental structure.
1.5.2 Test process stages
See BS7925-2 for diagram of test process. Test planning involves producing a document that describes an overall approach and test objectives noting any assumptions you have made and stating any exceptions to your overall test strategy for your project. Test planning can be applied at all levels. Completion or exit criteria must be specified so that you know when testing (at any stage) is complete. Often a coverage target is set and used as test completion criteria.
Test specification (sometimes referred to as test design) involves designing test conditions and test cases using recognized test techniques identified at the planning stage. Here it is usual to produce a separate document or documents that fully describe the tests that you will carry out. It is important to determine the expected results prior to test execution.
Test execution involves actually running the specified test on a computer system either manually or by using an automated test tool.
Test recording involves keeping good records of the test activities that you have carried out. Versions of the software you have tested and the test specifications are software you have tested and the test specifications are recorded along with the actual outcomes of each test.
Checking for test completion involves looking at the previously specified test completion criteria to see if they have been met. If not, some test may need to be rerun and in some instances it may be appropriate to design some new test cases to meet a particular coverage target.
Note that BS7925-2 does not specify the format of any test documentation. However, The IEEE standard, known as 829, specifies in detail a standard for software test documentation.
BS7925-2 shows a diagram of a suggested hierarchy of test documentation.
HOME WORK
Exercise
Putting aside management problems. Read through test documentation examples in BS7925-2 and answer following questions:
What test techniques does component test strategy stipulate?
What percentage of decision coverage is required?
What should be done if errors are found?
The project component test plan is useful because the approach outlined allows:
a) Strict adherence to the component test strategy
b) More faults to be identified in the LOG components
c) A basic working systems to be established as early as possible
d) Isolation of the components within the test strategy
The component test plan must consist of a single document? TRUE/FALSE
The component test plan must specify test completion criteria? TRUE/FALSE
Why does component test plan specify 100% DC whereas strategy required 90%?
Which test case deals with non-numeric input?
List the expected outcome and the test condition
Why does the CTP have additional/altered test cases?
What action has been taken as a result of the test report?
No comments:
Post a Comment