Page 223 -

206 Chapter 8 Software testing

Testing is intended to show that a program does what it is intended to do and to dis-
cover program defects before it is put into use. When you test software, you execute
a program using artificial data. You check the results of the test run for errors, anom-
alies, or information about the program’s non-functional attributes.
The testing process has two distinct goals:

1. To demonstrate to the developer and the customer that the software meets its
requirements. For custom software, this means that there should be at least one
test for every requirement in the requirements document. For generic software
products, it means that there should be tests for all of the system features, plus
combinations of these features, that will be incorporated in the product release.
2. To discover situations in which the behavior of the software is incorrect, unde-
sirable, or does not conform to its specification. These are a consequence of
software defects. Defect testing is concerned with rooting out undesirable sys-
tem behavior such as system crashes, unwanted interactions with other systems,
incorrect computations, and data corruption.

The first goal leads to validation testing, where you expect the system to perform
correctly using a given set of test cases that reflect the system’s expected use. The
second goal leads to defect testing, where the test cases are designed to expose
defects. The test cases in defect testing can be deliberately obscure and need not
reflect how the system is normally used. Of course, there is no definite boundary
between these two approaches to testing. During validation testing, you will find
defects in the system; during defect testing, some of the tests will show that the pro-
gram meets its requirements.
The diagram shown in Figure 8.1 may help to explain the differences between
validation testing and defect testing. Think of the system being tested as a black
box. The system accepts inputs from some input set I and generates outputs in an
output set O. Some of the outputs will be erroneous. These are the outputs in set O
e
that are generated by the system in response to inputs in the set I . The priority in
e
defect testing is to find those inputs in the set I because these reveal problems with
e
the system. Validation testing involves testing with correct inputs that are outside I .
e
These stimulate the system to generate the expected correct outputs.
Testing cannot demonstrate that the software is free of defects or that it will
behave as specified in every circumstance. It is always possible that a test that you
have overlooked could discover further problems with the system. As Edsger
Dijkstra, an early contributor to the development of software engineering, eloquently
stated (Dijkstra et al., 1972):

Testing can only show the presence of errors, not their absence

Testing is part of a broader process of software verification and validation (V & V).
Verification and validation are not the same thing, although they are often confused.