Page 187 -

It is rare for no defects to be uncovered in the first test iteration. Usually, some test cases
fail and defects must be reported. Once the iteration is complete, the defects are triaged
(see below) and the programmers begin repairing the software. When a new build is deliv-
ered, the next iteration of testing begins.
After each iteration, the testers create a test report. This is a document (usually a spread-
sheet or word processor document) that simply contains a list of all test cases that failed or
were not executed. The purpose of the report is to give the project team a good idea of
how much of the application was exercised in the test.
For each test case that failed, a tester creates a defect report. These reports are used to deter-
mine the general health of the product, in order to allow the project team and stakehold-
ers to understand its maturity and readiness for release (see below about defect tracking).
Testing is complete when either no defects are found or (more likely) all of the defects that
have been found satisfy the acceptance criteria in the test plan. These criteria will typically
include several rules: that all test cases have been executed, that the project stakeholders
have reviewed all of the defects and personally determined that the software can be
released with those known defects, and that there is consensus among the engineering
team that the product is ready for release. Table 8-6 shows an example of typical accep-
tance criteria from a test plan.

TABLE 8-6. Acceptance criteria from a test plan
1. Successful completion of all tasks as documented in the test schedule.
2. Quantity of medium- and low-level defects must be at an acceptable level as determined by the software testing
project team lead.
3. User interfaces for all features are functionally complete.
4. Installation documentation and scripts are complete and tested.
5. Development code reviews are complete and all issues addressed. All high-priority issues have been resolved.
6. All outstanding issues pertinent to this release are resolved and closed.
7. All current code must be under source control, must build cleanly, the build process must be automated, and the
software components must be labeled with correct version numbers in the version control system.
8. All high-priority defects are corrected and fully tested prior to release.
9. All defects that have not been fixed before release have been reviewed by project stakeholders to confirm that they

are acceptable.
10.The end user experience is at an agreed acceptable level.
11.Operational procedures have been written for installation, set up, error recovery, and escalation.
12.There must be no adverse effects on already deployed systems.

The list of acceptance criteria can also include any specific performance requirements
(“server must support 80 users,” “product must perform under high load for 72 hours
without failure,” etc.), as well as any security requirements specific to the software.
The majority of the acceptance criteria in the example focus on the consensus of the peo-
ple involved in the software project. It is important that everyone agrees that any defects
that the software ships with will not affect the users.

SOFTWARE TESTING 179