Page 190 -

All of these things will affect how the software is tested. Even though all of the details are
not known when the vision and scope document is written, all of these issues should at
least be raised, and any known information should be recorded. That way, the require-
ments analysts will make sure to discover the answers to these questions and include
them in the software requirements specification as nonfunctional requirements (see
Chapter 6). The testers, in turn, will take those nonfunctional requirements and use them
to build a test environment and plan a set of performance tests.

Most performance testing requires automated tools. Sometimes these tools can be bought
off the shelf; at other times, they must be developed separately by the programmers. Per-
formance requirements should be known as early as possible in order to properly plan for
them. It is a very common mistake for an organization to demand performance require-
ments, yet fail to budget for licensing fees for performance testing tools (which can be
expensive), a hardware environment that mirrors the production environment (which
costs as much as the production environment itself), and, most importantly, the time for
the testers to set up this environment. This is another reason the project manager must be
highly involved in planning for the performance testing—it can be a very costly operation,
but it is one that must be fought for if the software is to perform adequately.
Even when the software under test does not have to meet demanding performance
requirements, it can be challenging to plan for, budget, and set up an adequate test envi-
ronment. If the users will use multiple hardware or operating system platforms, each of
those must be replicated in the test environment. If web-based software has a specific net-
work configuration (routers, load balancers, separate network segments, firewalls, DMZ,
security, etc.), then the test environment must have all of the same equipment. If not, the
testers will never be able to replicate the real-world conditions under which the software
might break.

It is very common to shortchange the test environment by using virtual machines, by
using one machine for both the database and the web server, by getting rid of load balanc-
ers, or by introducing other differences between the test environment and the production
environment. That amounts to a waste of time when it comes to performance testing.
Maybe the team will find some memory leaks and bugs, but much of the time, perfor-

mance problems are in configuration, not in code. This is often overlooked, and it can lead
to large problems and user dissatisfaction when the software is rolled out.
It is important that the test environment be completely separate from production. This is
especially true of software that has already been released to the clients (especially web-
based software, or software that depends on a central database located in the same office
as the programmers and testers, or that relies on their network). It is surprisingly easy to
take down a production system during testing if the test environment is not completely
separate. For example, it may seem “safe” to save money by sharing only a firewall, load
balancer, or router. Yet there are serious kinds of bugs that may occur only in the test ver-
sion, which will make it effectively impossible for production requests to reach the soft-
ware. If this happens, the software testers will be blamed for the production downtime—
even if it was caused by a bug introduced by the programmers!

182 CHAPTER EIGHT