Page 305 - Mechanical Engineers' Handbook (Volume 2)

296 Analysis, Design, and Information Processing

not of special significance with respect to justification of the need for a system, they
may be important in determining operational usage and, therefore, operational ef-
fectiveness of a system and the associated process.

5.6 Evaluation Test Instruments
Several special evaluation test instruments to satisfy test requirements and measure achieve-
ment of the evaluation criteria will generally need to be developed. These include investi-
gations of effectiveness in terms of performance objective attainment; selection of appropriate
scenarios that affect use of the system, use of the system subject to these scenarios by a test
group, and completion of evaluation questionnaires; and questionnaires and interviews with
operational users of the system.
Every effort must be made to ensure, to the extent possible, that evaluation test results
will be credible and valid. Intentional redundancy should be provided to allow correlation
of results obtained from the test instruments to ensure maximum supportability and reliability
of the facts and opinions to be obtained from test procedures.
The evaluation team should take advantage of every opportunity to observe use of the
system within the operational environment. Evaluation of personnel reactions to the aid
should be based on observations, designed to be responsive to critical evaluation issues, and
the response of operational environment personnel to test questionnaires. When any of a
number of constraints make it difficult to obtain real-time operational environment obser-
vation, experiential and anecdotal information becomes of increased value. Also, retro-
spective evaluation of use of a system is definitely possible and desirable if sufficiently
documented records of past usage of an aided process are available.
Many other effectiveness questions will likely arise as an evaluation proceeds. Questions
specific to a given evaluation are determined after study of the particular situation and the
system being evaluated. It is, however, important to have an initial set of questions to guide
the evaluation investigation and a purpose of this section to provide a framework for accom-
plishing this.
One of the important concerns in evaluation is that of those parts of the efficacy eval-
uation that deal with various ‘‘abilities’’ of a system. These include producibility, reliability,
maintainability, and marketability. Figure 9 presents a listing of attributes that may be used
to ‘‘score’’ the performance of systems on relevant effectiveness criteria.

6 CONCLUSIONS
In this chapter, we have discussed salient aspects concerning the systems engineering of
large and complex systems. We have been concerned especially with systems design engi-
neering and associated information-processing and analysis efforts. To this end, we suggested
a process for the design and evaluation of systems and how we might go about fielding a
design decision support system. There are a number of effectiveness attributes or aspects of
effective systems. Design of an effective large-scale system necessarily involves integration
of operational environment concerns involving human behavior and judgment with mecha-
nistic and physical science concerns. An effective systemic design process should:
1. Allow a thorough and carefully conducted requirements specification effort to
determine and specify needs of stakeholders prior to conceptual design of a system
process to accomplish the desired task
2. Be capable of dealing with both quantitative and qualitative criteria representing costs
and effectiveness from their economic, social, environmental, and other perspectives