Page 304 - Mechanical Engineers' Handbook (Volume 2)

5 System Design 295

Behavioral or human factor evaluation criteria used to evaluate performance include political
acceptability, institutional constraint satisfaction, implementability evaluation, human work-
load evaluation, management procedural change evaluation, and side-effect evaluation.
Efficacy Evaluation
Two of the three first-level evaluation criteria concern algorithmic effectiveness or perform-
ance objective achievement and behavioral or human factors effectiveness. It is necessary
for a system to be effective in each of these for it to be potentially capable of truly aiding
in terms of improving process quality and being acceptable for implementation in the op-
erational environment for which it was designed. There are a number of criteria or attributes
related to usefulness, service support, or efficacy to which a system must be responsive.
Thus, evaluation of the efficacy of a system and the associated process is important in
determining the service support value of the process. There are seven attributes of efficacy:

1. Time Requirements. The time requirements to use a system form an important service
support criterion. If a system is potentially capable of excellent results but the results
can only be obtained after critical deadlines have passed, the overall process must
be given a low rating with respect to a time responsiveness criterion.
2. Leadership and Training. Leadership and training requirements for use of a system
are important design considerations. It is important that there be an evaluation com-
ponent directed at assessing leadership and training needs and trade-offs associated
with the use of a system.
3. Communication Accomplishments. Effective communication is important for two rea-
sons. (1) Implementation action is often accomplished at a different hierarchical level,
and therefore by a different set of actors, than the hierarchical level at which selection
of alternative plans, designs, or decisions was made. Implementation action agents
often behave poorly when an action alternative is selected that they regard as threat-
ening or arbitrary, either personally or professionally, on an individual or a group
basis. Widened perspectives of a situation are made possible by effective commu-
nication. Enhanced understanding will often lead to commitment to successful action
implementation as contrasted with unconscious or conscious efforts to subvert im-
plementation action. (2) Recordkeeping and retrospective improvements to systems
and processes are enhanced by the availability of well-documented constructions of
planning and decision situations and communicable explanations of the rationale for
the results of using the system.
4. Educational Accomplishments. There may exist values to a system other than those
directly associated with improvement in process quality. The participating group may,
for example, learn a considerable amount about the issues for which a system was
constructed. The possibility of enhanced ability and learning with respect to the issues
for which the system was constructed should be evaluated.
5. Documentation. The value of the service support provided by a system will be de-
pendent on the quality of the user’s guide and its usefulness to potential users of the
system.
6. Reliability and Maintainability. To be operationally useful, a planning-and-decision-
support system must be, and be perceived by potential users to be, reliable and
maintainable.
7. Convenience of Access. A system should be readily available and convenient to access
or usage will potentially suffer. While these last three service support measures are