Page 24 - Mechanical Engineers' Handbook (Volume 2)

3 Statistics in the Measurement Process 13

Resolution and Truncation
Let {s } be the theoretically possible set of measurements of unbound significant digits from
n
a measuring system of infinite resolution and let {x } be the actual set of measurements
n
expressed to m significant digits from a measuring system of finite resolution. Then the
quantity s x e is the resolution or truncation deficiency caused by the measurement
i
i
i
process. The unbiased estimates of mean and variance are
s (s s) 2
2
ˆ i s and ˆ i (23)
n n 1
} is available rather than {s }, the required mean and variance are
Noting that the set {x n
n
x e e (x x) 2
ˆ i i x i and ˆ i (24)
2
n n n n 1
The truncation or resolution has no effect on the estimate of variance but does affect the
estimate of the mean. The truncation error e is not necessarily distributed randomly and may
i
all be of the same sign. Thus x can be biased as much as e /n e high or low from the
i
unbiased estimate of the value of so that ˆ x e .
If e is a random variable observed through a ‘‘cloudy window’’ with a measuring system
i
of finite resolution, the value of e may be plus or minus but its upper bound is R (the
i
resolution of the measurement). Thus the resolution error is no larger than R and ˆ x
Rn/n x R.
If the truncation is never more than that dictated by the resolution limits (R) of the
measurement system, the uncertainty in x as a measure of the most representative value of
is never larger than R plus the uncertainty due to the random error. Thus ˆ x
ˆ
(W R) . It should be emphasized that the uncertainty interval can never be less than the
resolution bounds of the measurement. The resolution bounds cannot be reduced without
changing the measurement system.
Significant Digits
When x is observed to m significant digits, the uncertainty (except for random error) is never
i
m
m
more than 5/10 and the bounds on s are equal to x 5/10 so that
i
i
5 5
x s x (25)
i
10 m i i 10 m
The relation for ˆ for m significant digits is then from Eq. 24.
m
e (5/10 ) 5
ˆ x i x x (26)
n n 10 m
m
The estimated value of variance is not affected by the constant magnitude of 5/10 . When
the uncertainty due to significant digits is combined with the resolution limits and random
error, the uncertainty interval on ˆ becomes

ˆ x W R
5
ˆ
10 m (27)
This illustrates that the number of significant digits of a measurement should be carefully
m
chosen in relation to the resolution limits of the measuring system so that 5/10 has about
the same magnitude as R. Additional significant digits would imply more accuracy to the
measurement than would actually exist based on the resolving ability of the measuring sys-
tem.