Page 168 - Automated Fingerprint Identification Systems (AFIS)

BUYING AN AFIS SYSTEM: THE BASIC DOCUMENTS NEEDED 153

bling block that cannot be overcome. But it is important to use your own data,
run the benchmark at the vendors’ factories, and benchmark each vendor
against a standard. The benchmark should focus on the following:

• Ease of user interfaces
• Basic functional and administrative features
• Lights out versus best practices
• Reliability, relative reliability, and accuracy versus labor content to compute
a value factor

While there is no best practice for benchmarks, it is illustrative to look at the
FBI’s approach to benchmarking IAFIS. They had already converted their cards
to digital records and were paying the vendors for the benchmark time and
effort, thus the enormous background file.

• 300 latents were checked against a 500,000 record background file
• Hand-built “ground truth sets” were based on minutiae comparisons
• 30 of the 300 were compared using the same feature extractions for all
vendors
• Vendor-extracted minutiae were compared against known minutiae
• Two latent sets and two tenprint sets were developed
• Latents: some with a limit on the number of minutiae and some with all
available minutiae
• Tenprints: first searched masking some minutiae and then with all
minutiae

A benchmark plan should cover the

• Purpose
• Scope
• Source of sample data
• Tests to be run
• Evaluation procedures
• Pass/fail or rated criteria
• Personnel to be assigned and their roles and responsibilities

The benchmark plan should set limits on

• The location of benchmark sites
• Time, in both hours per day and days
• How early the background data will be available