How it could work
Compare current work, structured human-led work, and the same structured process with AI assistance. Hold staff skill, access to information and review standards constant; freeze the tested tool and settings.
Where it can fail
A shorter drafting step may create more review and rework. A team that saves time on routine cases can still miss rare consequential errors.
What to measure
Total staff-hours per eligible case, including training, review and rework. Report incorrect clearances and missed cases under the same independent audit rubric.
Design a test
Use matched fictional records with different difficulty, rotate order for repeat teams, and audit outputs against a key. Record setup time, overrides and whether disagreements changed the final decision.
When to pause or retest
Hold expansion if consequential errors increase or if review and rework erase the staff time benefit that justified the test.
What this evidence can support
Results describe the tested configuration and workload. They do not establish that AI in general pays off or that the measured effect will transfer to a different organization.
Trace the decision
From a faster AI step to a decision about value
Faster AI drafting only matters if audited total workload, service or cost improves under the same review requirements. Measure the burden shifted to downstream teams.
- Match eligibility and staffing across workflows
- Audit total time, mistakes and rework
- Compare incremental resources against a predeclared target
Hold capacity fixed and retain error records. Repeat across different case mixes and state which costs and harms were left out.
When this overlaps another topic
Review time can shift instead of falling
An AI tool might flag more candidate discrepancies while giving a reviewer more duplicates to reconcile. Count downstream verification and rework.
Check: Give all arms the same common-origin evidence and scoring key. Record total staff time and the number of consequential errors and corrections.
Source independence and corrections
Time saved and cash saved are different outcomes
Lower manual time may free capacity without reducing spending, while training, tools, verification and incident follow-up may add costs.
Check: Log review and rework hours separately, state the valuation if comparing money, and test whether the option clears the proposed minimum improvement after costs.
Cost, resources and competing benefits
An AI comparison needs a matched human process
Comparing AI with a fragmented baseline credits the AI arm for improved formatting and coordination. Compare the same organized workflow with and without the tested AI step.
Check: Use the same eligible cases, staffing, release times and independent scoring. Report both contrasts and retain outputs from errors and overrides.
Sources and scope
These references support particular methods or historical context. The research design above is a proposal.
- NIST AI Risk Management Framework: Measure · Methods guidance
Supports documenting tests, errors, human overrides and independent review.
- HM Treasury Magenta Book: Analytical methods · Methods guidance
Explains comparison designs, attribution, evidence synthesis and value for money methods.
Original hypotheses to examine
These source records remain untested.