How it could work
Write a comparison protocol before testing. Hold eligibility, resources and release times constant, freeze the tested design, and judge outputs against an independent scoring rule.
Where it can fail
A before-and-after improvement could reflect better staffing, case mix, seasonality or learning. High quality individual observations still may not be independent when teams share workflows.
What to measure
Report eligible cases, successes, confirmed negatives, unresolved outcomes and harms by workflow. Use the team or session as the analysis unit when assignment occurred at that level.
Design a test
Predefine a comparable, counterbalanced tabletop; replicate it across independent teams. When testing a pair of changes, design four matched conditions before making an interaction claim.
When to pause or retest
Require independent evidence and a remedy for material harms before moving beyond a tabletop. A positive single-run result can inform a retest.
What this evidence can support
Missing-outcome bounds and descriptive differences do not correct bias, identify effects, or quantify statistical uncertainty.
Trace the decision
From a faster AI step to a decision about value
Faster AI drafting only matters if audited total workload, service or cost improves under the same review requirements. Measure the burden shifted to downstream teams.
- Match eligibility and staffing across workflows
- Audit total time, mistakes and rework
- Compare incremental resources against a predeclared target
Hold capacity fixed and retain error records. Repeat across different case mixes and state which costs and harms were left out.
When this overlaps another topic
An AI comparison needs a matched human process
Comparing AI with a fragmented baseline credits the AI arm for improved formatting and coordination. Compare the same organized workflow with and without the tested AI step.
Check: Use the same eligible cases, staffing, release times and independent scoring. Report both contrasts and retain outputs from errors and overrides.
AI value, staff time and errors
Missing cases can change who appears to benefit
If unresolved outcomes are concentrated in an underserved group, a complete-case comparison can flatter overall service and obscure local harm.
Check: Calculate eligible-case bounds for each arm and report them by group where permitted. Design a follow-up that addresses the missing reporting, then reassess the group floor.
Sources and scope
These references support particular methods or historical context. The research design above is a proposal.
- HM Treasury Magenta Book: Analytical methods · Methods guidance
Explains comparison designs, attribution, evidence synthesis and value for money methods.
- FEMA Homeland Security Exercise and Evaluation Program · Exercise guidance
Provides exercise design, observation and improvement planning resources.
- NIST AI Risk Management Framework: Measure · Methods guidance
Supports documenting tests, errors, human overrides and independent review.
Original hypotheses to examine
These source records remain untested.