INDEPENDENT POLICY & RESEARCHConceptual framework Edition 0.22.0
Evaluation and cooperation

Evidence, pilots and hypothesis testing

Ask what decision the evidence should inform, which outcome would change the decision, and what would have happened under a credible alternative.

How it could work

Write a comparison protocol before testing. Hold eligibility, resources and release times constant, freeze the tested design, and judge outputs against an independent scoring rule.

Where it can fail

A before-and-after improvement could reflect better staffing, case mix, seasonality or learning. High quality individual observations still may not be independent when teams share workflows.

What to measure

Report eligible cases, successes, confirmed negatives, unresolved outcomes and harms by workflow. Use the team or session as the analysis unit when assignment occurred at that level.

Design a test

Predefine a comparable, counterbalanced tabletop; replicate it across independent teams. When testing a pair of changes, design four matched conditions before making an interaction claim.

When to pause or retest

Require independent evidence and a remedy for material harms before moving beyond a tabletop. A positive single-run result can inform a retest.

What this evidence can support

Missing-outcome bounds and descriptive differences do not correct bias, identify effects, or quantify statistical uncertainty.

Trace the decision

From a faster AI step to a decision about value

Faster AI drafting only matters if audited total workload, service or cost improves under the same review requirements. Measure the burden shifted to downstream teams.

  1. Match eligibility and staffing across workflows
  2. Audit total time, mistakes and rework
  3. Compare incremental resources against a predeclared target

Hold capacity fixed and retain error records. Repeat across different case mixes and state which costs and harms were left out.

When this overlaps another topic

An AI comparison needs a matched human process

Comparing AI with a fragmented baseline credits the AI arm for improved formatting and coordination. Compare the same organized workflow with and without the tested AI step.

Check: Use the same eligible cases, staffing, release times and independent scoring. Report both contrasts and retain outputs from errors and overrides.

AI value, staff time and errors

Missing cases can change who appears to benefit

If unresolved outcomes are concentrated in an underserved group, a complete-case comparison can flatter overall service and obscure local harm.

Check: Calculate eligible-case bounds for each arm and report them by group where permitted. Design a follow-up that addresses the missing reporting, then reassess the group floor.

Fairness and who receives service

Sources and scope

These references support particular methods or historical context. The research design above is a proposal.

Original hypotheses to examine

These source records remain untested.