How to Measure Civilian Resilience Outcomes
Define the units, denominators and evidence needed to measure service continuity, warning accuracy, human workload and independent recovery.
What counts as a civilian outcome?
Choose a measure that changes when people actually receive a service or when an essential function recovers. A notice sent is an activity. A notice independently checked before the decision deadline is a process result. A delivery confirmed usable and received within the period is a civilian service result.
Publish those layers separately. Pair a primary outcome with a quality or safety measure so a speed gain cannot hide an error, an unmet local service requirement, or a disclosure outside the agreed purpose.
Pick a numerator, denominator and observation window
| Measure | Definition and unit | Where to calculate |
|---|---|---|
| Confirmed delivery | Usable units delivered during the stated period. Compare with new demand in the same commodity units. Keep offers, pending transfers and receipts distinct. | Daily inventory and confirmed receipts |
| Service coverage | Sum of service delivered toward demand divided by the sum of known demand over eligible intervals. Report low-coverage hours and intervals with unknown coverage separately. | Service Coverage and Shortfall Hours Calculator |
| Independent restoration | Minutes from outage to verified restoration. Record the data gap in minutes separately, and identify which essential functions can operate without the shared channel. | Recovery Time and Data-Loss Objective CheckDependency and tested fallback exercise |
| Alert quality | Precision uses classified alerts. Recall uses distinct audited events in a defined window. Unknown alerts and unaudited misses remain visible in separate counts. | Alert, action and interruption ledger |
| Staff workload | Total paid or volunteer staff-hours for the same cases, including AI drafting, human review and confirmed rework. Compare observed errors on the same audit standard. | Measured manual and AI review workload |
| Distribution | Service received divided by eligible need within each aggregate group. Record denominators, overlap and suppressed small cells. | Aggregate service coverage differences |
Report a measured difference with its limits
Compare a share in percentage points. In a fictional audit of 100 eligible records per arm, the human-led team verifies 72 by the deadline and the AI-assisted team verifies 78. The descriptive difference is 78% − 72% = 6 percentage points. Report both counts and the eligibility definition. Track whether the six extra verifications came with more incorrect clearances or staff-hours.
A single team exercise per arm cannot establish that the result would repeat across teams or sites. Use independent replications, report variation across runs, and choose uncertainty methods suited to how cases were assigned. Paired exercises require paired analysis. If whole teams were assigned together, records from a team should not be treated as independent runs.
If an independently justified before/after comparison has treated and comparison groups observed over aligned periods, describe the change in the treated group minus the change in the comparison group. Discuss the credibility of parallel trends, changing group composition, concurrent changes and spillover before any causal interpretation. The Workbench computes a descriptive four-cell contrast. HM Treasury Magenta Book, analytical methods
Check paired rates and descriptive difference-in-differencesMake missing and disputed evidence visible
Declare the expected cases and separate confirmed positive, confirmed negative, unknown, and outside scope. If 70 of 100 eligible records were verified and 10 are unresolved, the share over all eligible records would range from 70% to 80% if each unresolved case can contribute either zero or one. Report this as a missing-outcome range. A statistical confidence interval requires a separate uncertainty calculation.
Give the cutoff and time zone for event, publication, retrieval and revision dates. Collapse copies into their declared common origin when evidence supports that relationship. If groups overlap or definitions changed, report the affected measures separately and explain the comparison limit.
Trace shared or unknown evidence originsCheck source freshness and comparabilityPublish enough detail to reproduce the result
- Versioned protocol, fixture or public source reference, eligibility rules, raw aggregate counts, measurement units, and time windows.
- Numbers of missing, late, disputed, excluded or suppressed cases by arm, with a reason and the sensitivity of the result to assumptions.
- Separate estimates for civilian service, safety, staff burden and fallback. Disclose qualitative harms or material adverse events alongside numerical results.
- The supplied assumptions and any independent review. Report what would change the interpretation. The Lab has no validated MAR effect or estimate of war probability.