Building Resilient AI Infrastructure and Enterprise Software
Resilient AI infrastructure starts with the service that has to continue. Map the model, software, data, identity, network, power and people it depends on. Set a tolerated level of degraded operation, design a fallback that can survive the same disruption, and verify it in a controlled exercise.
Define the service and tolerated degradation
Write the smallest service commitment an operator needs to meet during a disruption. Specify who it serves, the minimum throughput, and the longest acceptable interruption. A request service might continue to accept and record urgent requests while deferring classification and routine notifications. Record who can switch modes and what triggers the switch.
Map dependencies across the full workflow
- Application: intake, validation, routing, queues and software release dependencies.
- AI: model endpoint, model version, retrieval or other data sources, and output review.
- Operations: identity, credentials, storage, network, electricity and recovery staff.
- Fallback: the dependencies of every substitute, including staff availability and access to current records.
Two model endpoints provide redundancy. If they share identity, network or a data source, a failure in that shared component can remove both. Evidence of independence comes from the dependency map and a test that actually removes the common component.
How AI changes enterprise software failure modes
A conventional software outage may fail visibly. An AI endpoint can return a well-formed response while offering a wrong classification or relying on outdated evidence. Model changes, input drift and delays also affect a service while its health check still reports availability.
Check output quality, latency and input freshness independently. Put review and fallback at the service boundary. When rolling back a model or configuration, verify that its data format, prompt or workflow version, and access rules are still compatible.
Compare redundancy, degraded operation and manual fallback
Illustrative exercise. A dispatch service must keep accepting urgent requests during a two-hour outage. Compare the three designs with the same staff and request volume. Dependencies below are stipulated.
| Design | Failure covered | Shared dependency | Residual gap | Test | Trade-off |
|---|---|---|---|---|---|
| Two model endpoints | One model endpoint stops responding | Both use one network and identity system | A shared outage blocks both routes | Switch to the alternate endpoint in an approved test and verify output | Extra provider costs and version checks. |
| Degraded intake and queue | Model classification unavailable | Intake still uses its network and storage | Intake fails if the shared platform is unreachable | Disable classification in the exercise and verify urgent records | Staff triage the queue and reconcile later. |
| Local phone and paper intake | Web intake or AI provider unavailable, if phone and staff remain | Staff and phone service may still share local power | Prolonged loss of staff or telephone service | Rehearse calls, logging and later reconciliation | Slower processing and additional staffing. |
Assign ownership and verify recovery
Set a time target and a tolerable data-loss window for each essential function. Keep a record of what was received while degraded and compare it with restored records before closing the exercise. The service owner signs off on restored function and integrity. The technical owner records version and access checks.
The dependency and tested fallback exercise evaluates retained, unavailable and unknown functions from declared dependencies and evidence. The data-withdrawal tabletop scenario gives another way to ask what remains usable after shared access is withdrawn.
When a fallback depends on a battery, compare energy, load and reserve using the backup battery runtime calculator.
Sources and what they support
- NIST — AI Risk Management Framework 1.0: AI Risks and Trustworthiness. Published January 2023. Supports designing for resilience and evaluating changes to performance across conditions. Retrieved .
- NIST — Contingency Planning Guide for Federal Information Systems (SP 800-34 Rev. 1). Published 2010. Supports contingency priorities, alternative processing, recovery objectives and verification. Retrieved .
- NIST — Guide for Cybersecurity Event Recovery (SP 800-184). Published December 2016. Supports exercising recovery plans and validating a return to service. Retrieved .