Objective.
Stop every machine of a hosting site, start them again, and verify that every application and database returns with its data.
Scenario.
Every machine was stopped at the provider and started again.
Expected behaviour.
- Every application and database returns.
- No data changes.
Measured recovery.
- Recovery time RTO
- 6 min 23 s
- How long recovery took.
- Recovery point RPO
- 0 s
- Every committed write kept.
- Application health
- Pass
- Database consistency
- Pass
Customer impact.
- Application availability
- App back 383 s after the start
- Data integrity
- Verified
- Committed data
- Fully preserved
What happened.
- Every machine started again.
- PostgreSQL answering.
- Every MySQL instance answering.
- App answering through the edge; data unchanged.
Correctness checks.
- Committed data
- Fully preserved
What we learned.
- Everything returned on its own, with its data unchanged.
What we changed.
- The edge proxy refreshes its lookup connection regularly and retries a failed lookup after 1 second.
- The drill now checks every database even while the App is down.
Scope of this drill.
Measured with one App, one PostgreSQL and one MySQL database.