Objective.
Stop the machine running a single-instance PostgreSQL database and measure how long the database takes to serve again from another machine.
Scenario.
The machine was stopped at the cloud provider while an application and a cache on the same site were probed continuously.
Expected behaviour.
- Confirm with the provider that the machine is down.
- Move the database and its volume to a running machine.
- Serve again within the 10-minute drill window.
Measured recovery.
- Recovery time RTO
- 3 min 1 s
- Target 10 min · met
- Database consistency
- Pass
Customer impact.
- Application availability
- Cache unaffected; the App's longest pause was 10.8 s
- Data integrity
- Verified
What happened.
- Machine stopped at the provider.
- Provider confirmed the machine stopped.
- Database serving from another machine.
- Machine running again.
What we learned.
- The database's volume followed it to a running machine.
What we changed.
- Every machine now joins through a private load balancer instead of one machine's address.
- Faster boot storage and steadier election timing for the coordination service.
Scope of this drill.
A single-instance database has no standby, so recovery depends on moving it. High-availability databases fail over in seconds.