← All drills
Plexivo reliability report
Region lost: the Plexivo Brain detected it and rebuilt everything in a second region
Objective.
Take a whole region out of service and let Plexivo rebuild the site in a second region and move every resource, with no operator.
Scenario.
The primary region's provider interface was made unreachable and every machine of the site was stopped.
Expected behaviour.
- Declare the region out from independent signals.
- Build a replacement site in the second region.
- Restore every database from its replicated backups, with its data.
Measured recovery.
- Detection time
- 4 min 53 s
- Recovery time RTO
- 23 min 47 s
- How long recovery took.
- Recovery point RPO
- 0 s
- Every committed write kept.
- Application health
- Pass
- Database consistency
- Pass
Customer impact.
- Application availability
- App answering 1,221 s after the loss
- Management operations
- Fully automatic, no operator
- Data integrity
- Verified
- Committed data
- Fully preserved
What happened.
- Region taken out of service.
- Region declared out; replacement ordered in the second region.
- Replacement site active.
- Every resource moved.
- App answering.
- PostgreSQL restored from replicated backups; every row present.
- Every MySQL instance answering.
Correctness checks.
- Committed data
- Fully preserved
Every measurement of this drill.
| Date | Measured recovery | Report |
|---|---|---|
| 8 min 43 s | View report | |
| 23 min 47 s | This report |
What we learned.
- A whole region is replaced in about 24 minutes, with no operator and no committed data lost.
What we changed.
- No change needed.
Scope of this drill.
Regional recovery is part of Enterprise and uses a recovery region set up in advance.