← All drills
Plexivo reliability report
Region lost: the Plexivo Brain detected it and rebuilt everything in a second region
Objective.
Take a whole region out of service and let Plexivo rebuild the site in a second region and move every resource, with no operator.
Scenario.
The primary region's provider interface was made unreachable and every machine of the site was stopped.
Expected behaviour.
- Declare the region out from independent signals.
- Build a replacement site in the second region.
- Restore every database from its replicated backups, with its data.
Measured recovery.
- Detection time
- 1 min 52 s
- Recovery time RTO
- 8 min 43 s
- Target 10 min · met
- Recovery point RPO
- 0 s
- Every committed write kept.
- Application health
- Pass
- Database consistency
- Pass
Customer impact.
- Application availability
- App answering 502 s after the loss
- Management operations
- Fully automatic, no operator
- Data integrity
- Verified
- Committed data
- Fully preserved
What happened.
- Region taken out of service.
- Region declared out; replacement ordered in the second region.
- Replacement site active.
- Every resource moved.
- App answering.
- PostgreSQL and MySQL restored from replicated backups; every row present.
Correctness checks.
- Committed data
- Fully preserved
Every measurement of this drill.
| Date | Measured recovery | Report |
|---|---|---|
| 8 min 43 s | This report | |
| 23 min 47 s | View report |
What we learned.
- A whole region is replaced in under 9 minutes, with no operator and no committed data lost.
What we changed.
- New sites start from a prepared image in both regions.
- Every new database takes its first full backup at once, so it can be restored elsewhere the same day.
Scope of this drill.
Regional recovery is part of Enterprise and uses a recovery region set up in advance.