Eitan Blumin can’t reach quorum:
The incident started with a late-night phone call from one of our customers (it’s always a late-night phone call, isn’t it?).
They reported that during a DR exercise on their production environment (Chaos Engineering, anyone?) their entire cluster failed and they weren’t able to bring any of the replicas back online.
Click through for the full story, including what happened, why it happened, and what you can do to prevent similar problems in the future.