Two is one: why high availability starts at three
What split brain is, why a two-node cluster cannot avoid it and why every Claryon cluster has three nodes
Network engineers know the saying "two is one and one is none." It is why we buy redundant supervisors, run VRRP pairs and dual-home our uplinks. So when people see that a highly available Claryon cluster needs three nodes, their first question is a fair one: why not two?
The short answer is split brain. The longer answer starts with something that happened in our own lab.
Forty seconds with two masters
While we were building Claryon's cluster updates, we ran a drill: take the node that holds the cluster's address out of service and watch the address move to a healthy node. It moved. Then something we did not expect happened. For about 40 seconds, two nodes both announced that they owned the address.
Nothing had crashed. Every node was healthy and the network carried traffic normally. What it did not carry reliably were the small multicast messages VRRP uses to say "I am the master, stand down." The virtual switch in our lab was filtering multicast (IGMP snooping), so each of the two backup nodes concluded that nobody was in charge and promoted itself.
Plenty of production networks run IGMP snooping by default, so this was not a lab curiosity. We changed the design: cluster nodes now send those messages directly to each other instead of relying on multicast, and the problem disappeared.
The lesson reaches well beyond VRRP. A node that stops hearing its partner cannot tell whether the partner died or the network between them failed. From where it stands, the two look exactly the same.
The two-node dilemma
Now apply that to a two-node cluster that holds data: a database, an event store, a list of open alarms. The link between the nodes breaks. Each node has to decide on its own, with no way to ask the other. It has two options, and both are bad.
Keep going. If the partner is in fact alive and makes the same choice, both nodes accept changes independently. That is split brain: two versions of the truth that drift further apart every second. When the link returns there is no safe way to merge them. In a monitoring system that means an alarm acknowledged on one side and still open on the other, or events that exist in one history and not the other.
Stop. This is safe, but now losing either node stops the whole cluster. That is exactly what high availability was supposed to prevent.
No setting fixes this. Two nodes simply do not have enough information to make the right call.
Why three works: the majority rule
A cluster of three decides by majority. The rule is simple: a node may only act as the leader while it can see a majority of the cluster, which with three nodes means two.
- One node fails. The remaining two are still a majority, so they carry on.
- The network cuts one node off. The pair knows it is the majority and carries on. The isolated node knows it is in the minority and steps back on its own. Anything that decides by majority never has a second leader.
- Two nodes fail. Now no majority exists, so the last node stops. It stops rather than risk writing a second version of the truth.
Three is the smallest cluster that survives one failure without any risk of split brain. The same rule sits inside most systems that have to agree with themselves, from etcd to Kafka-compatible streaming platforms, and it is why their documentation recommends odd numbers of nodes.
Why not four? The majority of four is three, so a four-node cluster still survives only one failure. You pay for an extra node and gain no resilience. The next step up is five, which survives two.
What about two nodes and a witness?
There is a fair alternative, and some products offer it: two full nodes plus a small third voter, often called a witness or tie-breaker. The witness stores no data. It only casts the deciding vote when the two real nodes lose sight of each other. It is cheaper than a third full node.
The trade-off is worth knowing. A witness breaks ties but holds no copy of your data, so you keep two copies instead of three. It also has to live somewhere that cannot fail together with either node. A witness on the same host as one of them breaks the moment that host does.
Claryon uses three full nodes today because each one also keeps a complete copy of your events, alarms and configuration. When one node goes down you still have two complete copies, not one copy and a vote.
How this shapes a Claryon cluster
- Three nodes, three complete copies. Losing any one node stops no function, and the two remaining nodes still hold complete copies.
- One address for your devices. Devices send their traps to a single cluster address, and the cluster moves that address to a healthy node when it needs to. In our lab drills, when the node holding the address became unhealthy, the address moved within about six seconds and went unanswered for about half a second. The cluster address works differently: it doesn't vote. If the nodes lose sight of each other while the network still works, more than one can claim it. Claryon detects that and says so.
- Updates that never take the cluster down. An update rolls through one node at a time and the node holding the address goes last, so the whole update costs one brief move of that address.
- One port between the nodes. The nodes talk to each other over a single encrypted connection, so the network between them needs one rule, not a long list.
"Two is one" is good advice for links and power supplies. For anything that has to agree with itself about what is true, the number to remember is three.
Thomas Stolt, Netralux