What happens when the leader dies

Every post so far has ended with some version of "…and if the leader dies, the others take over".

Let's actually kill it.

There's a button on the BeeDB demo that kills whichever node is currently the leader. Not a simulation, a real process, gone. Here's that minute, from both sides of the glass.

The ten seconds before

Press the button and nothing happens yet. The screen says the leader is going to die in ten seconds, and starts counting.

The countdown isn't politeness, it's so you can watch. Keep writing during those ten seconds and everything behaves normally: the leader takes your writes, copies them to both followers, waits for a majority, says STORED. The numbers on screen keep climbing.

Then the countdown hits zero and the process is killed.

The gap

For a moment, nothing happens at all. That's the strangest part of watching this: the interesting part of a leader dying is the silence.

The two survivors don't know anything has happened. They're just not being talked to. Each of them is running a timer that the leader's heartbeats used to reset, and now nothing is resetting it. Each timer is a random length, which is what stops them both waking up at the same instant and splitting the vote.

One of them times out first. It bumps the term, votes for itself, and asks the other for a vote. The other checks one thing, is this candidate's log at least as complete as mine?, and says yes. Two votes out of three is a majority. There's a new leader.

On screen you see the term number tick up. That's the only thing that moves during the gap: the commit index and last-applied numbers sit frozen, because with no leader nothing can be committed and nothing new can be applied. The dead node goes grey.

Then the counters start climbing again, and the cluster is running on two nodes.

What your writes were doing meanwhile

If you send a write during the gap, it fails. The gateway can't find a leader, so it answers 503 with Retry-After: 2 and a message saying an election is in progress. Nothing was written and nothing was promised. Fine.

The interesting writes are the ones that were already in flight when the process died. Take three of them.

One: the leader had it on its own disk, and one follower had acked it. That's two out of three, so it was committed, but the leader was killed before it could say so.

Two: the leader had it on its own disk, and neither follower had it yet.

Three: the leader had already replied STORED.

Write one survives. It's on a majority of disks, so the node holding it cannot be out-voted, and the new leader has it. Write three survives for the same reason.

Write two is the interesting one. It was never committed, so Raft promises nothing about it. In this election it disappears: neither survivor has it, and the new leader's own entries take that slot. Had the node holding it won the election instead, it could have survived and committed later. Uncommitted means undetermined, not doomed.

Now the part I find genuinely uncomfortable. What does the client see for writes one and two?

The same thing, and it is worth being exact about why. Three different things are in play: your connection to the gateway, the gateway's connection to the leader, and the entry's state inside Raft. The one that broke is the middle one. The leader's process died with the gateway's request in flight, so the gateway got a dead socket and answered 503 to your browser. It never found out whether the entry committed, so it says the same thing for a write that did and a write that didn't.

An unconfirmed write is not a failed write. It may have committed, it may not have, and the timeout tells you only that nobody in the path knows which. Which is worth saying slowly: "unconfirmed" isn't a state your data is in. It's a state you are in. The data is perfectly definite. Your knowledge isn't.

(There's a wrinkle here I'm not proud of, and it's worth being clear about which part is code and which part is opinion. The code returns 503 for a dead socket, today, in the version you can read. My opinion is that 504 is the honest answer, because 503 tells a client "this didn't happen, go ahead and retry", which is a lie for write one. It's harmless today only because the demo API exposes just the commands that are safe to apply twice, set, get, delete, and not append, which would double your value on a retry. The real fix is for the client to send a request id and for the server to remember what it already did with it. Raft has a chapter on that. I haven't built it.)

Why write one really is safe

It's worth being precise about why a committed write can't be lost, because when I first wrote this code I got it wrong.

A write is committed once it's on a majority of disks, with the qualification from the Raft post: a leader advances the commit point by counting copies only for entries from its own term, and anything left over from an earlier term becomes committed along with them. The leader is one of those disks, not a special coordinator, just another copy. And my first version of the commit rule quietly forgot that: it counted a follower's fsynced copy plus the leader's copy in memory as a majority, and told the client STORED.

Most of the time that's fine. But kill the leader in the narrow window before its own write hits the disk, and there was only ever one real copy. The surviving node that had it can be out-voted, and the acknowledged write is gone.

The fix is one line: the leader can't commit past its own fsync. It cost 19% of my write throughput, and I'd pay it again, because the faster number belonged to a system that could lose a write it had already promised. That whole story is its own post.

Reading right after an election

One more thing you can see on the demo if you're quick.

Immediately after a new leader takes over, read a key you wrote a second earlier and you might get nothing back. The write isn't lost. The new leader has the entry in its log, but it hasn't applied it to its cache yet, and BeeDB answers reads from whichever node you asked, without checking whether that node is current.

A second later it's there. But "a second later it's there" is exactly the gap between what BeeDB does and what linearizability means, and I'd rather point at it than pretend it's not there.

And then it comes back

Fifteen seconds after it was killed, the dead node restarts on its own.

It boots as a follower, with a log that's missing everything from the last fifteen seconds. The leader notices how far behind it is and starts feeding it the entries it missed, or a whole snapshot, if it's fallen further behind than the log still remembers.

On screen the grey node turns healthy, its numbers race to catch up with the others, and the cluster is back to three.

Then the button lights up again, and you can do the whole thing over.