What BeeDB doesn't promise yet

Every database has a list like this. Most of them don't publish it.

I'd rather publish mine, for a selfish reason: the interesting part of building a database isn't the features, it's knowing exactly where your guarantees stop. A system whose author can't tell you that is a system nobody should trust.

So here's where BeeDB's promises end, and what each gap would cost to close.

Reads can be stale

This is the big one.

Write a key and get STORED. Read it a moment later from a different node and you might get nothing back.

Nothing is lost, the write is committed, it's on a majority of disks, and it'll be there. But BeeDB answers reads from whichever node you asked, straight out of that node's cache, with no check that the node is current or even that it's still part of the cluster. A node that's been cut off from the other two will keep serving you confidently out of a snapshot of the past.

The fix is ReadIndex, from chapter 6 of Ongaro's Raft dissertation (the shorter Raft paper is the place to start). Before answering a read, the leader notes its commit index, confirms with a majority that it's still the leader, waits until it has applied everything up to that index, and only then answers.

And it has to be that whole dance. "Just send reads to the leader" isn't enough, because a leader that's been partitioned away still believes it's the leader, that's the exact case that serves stale data with complete confidence. The majority check is what turns "I think I lead" into "I led at the moment your read arrived".

The cost is a round trip per read, or a clock assumption if you use leases instead. I haven't built it.

Writes are at-least-once

Your write times out. That means "I don't know", not "it didn't happen". The entry may be sitting in the log, about to commit.

So you retry. And if the write does commit twice, BeeDB has no way to tell that this was one intent arriving twice. Its log entries carry a request id, but it's generated on the server, so a retry looks like new work.

For set that's harmless: writing the same value twice leaves the same value. For append it isn't: you'd get the suffix twice, silently.

My failover probe measured this: 1,000 keys present for 979 acknowledged writes. Twenty-one writes that were retried after a timeout, all of them harmless because of what I did about it, the gateway exposes only the commands that are safe to apply twice. append, prepend, add and replace exist in the server and aren't reachable from the API.

The real fix is client sessions with request dedup (§6.3): the client generates the id, and the server remembers what it already did for that id. That turns at-least-once into exactly-once. Not built.

An absent node can disrupt a healthy cluster

Suppose one node gets cut off. Its election timer keeps firing, and every time it campaigns it bumps its term. Nobody answers, so it does it again. After a few minutes its term is far ahead of everyone else's, while the other two are serving happily.

Then the network heals.

That node's next message carries its inflated term, the healthy leader sees a term higher than its own, and steps down, even though the returning node's log is behind and it can't win the election it just forced. You lose a working leader to a node that has nothing to contribute, and the cluster stops writing while it sorts out the confusion.

PreVote fixes this (§9.6). Before bumping its term, a candidate asks the others a hypothetical: if I ran, would you vote for me? Nodes say no if they've heard from a leader recently. So the isolated node's term never inflates, and when it comes back it's told, in effect, "we're fine, here's what you missed".

Its companion is CheckQuorum: a leader that can't reach a majority steps down instead of pretending. PreVote keeps outsiders from disrupting a healthy cluster; CheckQuorum keeps a stale leader from lying inside a partition. BeeDB has neither.

The cluster is exactly three nodes, forever

There's no membership change. You can't add a fourth node or replace a dead one while running. The three node ids are configuration, and the code assumes them.

Raft has a proper answer for this too (joint consensus), and it's genuinely fiddly: you're changing who counts as a majority while still needing majorities to agree. Skipping it was deliberate, and it's the reason I describe BeeDB as a demo cluster rather than something you'd run.

There's no partitioning either. Every node holds the entire dataset. That's fine for a demo and it doesn't scale past one machine's memory, the thing an earlier post in this series said was the whole reason distributed databases exist.

I argue for physical durability rather than proving it

This is the gap I'm least comfortable with, so it gets stated plainly.

BeeDB's durability test kills nodes and checks that every acknowledged write comes back. It passes. What it proves is logical durability: nothing was lost to truncation bugs, commit-index mistakes, or bad log repair. Those are real failures and the test rules them out.

What it doesn't prove is that the fsync is where I say it is. The test stops nodes inside the same process, so anything in the OS page cache survives the "crash" and comes back looking durable. Delete the force(true) call from my WAL and that test would still pass. The directory fsync after compaction is untested for the same reason.

Proving the real thing needs a WAL double that throws away unforced writes on a simulated crash, which in turn needs the WAL to be injectable into the Raft node instead of constructed by it. Until that exists: the design is right, and that specific part is argued, not measured.

And one more, because a list like this shouldn't only contain the flattering gaps: I have a test that fails about one run in three: RaftClusterTest.shouldReadWalAfterRestart, which writes a key, restarts a node, and checks what comes back from the WAL. It expects two entries and sees three, about a third of the time. I suspected the rollback path in propose() was truncating the log without journalling it, chased that, and found there was nothing to journal. The test is marked ignored until I know better, which is not the same as fixed.

Why publish this

Every item here is a thing I understand well enough to fix and haven't. That's a different claim from "it's finished", and it's the honest one.

If you're building something like this, I'd suggest writing your own version of this list early. Mine has been more useful than the feature list: it's what I reach for when someone asks what BeeDB actually guarantees, and it's the roadmap.