Writing to disk without lying

STORED is six characters, and every post in this series has been about earning them.

This one is about the last stretch: the bit between "my program has your data" and "your data is on a disk and will still be there after someone trips over the power cable."

It turns out there are more places to lie than I expected.

The lie I didn't know I was telling

You write to a file. The call returns. No error. Done, surely?

No. Your data is in the operating system's memory, the page cache, and the OS will get around to putting it on the disk when it feels like it. Maybe in a few seconds.

That's not useless. If your process is killed right then, the data still lands, because the kernel is the one holding it and the kernel is fine. But pull the plug and it's gone, and your program already told the client everything was fine.

To actually get it onto the device you have to ask: fsync, which in Java is force(true) on the channel. Now it's the disk's problem.

Even that isn't the end of the honesty chain, which I found slightly depressing: drives have their own caches, and some of them will happily say "written" before the bytes are anywhere permanent. You can only be as truthful as the layer beneath you. But fsync is where an application's responsibility ends, and skipping it is where most "we lost data and don't know why" stories start.

One thread, one fsync, many writes

Here's the problem with being honest: an fsync on my laptop takes around 5 milliseconds. If every write waits for its own fsync, that's 200 writes a second, and no amount of clever code changes it.

So BeeDB does what every serious log does. Writers don't touch the file at all. They drop their record on a queue and get back a future.

One thread owns the WAL file. It blocks on that queue, and when something arrives it doesn't grab just that one record, it takes everything waiting, writes the whole lot into the file, and calls fsync once for the batch. Then it completes all those futures together.

That's group commit, and it's the nicest kind of optimisation: the busier you get, the better it works. In my measurements the batches averaged about eleven records per fsync. Eleven writes, one 5-millisecond wait, shared.

It also keeps the fsync off everyone else's critical path. An earlier version of BeeDB held the Raft node's lock while fsyncing, and the node looked dead to its peers for five milliseconds at a time, which was enough to start elections. That's a story for the mistakes post.

What a record looks like, and what a crash does to it

Records are appended, never updated. Each one is a length, a checksum, the payload, and a newline:

00000142 3661184229 {"index":57,"term":4,"command":"set x 0 0 1\r\ny\r\n"}

Now kill the node halfway through writing that. You get half a record on disk.

On restart, replay walks the file record by record. It reads the header, reads that many bytes of payload, recomputes the checksum, and compares it to the stored one. The moment they don't match, it truncates the file there and stops. Everything before that point is intact and real. The half-written tail is gone.

Why not just trust the length field? Because the length only tells you how many bytes were meant to be there. It says nothing about whether the bytes that arrived are the ones you wrote. The checksum is the part that knows.

The bytes were durable; the name wasn't

Compaction rewrites the log into a fresh file and swaps it in. The temp file is written, fsynced, and then renamed over the old one with an atomic move.

Atomic move, fsynced file. Safe?

Not yet. The rename isn't inside the file, it's a change to the directory. The bytes can be durable while the name that finds them isn't. Crash at the wrong moment and you can come back to a directory that still points at the old file, or at nothing.

So after the rename, BeeDB opens the directory and fsyncs that too. Three lines, and one of the classic ways to lose a database if you skip them.

When fsync fails

Which brings us to the ugliest corner of this whole area.

fsync returns an error. What now? The instinct is to retry, because that's what you do with failed I/O.

Don't. On Linux, when writeback fails the kernel can report the error to exactly one fsync caller and then mark those pages clean, as though they'd been written. Your retry comes back successful, on data that no longer exists anywhere. An error you can handle; a false success you cannot.

PostgreSQL learned this the hard way in 2018, in an episode the internet named fsyncgate. Their fix shipped in the February 2019 releases, 11.2 and friends, back-patched to every supported version, and it's blunt: on fsync failure, PANIC, then replay from the write-ahead log. They also added a data_sync_retry setting for anyone certain their kernel doesn't behave that way. The release announcement's own wording is worth repeating: if PostgreSQL reissues the fsync, it will succeed, "but in fact the data has been lost".

BeeDB does the same thing in miniature. If a WAL write fails, the node doesn't retry and doesn't continue: it logs the cause, writes it to stderr, flushes the log queue so the reason isn't lost, and dies.

Dying sounds dramatic for a database. It's the safest thing available. A node that halts is a node the other two can out-vote and carry on without. That's the entire point of running three. A node that keeps serving with a hole in its log is a node that will happily hand you the wrong answer forever.

What I can honestly claim

Put the pieces together and STORED means: the entry is in the log, in a checksummed record, fsynced on a majority of the machines' disks, and applied to the cache.

And here's the honest edge of that claim. My durability test kills nodes in the same process, so anything sitting in the page cache survives the "crash". It proves nothing was lost to truncation bugs, commit-index mistakes or bad log repair: real durability failures, all of them. It does not prove the fsync is where I say it is. That would need a test double that throws away unforced writes, and I haven't built it.

Which is the honest version: the design is right, and one specific part of it is argued rather than measured.