Distributed Systems / Consensus Engineering
RaftNova
The point of the system is not simply a distributed store. It is to implement consensus correctly — Figure 8, WAL persistence, votedFor durability, and pipelined replication — and to make every invariant visible and testable.
RaftNova is a correctness-first distributed key-value store implementing the complete Raft consensus algorithm in Go without any consensus library. Five nodes coordinate via gRPC, persist state through a WAL with CRC32-IEEE integrity, replicate log entries with etcd-style pipelining, and expose a React 19 dashboard that visualizes election state, replication lag, and cluster topology in real time.
Pipelined Throughput
~7,400
ops/sec — etcd-style concurrent pipeline, 3-node cluster
Sequential P50
2.1ms
WAL fsync + gRPC round-trip + majority ACK, 3-node
Cluster Size
5 nodes
tolerates 2 simultaneous failures with 3-node majority
The Core Problem
Distributed agreement is not hard. Correct distributed agreement is.
Most distributed KV stores hide the hard part behind library calls. RaftNova implements the full Raft paper from scratch — election, log replication, WAL durability, crash recovery — with every correctness invariant enforced in code, not assumed by the framework.
The three canonical failure modes that make consensus hard: split votes from correlated election timeouts, log divergence after partition, and the Figure 8 problem where a previous term's entry can be silently overwritten after leader change. All three are addressed explicitly.
Pipelined replication (etcd-style) means WAL fsync and gRPC replication happen concurrently on the leader — not sequentially. This is what separates ~7,400 ops/s from ~1,200 ops/s under concurrent load.
Naive implementation
~1,200 ops/s
sequential fsync + replicate
RaftNova (pipelined)
~7,400 ops/s
etcd-style concurrent pipeline

Five nodes in a Raft cluster. Each node runs the full stack: HTTP API, Raft consensus engine, WAL, and in-memory KV store. The dashboard polls /status at 250ms to build the live cluster view.
Architecture
System Components
Seven files. Each owns a precise correctness boundary. Each has a defined failure mode.

Full system: Client → HTTP API (307 redirect for non-leaders) → Raft Consensus → WAL (CRC32 + fsync) → KV Store state machine. Dashboard polls /status at 250ms intervals.
| File | Responsibility |
|---|---|
| node.go | Raft state machine — election, replication, commit, apply |
| replication.go | Pipelined follower sync, matchIndex/nextIndex bookkeeping, commitIndex advance |
| wal.go | Append-only persistence with CRC32 integrity and fsync |
| log.go | 1-indexed in-memory Raft log with truncation |
| http.go | REST API — PUT/GET/DELETE /kv, /status, /healthz, /simulate-crash |
| grpc.go | Lazy-connect gRPC transport with connection pooling |
| store.go | In-memory map[string]string state machine with Apply() |
Write Path
The ordering is not arbitrary — violating it loses data.
Every write follows this exact 6-step pipeline. Steps 3 and 4 run concurrently — this is the etcd-style pipelining that drives the throughput difference.
Client PUT → HTTP APINon-leader nodes return 307 Redirect to leader's address. Only leader proceeds.
SubmitCommand(entry)Entry appended to in-memory log. Assigned log index and current term.
WAL.Append(entry) + fsync()Protobuf-encoded entry written with CRC32-IEEE checksum. fsync() before replication begins.
SendAppendEntries (parallel)Pipelined gRPC replication to all followers simultaneously. Does not wait for WAL on followers before sending.
Majority ACK → advanceCommitIndex()Once ⌈N/2⌉+1 peers confirm matchIndex >= entry.Index, commitIndex advances. Term check (Figure 8) enforced here.
applyLoop() → KV Store.Apply()Background goroutine applies committed entries to in-memory map. lastApplied catches up to commitIndex.
The Lag Metric
commitIndex − lastApplied is the observable replication lag. The dashboard surfaces this per-node in real time. A lag above zero means the state machine is catching up to the committed log. A sustained lag on a follower indicates a slow disk or network issue.

The 8-stage consensus pipeline: Client → HTTP (307 redirect) → Append to Log → WAL Write + gRPC Replicate (parallel) → Majority ACK → Advance commitIndex → Apply to KV Store → 200 OK.

Three-state FSM: FOLLOWER → CANDIDATE on election timeout → LEADER on majority votes. Any node discovering a higher term immediately reverts to FOLLOWER. Candidate → FOLLOWER on higher term prevents split-brain.
Leader Election
One mechanism prevents all split-brain scenarios.
Waiting state — election timer running (150–300ms randomized)
Resets timer on any valid AppendEntries or RequestVote
Increments term, votes for self, sends RequestVote to all peers
Moves to LEADER on majority, back to FOLLOWER on higher term
Sends heartbeats every 50ms. Accepts client writes. Replicates log.
On election: submits no-op entry (§5.4.2) to commit pending entries
Split Vote Prevention
Randomized election timeouts (150–300ms uniform distribution) make correlated timeouts statistically improbable. The first node to time out has a head start on gathering votes.
No-Op on Election
A new leader immediately submits a no-op log entry in its own term. This forces the Figure 8 term check to pass, committing any pending entries from the previous leader's term.
WAL Format
Every acknowledged byte is on disk. No exceptions.
The WAL uses a binary record format: 4-byte record type (LE uint32) + 4-byte payload length + Protobuf-encoded payload + 4-byte CRC32-IEEE checksum. The CRC32 is computed over the full header + payload.
On replay after a crash, any record with a mismatched CRC32 halts replay at that point. The log self-heals to the last consistent record — partial writes at the tail are silently truncated.
Record Layout
[RecordType: 4 bytes LE uint32 ] [PayloadLen: 4 bytes LE uint32 ] [Payload: N bytes Protobuf ] [CRC32: 4 bytes LE uint32 ] ↑ computed over all three above
Record Types
0x01 LogEntry { term, index, command }
0x02 State { currentTerm, votedFor }
File rotation at 64 MB threshold
→ wal-000001.log, wal-000002.log, ...Durability
fsync() after every record
Batch Optimization
AppendMany() — single fsync for N entries

WAL binary record: 12 bytes fixed header + N bytes Protobuf payload. CRC32-IEEE covers all fields. fsync() after every write. File rotation at 64 MB.
Correctness Invariants
A crash at any point leaves a consistent state.
These invariants are enforced in code with inline comments at the enforcement site — not in documentation that diverges from the implementation.
System Guarantees
| currentTerm only increases | Monotonic term ensures stale leaders step down immediately on any RPC contact |
| votedFor persisted before vote reply | Prevents double-voting across crashes — written to WAL before RequestVoteResponse is sent |
| Committed entries never deleted | Log truncation only removes entries with index > commitIndex |
| Figure 8 term check on commit | advanceCommitIndex() skips any entry whose term != currentTerm — previous terms commit indirectly |
| No-op on election | New leader submits a nil command in currentTerm, forcing prior-term entries to commit safely |
| Reads only from leader | Followers return 307 redirect. linearizable reads guaranteed by leader exclusivity |
| matchIndex only increases | Once a follower confirms an entry, that entry is never unconfirmed — matchIndex is monotone |
| WAL fsync before ack | No client ever receives 200 OK for a write whose WAL record is not on disk |
Engineering Honesty
5 Real Bugs Found, Fixed, and Documented
These are actual bugs discovered during development, in chronological order. Not hypothetical edge cases. Not textbook examples. Real failures that surfaced under real conditions — some of which took hours to trace because the symptom pointed to the wrong module.
Election Timer Not Resetting on AppendEntries
Symptom
Under write load, followers would initiate elections even while receiving valid heartbeats from the elected leader. The cluster entered repeated election storms, stalling all writes.
Root Cause
The AppendEntries RPC handler processed log entries and responded correctly — but forgot to call resetElectionTimer(). The follower's election countdown ran to zero despite receiving valid leader contact.
Fix
One call to resetElectionTimer() at the top of the AppendEntries handler. The fix is one line. The root-cause hunt took two hours because the symptom (split-vote storms) looked identical to a vote-counting bug.
// BEFORE: timer never reset on valid heartbeats
func (n *Node) handleAppendEntries(req *pb.AppendEntriesRequest) *pb.AppendEntriesResponse {
// ... process entries ...
return &pb.AppendEntriesResponse{Success: true}
}
// AFTER: one line fixes liveness
func (n *Node) handleAppendEntries(req *pb.AppendEntriesRequest) *pb.AppendEntriesResponse {
n.resetElectionTimer() // ← this line was missing
// ... process entries ...
return &pb.AppendEntriesResponse{Success: true}
}In Raft, the election timer is the heartbeat of a follower's trust in the leader. Missing a single reset breaks the entire liveness guarantee — the cluster works until it doesn't, then fails visibly under load.
Double-Counting Votes from Duplicate RPC Replies
Symptom
Occasionally a node declared itself leader after receiving what it believed was a majority vote — but only 2 distinct peers had actually voted for it. The cluster briefly had two leaders.
Root Cause
When a RequestVote RPC timed out, the candidate retried. The original reply then arrived after the retry's reply — both replies were counted, inflating the vote total for that peer.
Fix
Added a votesReceived map[NodeID]bool to the election goroutine. Only the first response from each peer is counted. Subsequent responses for the same peer are discarded.
// BEFORE: duplicate replies double-counted
votesGranted := 1 // self-vote
for reply := range replyChan {
if reply.VoteGranted {
votesGranted++
}
}
// AFTER: track per-peer, count once
votesGranted := 1
votesReceived := map[string]bool{}
for reply := range replyChan {
if votesReceived[reply.PeerID] { continue }
votesReceived[reply.PeerID] = true
if reply.VoteGranted {
votesGranted++
}
}The network can deliver duplicate messages. Idempotency is not optional in distributed systems — every counter that affects a safety property must be deduplicated at the source.
WAL Replay Not Restoring votedFor
Symptom
After a crash and restart, a node voted for two different candidates in the same term — directly violating Raft's election-safety property. In theory, two leaders could have been elected.
Root Cause
WAL replay on startup correctly reconstructed currentTerm from persisted state records. It did not reconstruct votedFor. The node restarted believing it had never voted in any term.
Fix
The WAL replay path now fully restores both currentTerm and votedFor atomically from the same state record. Both fields are written together on every state change.
// BEFORE: replay restored term but not votedFor
func (n *Node) replayWAL(entries []WALEntry) {
for _, e := range entries {
if e.Type == WALStateRecord {
n.currentTerm = e.Term
// votedFor never restored ← BUG
}
}
}
// AFTER: both fields restored together
func (n *Node) replayWAL(entries []WALEntry) {
for _, e := range entries {
if e.Type == WALStateRecord {
n.currentTerm = e.Term
n.votedFor = e.VotedFor // ← restored
}
}
}Every piece of Raft state that affects safety must survive a crash. Missing votedFor is not a performance bug — it is a correctness violation that can elect two leaders in the same term.
commitIndex Advance Without Term Check (Figure 8)
Symptom
After a leader change, a key written under term 3 returned a stale value. The new leader had silently committed an old entry from term 3 without verifying it was safe to do so.
Root Cause
advanceCommitIndex() counted how many peers had matchIndex >= N, then committed. It did not check whether log[N].term == currentTerm. This violates the Raft paper's Figure 8 safety rule: a leader can only commit entries from the current term.
Fix
Added a term check inside advanceCommitIndex(): `if log.TermAt(idx) != currentTerm { continue }`. Entries from previous terms can only be committed indirectly — by committing a current-term entry that follows them.
// BEFORE: committed entries from any term
func (n *Node) advanceCommitIndex() {
for idx := n.commitIndex + 1; idx <= n.log.LastIndex(); idx++ {
if n.countMajority(idx) {
n.commitIndex = idx // ← UNSAFE for old-term entries
}
}
}
// AFTER: only commit current-term entries
func (n *Node) advanceCommitIndex() {
for idx := n.commitIndex + 1; idx <= n.log.LastIndex(); idx++ {
if n.log.TermAt(idx) != n.currentTerm { continue } // ← Figure 8
if n.countMajority(idx) {
n.commitIndex = idx
}
}
}Figure 8 is the hardest part of the Raft paper for a reason. The term check is non-obvious, and violating it allows data to silently disappear: an entry replicated to a majority in term 3 can be overwritten by a term-4 leader who hasn't seen it.
nextIndex Initialization Truncating Ahead Followers
Symptom
After leader election, keys that had been written and acknowledged by the old leader were absent from the cluster. The new leader had silently deleted committed entries from followers.
Root Cause
New leaders initialize nextIndex[peer] = lastLogIndex + 1 based on their own log length. If a follower had a longer log than the new leader (having received replication before the leader crashed), the AppendEntries probe sent empty entries — and the follower's conflicting suffix was truncated.
Fix
AppendEntries truncation logic now only removes conflicting entries — those whose term does not match. Extra valid entries beyond the leader's current index are preserved. The leader discovers each follower's true state through the probe protocol.
// BEFORE: truncated valid follower entries
func applyAppendEntries(leaderPrevIdx int, entries []Entry) {
n.log.Truncate(leaderPrevIdx + 1) // ← deleted committed entries
n.log.Append(entries...)
}
// AFTER: only truncate conflicting entries
func applyAppendEntries(leaderPrevIdx int, entries []Entry) {
for i, e := range entries {
idx := leaderPrevIdx + 1 + i
if idx < n.log.Length() && n.log.TermAt(idx) != e.Term {
n.log.Truncate(idx) // only truncate at conflict
break
}
}
n.log.AppendIfMissing(entries...)
}Log length is not a proxy for log correctness. A newly elected leader must discover each follower's actual state through the AppendEntries consistency check, not assume all followers have exactly its own log.
Performance
Benchmarks — 100-byte values, 5-node cluster
Sequential mode pays the full fsync + gRPC round-trip latency per write. The latency floor is: WAL fsync (leader) + gRPC to 2 peers + WAL fsync (peers) + majority ACK — approximately 2.8ms on commodity hardware. Pipelined mode overlaps these costs across concurrent writes.
Benchmark Results
| Workload | Cluster | Throughput | P50 | P99 | Note |
|---|---|---|---|---|---|
| Concurrent writes (pipelined) | 3-node | ~7,400 ops/s | — | — | Full etcd-style pipelining |
| Sequential writes (no pipeline) | 3-node | ~1,200 ops/s | 2.1ms | 8.4ms | WAL + gRPC round-trip |
| Sequential writes (no pipeline) | 5-node | ~900 ops/s | 2.8ms | 11.2ms | 2 extra peers in quorum |
The 2.8ms Sequential Write Floor
Durability over throughput
Sequential writes are hard-capped by the physical cost of two fsync() barriers (leader WAL + majority follower WALs) plus gRPC round-trip latency. 2.8ms P50 on a 5-node cluster is close to the hardware floor for unbatched synchronous consensus on commodity NICs. No amount of CPU tuning reduces this — the bottleneck is network + disk.
Pipelining Multiplies Throughput 6×
etcd-style concurrent pipeline
In pipelined mode, the leader sends the next AppendEntries before waiting for the previous one's ACK. WAL fsync and gRPC replication overlap across concurrent writes. 7,400 ops/s vs 1,200 ops/s is not a tuning win — it is a structural win from overlapping I/O that would otherwise serialize.
5-Node vs 3-Node Latency Cost
Quorum size matters
Adding two more nodes to the quorum increases P99 from 8.4ms to 11.2ms in sequential mode. This is the raw cost of requiring one additional network round-trip for majority. The leader must wait for 3 of 5 nodes vs 2 of 3. Each additional peer in the quorum path adds latency proportional to the slowest peer in the majority.
Read Latency: Zero Extra Hops
Leader-only reads
Reads are served directly from the leader's in-memory KV store. No disk reads. No quorum required. The cost is: non-leaders pay one 307 redirect round-trip. The leader serves the value from its local map in microseconds. This is intentional — linearizability at the cost of a redirect is a better tradeoff than stale reads from followers.
Design Tradeoffs
| Decision | Why | Cost |
|---|---|---|
| WAL fsync on every record | No acknowledged write can be lost across crash | ~1-5ms per write; sequential throughput bounded by disk |
| Leader-only reads | Linearizability — no stale reads possible | Non-leaders pay one 307 redirect round-trip |
| In-memory KV store | O(1) reads after apply; no disk on read path | State is volatile; WAL replay required on restart |
| Pipelined replication | Overlaps disk I/O and network I/O across concurrent writes | Out-of-order ACKs require careful matchIndex bookkeeping |
| Single goroutine owns Raft state | Eliminates data races; all invariants hold trivially | No parallelism in consensus path; throughput bounded by one core |
| gRPC transport | Multiplexed, streaming-ready, protobuf-typed RPCs | Connection overhead; not ideal for sub-millisecond latency targets |
Running It
How to Run RaftNova
The cluster ships with a React 19 + D3 live dashboard that visualizes election state, log replication lag, and cluster topology in real time — the same observability surface a production Raft cluster would need.
Live Demo
RaftNova Testing Arena
A deployed 5-node Raft cluster with the full observability dashboard. Write keys, kill nodes, trigger elections, observe replication lag — all live.
Docker Compose (5-node cluster)
# Spin up the full 5-node cluster docker-compose up --build -d # Verify all nodes are healthy curl http://localhost:8080/status curl http://localhost:8081/status # Nodes available at: # HTTP: ports 8080–8084 # gRPC: ports 9090–9094
Build from Source
# Requires Go 1.22+ and protoc go build -o bin/server ./cmd/server # Run local 5-node cluster (PowerShell) ./start_servers.ps1 # Run tests go test -v ./internal/... go test -v -timeout 120s ./tests/chaos/... # Dashboard cd dashboard && npm install && npm run dev # http://localhost:5173
HTTP API Reference
All requests to any node — non-leaders return 307 redirect.Write a key
curl -X PUT http://localhost:8080/kv/mykey \
-d '{"value": "hello"}'Read a key (linearizable)
curl http://localhost:8080/kv/mykey
# → {"value": "hello"}Cluster status
curl http://localhost:8080/status
# → {node_id, state, term, commit_index,
# last_applied, peers, leader_id}Chaos — crash simulation
curl -X POST http://localhost:8080/simulate-crash # Triggers crash scenario for observability curl http://localhost:8080/healthz # liveness probe
Dashboard Panels
NodeCard
Per-node state chip: LEADER / FOLLOWER / CANDIDATE. Live term, commitIndex, lastApplied, and replication lag (commitIndex − lastApplied).
CommitChart
Rolling 30-second sparkline of commit_index per node. Election events marked as vertical lines on the timeline.
TopologyGraph
D3 force-directed graph of peer edges derived from status.peers. Lag severity visualized as edge color overlay.
WriteTester
Target-node selector + 307 redirect observability. Send writes, observe redirects to leader, verify replication.
ChaosPanel
Trigger simulate-crash endpoint with configurable batch flood writes. Observe election and recovery in real time.
Cluster Polling
All 5 nodes polled at 250ms via /status. Global cluster view assembled client-side from node responses.
Lag Monitor
commitIndex − lastApplied surfaced per-node. Sustained lag indicates follower disk or network pressure.
Election Markers
Term changes detected via CommitChart delta. Visual event markers show when elections occur and which node wins.
Next Case
NullRing