← Selected work

Storage systems

Production-Grade WAL Engine

A segmented write-ahead log with CRC32C integrity checks, group commit, deterministic crash recovery, checkpoints, fault injection, and operational tooling.

Stack
Go · Cobra · Prometheus · Zap · CRC32C · pprof
Evidence
~1M records/sec replay · 90+ tests · race-safe concurrency

The challenge

A write-ahead log is only useful when it preserves an unambiguous history through partial writes, corruption, process crashes, and concurrent callers. The project treats recovery behavior—not the append API—as the core product.

Built a production-style Write-Ahead Log (WAL) engine in Go with segmented append-only storage, CRC32C corruption detection, deterministic crash recovery, and checkpoint-assisted replay

Engineering decisions

  1. Designed a concurrent group-commit architecture using a bounded write queue and single writer goroutine with configurable sync, batch, and async durability modes

    amortizes fsync overhead across concurrent writers while preserving strict LSN ordering guarantees

    achieved ~9.8K writes/sec in batch mode with 100 concurrent writers

  2. Implemented crash-consistency and corruption handling features including replay modes, manifest reconciliation, atomic checkpoints, partial-write detection, and fault-injection testing

    ensures prefix-consistent recovery behavior under truncation, corruption, ENOSPC, and interrupted flush scenarios

    validated across 90+ tests including fuzzing, chaos injection, idempotency, and race-safe concurrency testing

  3. Built operational tooling including WAL inspection, integrity verification, corruption visualization, Prometheus metrics, benchmarking suites, and graceful shutdown handling

    provides production-style observability and debugging capabilities for storage-engine behavior under stress and failure

    exposed 13 Prometheus metrics and verified correctness under randomized chaos and recovery loops

Measured outcomes

  • sustained ~1M replayed records/sec with deterministic recovery guarantees
  • achieved ~9.8K writes/sec in batch mode with 100 concurrent writers
  • validated across 90+ tests including fuzzing, chaos injection, idempotency, and race-safe concurrency testing
  • reduced recovery time by ~2.5x using checkpoint-assisted replay
  • exposed 13 Prometheus metrics and verified correctness under randomized chaos and recovery loops