Distributed systems
Distributed Job Queue
Priority scheduling, delayed retries, worker recovery, and end-to-end observability, tested under sustained concurrent load and failure scenarios.
- Stack
- Python · FastAPI · PostgreSQL · Redis · React · Docker · Kubernetes · OpenTelemetry · Grafana · k6
- Evidence
- ~295K requests · 0 API errors · ~153 req/s breakpoint
The challenge
Queue APIs often look correct until retries, worker crashes, and ingress beyond processing capacity expose ambiguous state. This system was built around those failure paths and measured under load rather than demonstrated only at happy-path scale.
Built a distributed job queue system with priority scheduling, retry-aware execution, and idempotent request handling
Engineering decisions
Designed a multi-queue Redis scheduling architecture using FIFO, priority, and delayed retry queues
supports deterministic job ordering, exponential backoff retries, and fair scheduling behavior
sustained ~153 req/s during breakpoint testing at 200 VUs
Implemented worker recovery, cancellation handling, execution timeouts, and per-job execution logging
improves fault tolerance and observability during crashes and long-running task execution
recovered stuck processing jobs automatically on worker startup
Added a full observability stack with OpenTelemetry instrumentation, Prometheus metrics, Grafana dashboards, Loki logs, and Tempo traces, shipped as a GHCR image via an init container
made queue depth, worker throughput, and per-job latency directly measurable rather than inferred
Built and deployed the system using Docker, GitHub Actions CI/CD, and an Oracle ARM64 VM behind Nginx and Cloudflare
enabled automated deployments and reproducible infrastructure setup
deployed API, worker, PostgreSQL, and Redis as containerized services
Measured outcomes
- validated across ~295k requests with 0 API-level errors
- sustained ~153 req/s during breakpoint testing at 200 VUs
- recovered stuck processing jobs automatically on worker startup
- queue depth grew to ~26k jobs under ~100 creates/sec ingress during worker stress testing
- deployed API, worker, PostgreSQL, and Redis as containerized services