Skip to main content
InternalMarkdown

ADR 0005. Phased Event-Log Infrastructure

Status​

Accepted.

Phase 2 is triggered by spec §35.4 measurements, not a calendar date: sustained inbox write/worker saturation, independent consumers that need replay, lag that must be isolated, material ordering/partitioning, backfills competing with live ingest, or operational benefit that exceeds running another stateful platform. Dagster is the initial batch/backfill scheduler; Airflow is not the default (spec §35.1).

Context​

PlayerPal must be cheap to run at low volume and scale to high volume without a redesign. Committing to Kafka/Redpanda on day one adds operational overhead (cluster management, JVM footprint or equivalent, monitoring, minimum viable cluster size) that isn't justified at low traffic. By contrast, an ordinary application log file lacks replication, consumer offsets, replay tooling, and backpressure handling. It isn't a substitute for a real durable log at higher volume.

Decision​

Two-phase approach:

Phase 1 (lean MVP): API -> Postgres append table -> workers, or for batch-heavy activity, API -> object storage append/batches -> workers. Appropriate while volume is low/moderate and operational simplicity matters most.

Phase 2 (real event log): API -> Redpanda/Kafka-compatible log -> reward worker / analytics writer / archival. Move here when traffic growth is sustained, more independent consumers appear, replay becomes important, or Postgres ingestion becomes a bottleneck.

Redpanda is the leading Kafka-compatible candidate (C++, no JVM, no ZooKeeper, generally lower operational overhead) but its licensing for managed/enterprise tiers is not confirmed free and must be checked against current terms before committing. Self-hosting vs. managed is a real cost decision, not just a technical one.

Consequences​

  • No premature complexity at low volume.
  • Phase 2 starts when a §35.4 condition is observed in production metrics, not when a date or guessed events/sec target arrives. Record the measured evidence in a new ADR before introducing Redpanda. Confirm Redpanda licensing at that time; it is still unchecked.