ADR 0006. Large Batches via Object Storage + Manifest, Not One Giant HTTP Payload

Status

Accepted.

Inline batch endpoint in the current spec is POST /v1/activity-batches (spec §37.6), not POST /v1/event-batches. Formats (NDJSON, NDJSON.gz, Parquet, CSV via signed upload + manifest) still stand. There is no api/openapi.yaml in the repository yet; cite spec §37.

Context

Not every partner supports real-time integration; many will export periodically (every few minutes to end-of-day). Sending millions of records as one HTTP JSON payload is expensive and fragile (memory pressure, timeout risk, poor retry semantics).

Decision

Small/medium batches may be submitted inline via POST /v1/event-batches. For very large exports: request a signed upload URL, upload compressed NDJSON/Parquet/CSV directly to S3-compatible object storage, then submit a manifest referencing that upload for asynchronous processing.

Consequences

• Batch and real-time paths still converge on the same normalized event model post-ingestion (see ADR 0004). This only changes how bytes arrive, not how they're processed afterward.
• Requires manifest/checkpoint design and idempotent batch processing (same (source_party_id, source_event_id) uniqueness as real-time).
• Exact accepted formats are fixed (NDJSON, NDJSON.gz, Parquet, CSV). See api/openapi.yaml.