Why does the nightly export write duplicate rows?

The duplicates come from the retry path, not the writer. When a nightly batch times out, scheduler.ts re-enqueues the whole batch — and because export jobs run with at-least-once delivery, both attempts can land. The dedupe step only checks keys within a single run, so a retried batch from the previous window slips through untouched.

In short: the pipeline is working as designed, and the design has a gap. The trail below shows why it was built this way — and one open question sits exactly on the code path you’d change.