A timed-out `docker compose run` in deploy.sh SIGTERMs the compose CLI but
can leave the migrator container (and its DB connection) running detached,
which keeps Prisma's session-level migration advisory lock held and wedges
the next deploy with P1002.
- deploy.sh: run_migrator() names each one-off migrator container from a
single MIGRATOR_NAME_PREFIX and force-removes it (inline + $$-scoped EXIT
sweep) so a timed-out run can't orphan it and leak the lock; add timeout -k.
- migrate-deploy.sh: recognize P1002 advisory-lock timeouts and print
copy-paste diagnosis + cleanup guidance instead of a generic exit 1; never
auto-terminate a backend (an active CONCURRENTLY build legitimately holds it).
- tests: P1002 fake-npx case + non-vacuous recovery test; assert the
name/sweep-filter prefix single-source-of-truth invariant.
- env.example: document the recommended higher MIGRATION_TIMEOUT and the new
force-remove-on-timeout behavior.
A bare CREATE INDEX CONCURRENTLY aborted mid-build (the in-image step
timeout firing at its 1800s default when a raised MIGRATION_TIMEOUT was
not forwarded, an external stop, or OOM) leaves the migration failed and
an INVALID index of the target name, wedging the deploy. Deployed images
reported only a bare "prisma migrate deploy failed (exit 143)" with no
recovery steps, so operators had no way forward.
- deploy.sh: forward MIGRATE_STEP_TIMEOUT into the migrator so a large
MIGRATION_TIMEOUT is not silently capped at the image default (1800s)
and kill a slow CREATE INDEX CONCURRENTLY early (the root cause).
- migrate-deploy.sh: normalize BusyBox `timeout` 143 -> 124 so a step
timeout hits the timeout branch, not the generic one.
- migrate-deploy.sh: emit_interrupted_recovery_hint() prints copy-paste
recovery (drop the INVALID index, roll the record back, re-run) for an
interrupted CONCURRENTLY build, from BOTH the 124 timeout branch (where
a normalized 143 -- the incident's own signal -- lands) and the generic
non-gate branch (OOM/137), plus the existing P3009 re-run path. Every
path prints guidance only and NEVER auto-resolves a bare CREATE.
- env.example: document the now-forwarded MIGRATE_STEP_TIMEOUT knob.
- tests: cover 143 normalization, the incident's 143->124 timeout-branch
recovery, the P3009 and non-gate bare-create recovery, and the
forwarded step timeout.
The bare CREATE stays bare and fail-loud on purpose: 20260613000001
ships in v18.11.0-v18.14.0 and is applied natively on healthy deploys,
so converting it to a drop-then-create shape would change its checksum
and break every DB that already applied it.
Recovery for the current wedged deploy (20260613000001):
DROP INDEX CONCURRENTLY IF EXISTS "operations_entity_ids_gin";
prisma migrate resolve --rolled-back \
20260613000001_add_operation_entity_ids_gin_index
then re-run the deploy with the step timeout raised enough for the GIN
build (and clear any idle-in-transaction blocker first).
Prisma's default pool is host-core-scaled and silently consumes the
entire postgres max_connections cap. Document the required
?connection_limit=N&pool_timeout=20 in env.example so operators and
deploy.sh keep the bound (host .env is not in the repo).
The throttle in deleteOldSyncedOpsForAllUsers now caps each cleanup pass to
short, bounded batches, so the original "shield warmup from DB-heavy retention
work" rationale no longer applies. A long initial delay only creates a
starvation risk on frequently-restarted deployments (rolling deploys or crash
loops could land inside the 30-min window every time and skip cleanup
entirely).
Revert to a fixed 10s warmup and remove the now-pointless knob from envs, the
compose file, and the Helm chart.
Daily retention cleanup previously issued one unbounded `deleteMany` per user
that covered every operation older than the retention cutoff. On large
backlogs this monopolized Postgres and made user-facing sync slow during the
window, which matched the production "sync is still slow" reports.
- Cap per-batch deletes (default 5k) and per-run total deletes (default 25k),
with a select-then-delete-by-id pattern that keeps each statement short.
- Drain each user across batches before moving on so a fresh user with a tiny
backlog isn't blocked behind a large one until next pass.
- Order candidates by `snapshotAt asc` so the stalest users win the budget
when it's tight.
- Defer the initial cleanup pass after startup to 30 min in production (was
10s) so retention work doesn't compete with deploy/restart warmup; call
`unref()` on cleanup timers so they never hold the process open.
- Expose all knobs via env vars wired through docker-compose, env examples,
and the Helm chart, with a strict positive-integer parser and clamped
maxima so a misconfiguration cannot unwind the bound.
- B11: Remove orphan deploy-db-scalar.mjs and dead RUN_POST_MIGRATION_INDEXES
references in env.example and README. Rewrite README deploy section to
match what scripts/deploy.sh actually does (drop false claims about
empty migration SQL, failed-migration recovery, and optional index builds).
- B12: Default RUN_MIGRATIONS_ON_STARTUP to false in docker-compose.yml so
the compose, Dockerfile ENV, and env.example agree on the safer default
(manual deploy.sh runs migrations once before app restart). Update the
migration-sql.spec assertion to match.
- I9: Wrap prisma migrate deploy in deploy.sh with a MIGRATION_TIMEOUT
(default 900s). CREATE INDEX CONCURRENTLY can block on long-running
transactions; on timeout (exit 124) the script now fails loudly with a
clear error rather than hanging.
Switch from server-side builds to pre-built images:
- Add build-and-push.sh for local builds to GHCR
- Update deploy.sh to pull from registry (30s vs 20min)
- Add docker-compose.build.yml for local build fallback
- Update docker-compose.yml to use registry image
- Add GHCR_USER/GHCR_TOKEN to env.example
- Add npm scripts: docker:build, docker:deploy, docker:backup
