Prisma's default connection pool is ~(host_cpu_count * 2), read from host
cores not the container cpu limit. On the production VPS that silently
consumed the entire postgres max_connections=40 cap, leaving zero
headroom for psql, migrations, the old-ops cleanup job, or the reconnect
stampede after an in-place crash recovery -> sustained 'sorry, too many
clients already', WS reconnect storms and 429s.
Raise max_connections to 120 (120 x 4MB work_mem + 384MB shared_buffers
stays well inside the 1.5g mem_limit). The app must also bound its pool
via DATABASE_URL ?connection_limit=60 (host .env, not in repo).
busybox wget in caddy:2.11-alpine resolves `localhost` to ::1 first, but Caddy binds its admin API to IPv4 127.0.0.1:2019. The healthcheck got "connection refused" and marked a healthy Caddy (serving prod traffic with valid TLS) as unhealthy, making deploy.sh report a false "Container startup failed\!".
Right-size container limits to fit a 4GB / 2-vCore VPS host without
OOM risk, and apply minimal best-practice hardening informed by
multi-agent review.
Container memory budget (sum 2816 MB, leaves ~1.2 GB host headroom):
- postgres: 1g → 1.5g (was at 74% under 199 WSS connections)
- supersync: 1g → 768m (uses 143 MiB steady state)
- dozzle: add 64m cap (was unbounded)
- uptime-kuma: add 192m cap (was unbounded)
- caddy: unchanged at 256m
Postgres tuning (1.5g cap, shm_size 256m to allow shared_buffers > 64m):
- shared_buffers=384MB, effective_cache_size=1GB
- work_mem=4MB, maintenance_work_mem=128MB
- max_connections=40 (Prisma pool is single-digit; 40 leaves headroom
for migrations + psql; 100 was an OOM landmine vs 1.5g cap)
- random_page_cost=1.1 (SSD)
- idle_in_transaction_session_timeout=300000 (5 min — catches leaked
Prisma idle-in-tx but stays well above migrate-deploy.sh
STEP_TIMEOUT=1800 for CONCURRENTLY index recovery)
- wal_compression=on, huge_pages=off
Other hardening:
- Standardize restart policy to unless-stopped across all services
- Add caddy healthcheck via admin API at localhost:2019; Caddyfile
comment warns future edits not to disable admin
- Add ulimits.nofile=65535 to supersync (Alpine soft default 1024 is
tight for 200+ WSS + reconnect bursts)
- Pin networks.internal.name=super-sync-server_internal so the
monitoring overlay can attach via external: true regardless of
COMPOSE_PROJECT_NAME
- Attach dozzle + uptime-kuma to internal network so Kuma can probe
http://supersync:1900/health by service name
Notes for future maintenance:
- oom_score_adj was tried and removed: Compose v2 silently ignores it
(verified on host via /proc/<pid>/oom_score_adj returning 0).
Would need a systemd post-up hack to apply — not worth it.
- The reconnect storm that motivated this work is fixed separately
on master in b404bf8a3.
* docs(sync): add super sync server perf plan
* perf(sync): implement supersync server perf phases
* fix(sync): bracket auth cache invalidation
* fix(sync): avoid empty replay state stringify
* fix(sync): harden supersync batch uploads
* fix super sync review findings
* fix(sync): guard payload bytes backfill rollout
* perf(sync): speed up payload_bytes backfill and index its scan
Raise the backfill batch size (DEFAULT 5->500, MAX 25->1000) so a
100M-row operations table backfills in minutes rather than tens of
hours. Add a CONCURRENTLY partial index on (user_id, id) WHERE
payload_bytes = 0: it drains to empty post-backfill so the boot-time
backfill self-check and the BOOL_OR quota probe stop doing a full
sequential scan to prove absence, and it makes the backfill's per-user
keyset paging a true index seek. Wire the new concurrent-index
migration into both deploy scripts' P3018 recovery path. Add
migration-SQL guard tests for the ADD COLUMN (metadata-only fast path)
and the new partial index.
* fix(sync): bound auth cache invalidation map and bracket every delete
The auth verification cache's invalidationVersions map grew one entry
per lifetime-invalidated user with no eviction (unbounded heap on a
long-lived single replica). Cap it at the same 10k LRU bound as the
entries map, re-inserting the just-invalidated user at the MRU tail so
the CAS race protection still holds for the only window that matters
(one DB round trip). Bracket the passkey/magic-link registration
cleanup deletes with pre+post invalidate to match the documented
convention, and invalidate on verifyEmail so a freshly-verified user
isn't denied for up to the cache TTL.
* perf(sync): skip the redundant exact replay-state measurement
The delta accounting is a proven over-estimate of the serialized state
size, so when the running bound stays within the cap the true size is
too and the final exact JSON.stringify is provably redundant. Skip it
in that case (still measure-and-throw whenever the bound does not prove
safety). This collapses the common small/incremental replay back to
zero expensive full stringifications, matching the old per-op loop
instead of regressing it. Name the entity-key JSON overhead constant
and document that assertReplayStateSize's return value is load-bearing.
* refactor(sync): split processOperationBatch into pipeline stages
Extract the 297-line batch upload method into a thin orchestrator plus
six named single-responsibility stage helpers (validate+clamp, intra-
batch dedupe, classify existing duplicates, conflict-detect, reserve
seq + insert, full-state clock). Behavior-preserving: every stage
writes terminal rejections into the shared results array by index and
the two empty-set guards short-circuit exactly as before. Also share
the timestamp clamp, the duplicate-op SELECT, and the merged
full-state clock persistence between the batch and legacy paths so
they cannot silently diverge.
* test(sync): pin batch error-code divergence and aggregate-once
Strengthen the intra-batch duplicate test to assert same-id /
different-content yields DUPLICATE_OPERATION (deliberate divergence
from the legacy INVALID_OP_ID), and document the divergence in the
plan. Replace the single-full-state aggregate test with two
full-state ops + a spy asserting _aggregatePriorVectorClock runs
exactly once and last-write-wins — the old test could not catch a
per-op-aggregate regression. Add a makeOp fixture factory. Correct
the plan's overstated replay-stringification numbers.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: johannesjo <1456265+johannesjo@users.noreply.github.com>
The throttle in deleteOldSyncedOpsForAllUsers now caps each cleanup pass to
short, bounded batches, so the original "shield warmup from DB-heavy retention
work" rationale no longer applies. A long initial delay only creates a
starvation risk on frequently-restarted deployments (rolling deploys or crash
loops could land inside the 30-min window every time and skip cleanup
entirely).
Revert to a fixed 10s warmup and remove the now-pointless knob from envs, the
compose file, and the Helm chart.
Daily retention cleanup previously issued one unbounded `deleteMany` per user
that covered every operation older than the retention cutoff. On large
backlogs this monopolized Postgres and made user-facing sync slow during the
window, which matched the production "sync is still slow" reports.
- Cap per-batch deletes (default 5k) and per-run total deletes (default 25k),
with a select-then-delete-by-id pattern that keeps each statement short.
- Drain each user across batches before moving on so a fresh user with a tiny
backlog isn't blocked behind a large one until next pass.
- Order candidates by `snapshotAt asc` so the stalest users win the budget
when it's tight.
- Defer the initial cleanup pass after startup to 30 min in production (was
10s) so retention work doesn't compete with deploy/restart warmup; call
`unref()` on cleanup timers so they never hold the process open.
- Expose all knobs via env vars wired through docker-compose, env examples,
and the Helm chart, with a strict positive-integer parser and clamped
maxima so a misconfiguration cannot unwind the bound.
- B11: Remove orphan deploy-db-scalar.mjs and dead RUN_POST_MIGRATION_INDEXES
references in env.example and README. Rewrite README deploy section to
match what scripts/deploy.sh actually does (drop false claims about
empty migration SQL, failed-migration recovery, and optional index builds).
- B12: Default RUN_MIGRATIONS_ON_STARTUP to false in docker-compose.yml so
the compose, Dockerfile ENV, and env.example agree on the safer default
(manual deploy.sh runs migrations once before app restart). Update the
migration-sql.spec assertion to match.
- I9: Wrap prisma migrate deploy in deploy.sh with a MIGRATION_TIMEOUT
(default 900s). CREATE INDEX CONCURRENTLY can block on long-running
transactions; on timeout (exit 124) the script now fails loudly with a
clear error rather than hanging.
Previously, `npx prisma@5.22.0 migrate deploy` in CMD downloaded prisma
from npm on every container start, making startup slow and dependent on
network availability. Now prisma is installed in the image at build time.
Also increases health check start_period from 10s to 30s to give
migrations time to complete before Docker starts counting failures.
- Pin Caddy image to 2.11-alpine to prevent breaking changes from
floating tags
- Validate Caddyfile syntax before deploying to catch config errors
early
- Check all container states after startup to detect crashes before
waiting on the HTTPS health check
- Show logs from all services on failure, not just supersync
Caddy Alpine images set file capabilities on the binary via setcap.
The no-new-privileges security option prevents the kernel from honoring
file capabilities during exec, causing "operation not permitted" on
startup. Container remains hardened with cap_drop: ALL + cap_add:
NET_BIND_SERVICE.
Newer Caddy Alpine images set file capabilities on the binary, which
requires NET_BIND_SERVICE to bind to ports 80/443. Without it,
cap_drop: ALL prevents execution entirely.
- Remove email addresses from all log messages (~18 locations) for GDPR
compliance, replacing with userId where available
- Downgrade debug credential logs from info to debug level
- Remove options.user from passkey registration log (contained email in
JSON payload)
- Add container security hardening to docker-compose.yml:
- no-new-privileges on all containers
- cap_drop: ALL on supersync and caddy
- Memory limits (supersync 512m, postgres 1g, caddy 256m)
- CPU limit on supersync (1.0)
- Add .health-alert/ to gitignore
- Add health-alert.sh cron script for container monitoring with
OOM detection, disk checks, and email alerting
- Add automated Docker build/push workflow for SuperSync server
- Update docker-compose.yml to use super-productivity org GHCR image
- Add --no-cache flag support to build-and-push.sh script
- Workflow triggers on SuperSync changes or manual dispatch
- Uses GitHub's built-in token (no personal token needed)
Switch from server-side builds to pre-built images:
- Add build-and-push.sh for local builds to GHCR
- Update deploy.sh to pull from registry (30s vs 20min)
- Add docker-compose.build.yml for local build fallback
- Update docker-compose.yml to use registry image
- Add GHCR_USER/GHCR_TOKEN to env.example
- Add npm scripts: docker:build, docker:deploy, docker:backup
- Add server endpoints for restore points (GET /api/sync/restore-points, GET /api/sync/restore/:serverSeq)
- Add RestorePoint types and RestoreCapable interface
- Add getRestorePoints() and generateSnapshotAtSeq() to SyncService
- Add SuperSyncRestoreService for client-side restore orchestration
- Add DialogRestorePointComponent for restore point selection UI
- Add "Restore from History" button to sync settings (visible when Super Sync is active)
- Fix circular dependency in SyncSafetyBackupService using lazy injection
- Update docker-compose.yml to support local development (configurable NODE_ENV, PUBLIC_URL)
