build-and-push.sh defaulted the image namespace to the deprecated ghcr.io/johannesjo/supersync — the source of the stale 7-day-token :latest image in #7865. Default to the super-productivity org instead, matching the CI workflow, docker-compose.yml and the Helm chart.
Since you cannot docker login as an org, split the image namespace (GHCR_NAMESPACE, defaulting to the org) from the GHCR login account (GHCR_USER), and fail fast if a push is attempted with no GHCR_USER set.
Refs #7865
Reconstructs a single user's AppDataComplete by replaying their op log up to a chosen serverSeq and decrypting E2E-encrypted payloads — the recovery path the server's restore endpoint cannot offer for encrypted accounts (it throws EncryptedOpsNotSupportedError). Read-only on the DB; documented in docs/backup-and-recovery.md. Unverified against real encrypted data — see the script header.
- Tighten recovery guard to the idempotent drop-then-create shape (require
both DROP and CREATE INDEX CONCURRENTLY). A bare CREATE (20260511000000) is
now refused by gate, deterministically — matching its documented fail-loud
intent instead of relying on the CREATE erroring by accident. Reconciles
README with the committed migration + migration-sql.spec.ts.
- Escape single quotes in print_manual_recovery so the printed escape-hatch
commands are copy-pasteable for SQL containing quotes (e.g. the full-state
WHERE op_type IN ('SYNC_IMPORT', ...) migration).
- Fix MIGRATE_LOG temp-file leak across retry attempts.
- Add an in-script per-step timeout (with_timeout) so the Dockerfile CMD /
helm initContainer paths (no outer timeout) can't hang forever on a blocked
concurrent build; 124 fails loudly.
- Harden parse_failing_migration: sentence-anchored P3009 parse + reject
names outside the migration charset (path-traversal defence).
- Gate also accepts SQLSTATE 25001 (stable Postgres contract) not just the
localizable English message.
- MAX_ATTEMPTS -> documented tight bound (6); fail_loudly wording fixed
(was contradictory for the non-CONCURRENTLY case).
- Tests: bare-CREATE refusal, P3009 decoy-token parser hardening; trim
migration-sql.spec.ts to the architectural-invariant subset (no hardcoded
names) per 'test behavior not implementation'.
Full server suite: 36 files, 727 passed / 5 skipped.
prisma migrate deploy wraps each migration in a transaction; CONCURRENTLY
index migrations fail it (P3018/25001) and later deploys then stick (P3009).
Recovery was duplicated and migration-name-hardcoded in the host deploy.sh
and the in-image migrate-deploy.sh. The host script self-updates only via a
best-effort git pull, so a stale host deploy.sh had no recovery branch for a
new CONCURRENTLY migration and failed the deploy (the reported incident).
- migrate-deploy.sh: single, name-agnostic recovery. Parses the failing
migration from Prisma's own output, gates on the txn-block/P3009 signature
AND the migration's own SQL containing INDEX CONCURRENTLY, runs that SQL
out-of-band statement-by-statement, and only marks it applied if every
statement succeeded; otherwise fails loudly with manual steps. Bounded
retry loop; aborts instead of looping on re-failure.
- deploy.sh: ~290 lines of hardcoded host-side recovery removed; now invokes
the in-image scripts/migrate-deploy.sh (always version-locked to
prisma/migrations in the pulled image) and keeps only timeout/exit policy.
- tests: drive the script end-to-end via a fake npx (P3018, stuck P3009,
non-CONCURRENTLY refusal, statement-failure, re-failure abort, genuine
error passthrough, multi-migration chain); migration-sql.spec.ts updated
to the new contract.
- prisma/migrations/README.md: authoring rules the recovery relies on.
Design: docs/plans/2026-05-15-generic-concurrently-migration-recovery-design.md
* docs(sync): add super sync server perf plan
* perf(sync): implement supersync server perf phases
* fix(sync): bracket auth cache invalidation
* fix(sync): avoid empty replay state stringify
* fix(sync): harden supersync batch uploads
* fix super sync review findings
* fix(sync): guard payload bytes backfill rollout
* perf(sync): speed up payload_bytes backfill and index its scan
Raise the backfill batch size (DEFAULT 5->500, MAX 25->1000) so a
100M-row operations table backfills in minutes rather than tens of
hours. Add a CONCURRENTLY partial index on (user_id, id) WHERE
payload_bytes = 0: it drains to empty post-backfill so the boot-time
backfill self-check and the BOOL_OR quota probe stop doing a full
sequential scan to prove absence, and it makes the backfill's per-user
keyset paging a true index seek. Wire the new concurrent-index
migration into both deploy scripts' P3018 recovery path. Add
migration-SQL guard tests for the ADD COLUMN (metadata-only fast path)
and the new partial index.
* fix(sync): bound auth cache invalidation map and bracket every delete
The auth verification cache's invalidationVersions map grew one entry
per lifetime-invalidated user with no eviction (unbounded heap on a
long-lived single replica). Cap it at the same 10k LRU bound as the
entries map, re-inserting the just-invalidated user at the MRU tail so
the CAS race protection still holds for the only window that matters
(one DB round trip). Bracket the passkey/magic-link registration
cleanup deletes with pre+post invalidate to match the documented
convention, and invalidate on verifyEmail so a freshly-verified user
isn't denied for up to the cache TTL.
* perf(sync): skip the redundant exact replay-state measurement
The delta accounting is a proven over-estimate of the serialized state
size, so when the running bound stays within the cap the true size is
too and the final exact JSON.stringify is provably redundant. Skip it
in that case (still measure-and-throw whenever the bound does not prove
safety). This collapses the common small/incremental replay back to
zero expensive full stringifications, matching the old per-op loop
instead of regressing it. Name the entity-key JSON overhead constant
and document that assertReplayStateSize's return value is load-bearing.
* refactor(sync): split processOperationBatch into pipeline stages
Extract the 297-line batch upload method into a thin orchestrator plus
six named single-responsibility stage helpers (validate+clamp, intra-
batch dedupe, classify existing duplicates, conflict-detect, reserve
seq + insert, full-state clock). Behavior-preserving: every stage
writes terminal rejections into the shared results array by index and
the two empty-set guards short-circuit exactly as before. Also share
the timestamp clamp, the duplicate-op SELECT, and the merged
full-state clock persistence between the batch and legacy paths so
they cannot silently diverge.
* test(sync): pin batch error-code divergence and aggregate-once
Strengthen the intra-batch duplicate test to assert same-id /
different-content yields DUPLICATE_OPERATION (deliberate divergence
from the legacy INVALID_OP_ID), and document the divergence in the
plan. Replace the single-full-state aggregate test with two
full-state ops + a spy asserting _aggregatePriorVectorClock runs
exactly once and last-write-wins — the old test could not catch a
per-op-aggregate regression. Add a makeOp fixture factory. Correct
the plan's overstated replay-stringification numbers.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: johannesjo <1456265+johannesjo@users.noreply.github.com>
- B11: Remove orphan deploy-db-scalar.mjs and dead RUN_POST_MIGRATION_INDEXES
references in env.example and README. Rewrite README deploy section to
match what scripts/deploy.sh actually does (drop false claims about
empty migration SQL, failed-migration recovery, and optional index builds).
- B12: Default RUN_MIGRATIONS_ON_STARTUP to false in docker-compose.yml so
the compose, Dockerfile ENV, and env.example agree on the safer default
(manual deploy.sh runs migrations once before app restart). Update the
migration-sql.spec assertion to match.
- I9: Wrap prisma migrate deploy in deploy.sh with a MIGRATION_TIMEOUT
(default 900s). CREATE INDEX CONCURRENTLY can block on long-running
transactions; on timeout (exit 124) the script now fails loudly with a
clear error rather than hanging.
- Use upsert for userSyncState to avoid RecordNotFound on first-time users
- Reset (not merge) state on full-state ops; throw on malformed payloads
- Add typed EncryptedOpsNotSupportedError; stop leaking op count to clients
- Split JSON.parse vs decompress error paths (new invalid-json reason)
- Measure base64 transport size against decoded binary; decode once
- Guard concurrent cacheSnapshot writes against stale overwrite
- Move encryption-skip policy from route into cacheSnapshotIfReplayable
- Invalidate corrupt cached snapshot blobs instead of re-logging forever
- Use Buffer.byteLength for replay state size (was UTF-16 vs byte limit)
- Skip cache-replayability assert on the cache-hit fast-path
- Extract async gunzip/gzip into gzip.ts; stop blocking the event loop
- Replace inline tx structural types with Prisma.TransactionClient
- deploy.sh: set -euo pipefail with guarded vars and pipeline-safe diagnostics
Avoids the operations table entirely (sync_devices + users only) so it
returns in milliseconds even when operations grows past 1 GB. Useful when
the standard active-users command is too slow on large prod DBs.
The showUsage query ran two correlated subqueries over the full
operations table once per user, each calling pg_column_size(payload)
on every row, then sorted all users before LIMIT 20. On a 1.85 GB
operations table with 3.6k users this effectively never returned.
Aggregate per-user size and count in a single CTE pass, then join.
One sequential scan instead of scans × users.
When a full-state op (SYNC_IMPORT/BACKUP_IMPORT) arrived from another
client, mergeRemoteOpClocks reset the local clock to a "minimal" form
but only preserved the current client's counter from the incoming op's
clock. If the current client had issued ops (e.g. GLOBAL_CONFIG) not
reflected in the incoming full-state op's clock, its counter was dropped,
causing subsequent ops to reuse the same counter value. Downstream clients
then saw these ops as EQUAL (duplicate) and skipped them silently.
Fix: take max(mergedClock[clientId], currentClock[clientId]) when
rebuilding the clock after a full-state op reset.
Also add __SP_E2E_BLOCK_WS_DOWNLOAD flag to WsTriggeredDownloadService
to allow E2E tests to block automatic WS-triggered downloads during
concurrent conflict scenarios.
Fix archive conflict test by blocking WS downloads on Client A during
the concurrent edit phase so it doesn't auto-receive B's rename via
WebSocket before archiving (restoring the intended conflict scenario).
Fix LWW singleton test to assert convergence rather than specific winner.
Fix renameTask helper to avoid Playwright/Angular re-render races.
Fix shepherd.js import paths that broke the Angular dev server build.
The production container runs as non-root user `supersync` without tsx
(dev dependency excluded via --omit=dev). Analysis and monitor-all
commands failed because they tried to use tsx/npx tsx which couldn't
write to node_modules. All scripts are already compiled to dist/ during
the Docker build, so use those directly.
- Pin Caddy image to 2.11-alpine to prevent breaking changes from
floating tags
- Validate Caddyfile syntax before deploying to catch config errors
early
- Check all container states after startup to detect crashes before
waiting on the HTTPS health check
- Show logs from all services on failure, not just supersync
Prisma's $queryRaw tagged template treats ${} as parameterized values, not
raw SQL. Conditional fragments like `${userId ? \`WHERE ...\` : ''}` were
sent as string parameters, causing PostgreSQL syntax error 42601.
Mask user emails by default (--unmask to reveal). Replace all any[]
query results with typed interfaces. Fix showUsageHistory sort that
compared formatted byte strings. Add parseIntArg validation for CLI
flags. Fix unbounded recursion in analyzePayload. Replace execSync with
execFileSync for DATA_DIR to prevent shell injection. Add --no-save flag
for usage command.
* feat(sync): add active-users command to monitor CLI
Adds a new `active-users` command to the SuperSync monitor script that reports:
- Total registered and verified user counts
- Active users by time period (24h, 7d, 30d, 90d) based on device and sync activity
- New registration counts
- Recently active users table with device count and ops
- Users who never synced
Usage: npm run monitor -- active-users
https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP
* feat(sync): add engaged users metric to active-users report
Shows users who were active on 3+ distinct days in the last 2 weeks
with new sync operations, giving a measure of genuine recurring usage.
https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP
* docs(sync): add active-users to docker monitoring docs
Add missing active-users command to docker-monitor.sh case statement,
help text, and DOCKER-MONITORING.md guide.
https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP
* refactor(sync): improve active-users command from review feedback
- Fix timezone-unsafe DATE(): use AT TIME ZONE 'UTC' with explicit
double precision cast for consistent day boundaries
- Replace correlated subquery with LEFT JOIN for ops_7d count
- Show total active count when LIMIT truncates the table
- Add --threshold and --limit CLI flags for flexibility
- Combine device/ops metrics into single line (connected / syncing)
- Add skipInQuick to run-all-monitoring.ts
- Update docker-monitor.sh header, help text, and DOCKER-MONITORING.md
with active-users command, flags, and performance notes
- Clarify "never synced" output label
https://claude.ai/code/session_014Tc5vtXW4Z8QZFMDFWKErP
---------
Co-authored-by: Claude <noreply@anthropic.com>
Translate all missing keys to achieve full coverage (2086/2086)
for fr, pt-br, zh-tw, ro, ro-md, zh, es, de, and cs.
Also remove stale keys and include updated translations
for other locales. Remaining languages still have partial coverage.
Adds git pull --ff-only before docker compose pull so scripts,
docker-compose.yml, and other repo files are updated on each deploy.
Falls back gracefully if pull fails.
- Remove email addresses from all log messages (~18 locations) for GDPR
compliance, replacing with userId where available
- Downgrade debug credential logs from info to debug level
- Remove options.user from passkey registration log (contained email in
JSON payload)
- Add container security hardening to docker-compose.yml:
- no-new-privileges on all containers
- cap_drop: ALL on supersync and caddy
- Memory limits (supersync 512m, postgres 1g, caddy 256m)
- CPU limit on supersync (1.0)
- Add .health-alert/ to gitignore
- Add health-alert.sh cron script for container monitoring with
OOM detection, disk checks, and email alerting
- Add accounts-only pg_dump (users + passkeys) to backup script for
lightweight disaster recovery when clients still have data
- Add pipefail to backup script to catch silent dump failures
- Add test endpoint to simulate partial server revert (ops-after/:serverSeq)
- Add 6 e2e tests covering all disaster recovery scenarios:
complete data loss, partial revert, accounts-only restore (API + SQL),
full dump restore + reset account, and all-clients-lost recovery
- Add backup-and-recovery.md with setup, recovery procedures, and
decision tree
- Add automated Docker build/push workflow for SuperSync server
- Update docker-compose.yml to use super-productivity org GHCR image
- Add --no-cache flag support to build-and-push.sh script
- Workflow triggers on SuperSync changes or manual dispatch
- Uses GitHub's built-in token (no personal token needed)
Add reusable investigation scripts for analyzing SuperSync storage patterns,
user behavior, and operation anomalies.
New scripts:
- analyze-storage.ts: 9 specialized analysis commands (operation sizes,
timeline, types, large ops, rapid-fire detection, snapshots, user deep-dive,
export, compare users)
- run-all-monitoring.ts: Complete monitoring suite runner with quick mode
and save-to-file options
- MONITORING-README.md: Complete documentation with investigation workflows
New npm scripts:
- npm run analyze-storage -- <command>: Run specific analysis
- npm run monitor:all: Run complete monitoring suite
- npm run monitor:all:quick: Quick health check (skip deep analysis)
- npm run monitor:all:save: Save timestamped report to file
These tools provide structured workflows for investigating storage issues
like rapid-fire operations, unusually large operations, and sync loops.
Tombstones were used for tracking deleted entities but are no longer
needed with the operation log architecture. This removes:
- Tombstone table from Prisma schema
- Tombstone-related methods from SyncService
- Tombstone mocks and tests from all test files
- Database migration to drop tombstones table
The operation log now handles deletions through DEL operations,
making the separate tombstone tracking redundant.
- Add 'ops' command to analyze operation payload sizes
- Show breakdown by entity type with count, total, avg, max sizes
- Display largest operation with payload structure analysis
- Enhance 'usage' command to show ops vs snapshot breakdown
- Show operation count, average size per user
Each run of `npm run monitor -- usage` now saves a snapshot to
logs/usage-history.jsonl. New command `usage-history` shows past
snapshots with growth stats between runs.
- Add disk space info to monitor stats (root filesystem, data dir)
- Add top 5 tables by size to database stats
- Add GHCR login to deploy.sh for private image pulls
Switch from server-side builds to pre-built images:
- Add build-and-push.sh for local builds to GHCR
- Update deploy.sh to pull from registry (30s vs 20min)
- Add docker-compose.build.yml for local build fallback
- Update docker-compose.yml to use registry image
- Add GHCR_USER/GHCR_TOKEN to env.example
- Add npm scripts: docker:build, docker:deploy, docker:backup
This commit introduces a lightweight monitoring CLI tool for the super-sync-server.
Changes include:
- Enhanced logging: now supports writing logs to a file () when is set, enabling persistent log storage.
- New monitoring script: provides commands for:
- : Displays system vitals (CPU, RAM) and database connectivity/size.
- : Shows top 20 users by data storage usage within the database.
- : Allows tailing, searching, and filtering server logs from the file.
- update: Added a script to easily run the tool.
Refactored 'delete-user.ts' and 'clear-data.ts' to use Prisma Client, aligning them with the current PostgreSQL architecture.
Updated 'packages/super-sync-server/README.md' to reflect the Docker-based, PostgreSQL setup and added documentation for the administrative scripts.
Removed '@types/better-sqlite3' from 'package.json' as it's no longer needed.
SYNC_IMPORT, BACKUP_IMPORT, and Repair operations contain full app state
in their payload. When these operations were downloaded by another client
and applied via applyOperations(), convertOpToAction() was spreading the
payload directly into the action. But reducers expect action.appDataComplete,
so they received undefined and fell back to empty/initial state.
This caused complete data loss (no projects, no tags, tasks in wrong project)
when syncing SYNC_IMPORT operations between clients.
Fix: Add extractFullStatePayload() that wraps the payload in appDataComplete
for full-state operation types, matching what loadAllData action expects.
Also clears file-based storage directories in the clear-data script.
Previously, would silently create a new empty database if the configured was incorrect or missing, leading to a confusing 'success' message (deleting 0 rows) while the actual data remained untouched. Now, the script verifies that exists at the expected path before proceeding, and exits with an error if it's missing.
