The Prisma migration chain began with an ALTER on the operations table, so
a fresh database could not be initialized from migrations alone (the first
migration failed with no table to ALTER). Separately, the login_token /
login_token_expires_at columns and index existed in schema.prisma and were
used at runtime by src/auth.ts (magic-link login) but were never created by
any migration, so a migrate-only database crashed with
"column users.login_token does not exist".
- Add a 0_init baseline migration that creates the base tables (the
pre-2025-12-12 shape), sorting before the first incremental migration so
the existing ALTER migrations apply cleanly on top.
- Add 20260601000000_add_login_token to create the missing magic-link
columns and index (IF NOT EXISTS, safe on installs that already have them
from a prior db push).
- Add migration_lock.toml (provider = postgresql), the standard companion to
a real migration baseline.
- Document baselining for existing databases: `migrate resolve --applied
0_init` for installs with migration history, and resolving the whole chain
for db-push installs with none (covers the Helm/Docker unattended paths).
- Add regression tests asserting the baseline exists and that every @map
column in schema.prisma stays backed by a migration (guards the drift class
that caused this bug, not just login_token).
Verified by replaying all migrations on a fresh Postgres: the chain applies
cleanly and the resulting schema matches schema.prisma exactly.
Closes#8187
* docs(sync): add super sync server perf plan
* perf(sync): implement supersync server perf phases
* fix(sync): bracket auth cache invalidation
* fix(sync): avoid empty replay state stringify
* fix(sync): harden supersync batch uploads
* fix super sync review findings
* fix(sync): guard payload bytes backfill rollout
* perf(sync): speed up payload_bytes backfill and index its scan
Raise the backfill batch size (DEFAULT 5->500, MAX 25->1000) so a
100M-row operations table backfills in minutes rather than tens of
hours. Add a CONCURRENTLY partial index on (user_id, id) WHERE
payload_bytes = 0: it drains to empty post-backfill so the boot-time
backfill self-check and the BOOL_OR quota probe stop doing a full
sequential scan to prove absence, and it makes the backfill's per-user
keyset paging a true index seek. Wire the new concurrent-index
migration into both deploy scripts' P3018 recovery path. Add
migration-SQL guard tests for the ADD COLUMN (metadata-only fast path)
and the new partial index.
* fix(sync): bound auth cache invalidation map and bracket every delete
The auth verification cache's invalidationVersions map grew one entry
per lifetime-invalidated user with no eviction (unbounded heap on a
long-lived single replica). Cap it at the same 10k LRU bound as the
entries map, re-inserting the just-invalidated user at the MRU tail so
the CAS race protection still holds for the only window that matters
(one DB round trip). Bracket the passkey/magic-link registration
cleanup deletes with pre+post invalidate to match the documented
convention, and invalidate on verifyEmail so a freshly-verified user
isn't denied for up to the cache TTL.
* perf(sync): skip the redundant exact replay-state measurement
The delta accounting is a proven over-estimate of the serialized state
size, so when the running bound stays within the cap the true size is
too and the final exact JSON.stringify is provably redundant. Skip it
in that case (still measure-and-throw whenever the bound does not prove
safety). This collapses the common small/incremental replay back to
zero expensive full stringifications, matching the old per-op loop
instead of regressing it. Name the entity-key JSON overhead constant
and document that assertReplayStateSize's return value is load-bearing.
* refactor(sync): split processOperationBatch into pipeline stages
Extract the 297-line batch upload method into a thin orchestrator plus
six named single-responsibility stage helpers (validate+clamp, intra-
batch dedupe, classify existing duplicates, conflict-detect, reserve
seq + insert, full-state clock). Behavior-preserving: every stage
writes terminal rejections into the shared results array by index and
the two empty-set guards short-circuit exactly as before. Also share
the timestamp clamp, the duplicate-op SELECT, and the merged
full-state clock persistence between the batch and legacy paths so
they cannot silently diverge.
* test(sync): pin batch error-code divergence and aggregate-once
Strengthen the intra-batch duplicate test to assert same-id /
different-content yields DUPLICATE_OPERATION (deliberate divergence
from the legacy INVALID_OP_ID), and document the divergence in the
plan. Replace the single-full-state aggregate test with two
full-state ops + a spy asserting _aggregatePriorVectorClock runs
exactly once and last-write-wins — the old test could not catch a
per-op-aggregate regression. Add a makeOp fixture factory. Correct
the plan's overstated replay-stringification numbers.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: johannesjo <1456265+johannesjo@users.noreply.github.com>
- B11: Remove orphan deploy-db-scalar.mjs and dead RUN_POST_MIGRATION_INDEXES
references in env.example and README. Rewrite README deploy section to
match what scripts/deploy.sh actually does (drop false claims about
empty migration SQL, failed-migration recovery, and optional index builds).
- B12: Default RUN_MIGRATIONS_ON_STARTUP to false in docker-compose.yml so
the compose, Dockerfile ENV, and env.example agree on the safer default
(manual deploy.sh runs migrations once before app restart). Update the
migration-sql.spec assertion to match.
- I9: Wrap prisma migrate deploy in deploy.sh with a MIGRATION_TIMEOUT
(default 900s). CREATE INDEX CONCURRENTLY can block on long-running
transactions; on timeout (exit 124) the script now fails loudly with a
clear error rather than hanging.
- Add accounts-only pg_dump (users + passkeys) to backup script for
lightweight disaster recovery when clients still have data
- Add pipefail to backup script to catch silent dump failures
- Add test endpoint to simulate partial server revert (ops-after/:serverSeq)
- Add 6 e2e tests covering all disaster recovery scenarios:
complete data loss, partial revert, accounts-only restore (API + SQL),
full dump restore + reset account, and all-clients-lost recovery
- Add backup-and-recovery.md with setup, recovery procedures, and
decision tree
Mark PUBLIC_URL as required in the README.
I tried to run this locally and I got an error when I didn't define this in the .env, but no error for not defining the SMTP server which definitely seems to be needed.
Explain design decisions and security trade-offs:
- Why stateless JWTs (not stored in DB)
- Why token versioning over blacklisting
- Why verification tokens are plain strings
- Why bcrypt with 12 rounds
Include security features table, token lifecycle diagram,
and configuration reference.
Move scattered architecture docs into centralized locations:
- Move operation-log docs from src/app/core/persistence/operation-log/docs/
to docs/op-log/
- Flatten docs/sync/sync/ nested structure to docs/sync/
- Move supersync-encryption-architecture.md from docs/ai/ to docs/sync/
- Copy pfapi sync README to docs/sync/pfapi-sync-overview.md
- Update all cross-references to use new paths
This improves discoverability and keeps architecture documentation
separate from source code.
- Update hybrid-manifest-architecture.md status to Implemented
- Add documentation index (README.md) for operation-log docs
- Add index for docs/ai/sync/ with document categorization
- Replace duplicate docs with redirect stubs to canonical locations
- Add effect rules, multi-entity rules, and config constants to operation-rules.md
- Update vector-clocks.md with operation log integration section
- Expand super-sync-server README with API details and security features
- Update pfapi sync README with operation log integration info
Refactored 'delete-user.ts' and 'clear-data.ts' to use Prisma Client, aligning them with the current PostgreSQL architecture.
Updated 'packages/super-sync-server/README.md' to reflect the Docker-based, PostgreSQL setup and added documentation for the administrative scripts.
Removed '@types/better-sqlite3' from 'package.json' as it's no longer needed.
- Rewrote README to clarify custom operation-log architecture (not WebDAV).
- Added Vitest and implemented comprehensive unit tests for SyncService.
- Implemented Incremental Snapshots to improve performance.
- Reduced stale device cleanup threshold to 30 days to reduce DB bloat.
- Increased max payload size to 20MB to support large initial imports.
- Added in-memory DB support for faster testing.
