super-productivity/docs/sync-and-op-log
Johannes Millan 63d46f7a0b
feat(op-log): validate SQLite backend + IDB→SQLite migration + backend-aware init (#7931) (#7954)
* test(op-log): validate SqliteOpLogAdapter against a real sql.js engine

The 23 adapter specs ran only against an in-memory regex stand-in that
models the SQL shapes the adapter emits — it validates the translation
layer, not SQLite itself. Add sql.js (dev-only; never in the app bundle)
served into Karma, and run the behavioral contract against BOTH the fake
and a real SQLite engine.

This exercises genuine-engine behavior the stand-in could only model:
the real UNIQUE-constraint message -> ConstraintError mapping,
AUTOINCREMENT never reusing seq after clear(), compound-index + NULL
range handling, and real BEGIN IMMEDIATE rollback. 51/51 green.

B2 (translation-layer pass) per docs/sync-and-op-log/sqlite-migration.md.
The integration-harness second pass and the on-device real-engine run
remain.

* test(op-log): run store-port integration against real sql.js (B2 stage 2)

Parameterize the RemoteOperationApplyStorePort integration scenarios to
run against BOTH the default IndexedDB backend and a sql.js-backed
SqliteOpLogAdapter, exercising the store's COMPOSED flows (apply/mark/
merge-clock, partial-failure persistence, full-state import clearing,
vector-clock persistence) on a real SQL engine — not just the adapter in
isolation. 6/6 green.

Surfaced a real B3 wiring gap: OperationLogStoreService.init() is
IDB-shaped (opens+adopts an IndexedDB connection, never calls the
adapter's own init()). For a self-managing backend like SQLite the
tables would not exist. The sql.js setup creates them once on the shared
db to mirror the store-init change B3 must make on native (call
adapter.init() / skip the IDB open when the backend is SQLite).

* feat(op-log): add verified IDB->SQLite backend migration (C1)

One-time copy of the entire op-log from a source adapter (legacy
IndexedDB) to a dest adapter (SQLite) in a single dest transaction with
verify-before-commit: a mismatch in op count, last seq, or vector clock
throws and rolls the dest back, leaving it empty and the source
untouched.

Adapter-agnostic (talks only to the OpLogDbAdapter port), so it is
validated in CI with a real Chrome IndexedDB source + a sql.js SQLite
dest; the native @capacitor-community/sqlite dest behaves identically
through the same port. The generic iterate->put copy preserves ops seq
(incl. gaps) via the put-honors-seq path and writes singletons at their
out-of-line key uniformly, with no per-store special-casing.

Not wired into startup — Phase B3/C2 decide WHEN to run it (SQLite empty
+ legacy SUP_OPS present) and retain the IDB copy >= 1 release. 5/5
green, incl. seq-fidelity, AUTOINCREMENT-continues-past-migrated,
empty-source, non-empty-dest guard, and verify-rollback.

* docs(sync): record sql.js validation, C1, and the B1/B3 findings

Update the SQLite migration plan + follow-up backlog to reflect what
landed this pass and hand off the device-gated remainder:
- B2: real-engine (sql.js) adapter contract + store-port second pass are
  done in CI; only the on-device run remains.
- C1: the backend-migration algorithm + verify-before-commit are done
  and tested (real IDB -> sql.js); only the startup wiring remains.
- B3 finding: OperationLogStoreService.init() is IDB-shaped (opens+adopts
  IDB, never calls adapter.init()); native must call adapter.init() and
  skip the IDB open on SQLite.
- B1 perf note: bridge round-trips dominate on native; return lastId from
  the plugin's run response and add a runBatch/executeSet bulk path so
  appendBatch is one crossing, with RETURNING-seq for per-op seq.

* feat(op-log): make store init backend-aware for self-managing backends (B3)

OperationLogStoreService.init() and ArchiveStoreService._init() were
IDB-shaped: they unconditionally opened+adopted a WebView IndexedDB
connection and never called the adapter's own init(). For a self-managing
backend (SQLite) that meant (a) the adapter's tables were never created
and (b) it still touched the evictable WebView store this migration
exists to escape.

Now: when the adapter exposes no adoptConnection (i.e. it self-manages,
like SQLite), call adapter.init() and skip the IndexedDB open. The
adopt-connection (IndexedDB) path is unchanged. The new branch is dead in
production until B3 flips the native token, so this is risk-free now and
unblocks that flip.

Tested: two unit tests cover both branches (self-managing -> adapter.init,
no IDB open; IDB -> open+adopt, no adapter.init). The store-port
integration spec now drives the store fully on SQLite with _db undefined,
so its earlier pre-init workaround is removed. 521 persistence + 6
integration green.

* docs(sync): mark the B3 backend-aware init fix as landed

The store-init half of B3 (call adapter.init() / skip the IDB open for
self-managing backends) is implemented + CI-tested; only the device-gated
native token flip + SqliteDb wrapper remain.
2026-06-02 17:26:23 +02:00
..
diagrams docs(sync): consolidate sync docs + enforce the contributor model 2026-05-15 16:51:50 +02:00
contributor-sync-model.md docs(sync): consolidate sync docs + enforce the contributor model 2026-05-15 16:51:50 +02:00
file-based-sync-flowchart.md docs(sync): add file-based sync flowchart for Dropbox/WebDAV/LocalFile 2026-03-01 21:27:08 +01:00
operation-log-architecture.md Improve on sync (#7736) 2026-05-22 17:49:25 +02:00
operation-rules.md docs(sync): document archive-wins rule, singleton LWW, and stale op handling 2026-01-30 16:59:40 +01:00
package-boundaries.md refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
README.md docs(sync): consolidate sync docs + enforce the contributor model 2026-05-15 16:51:50 +02:00
sqlite-migration-followup.md feat(op-log): validate SQLite backend + IDB→SQLite migration + backend-aware init (#7931) (#7954) 2026-06-02 17:26:23 +02:00
sqlite-migration.md feat(op-log): validate SQLite backend + IDB→SQLite migration + backend-aware init (#7931) (#7954) 2026-06-02 17:26:23 +02:00
supersync-encryption-architecture.md docs(sync): add encryption architecture and scenario documentation 2026-03-01 21:27:08 +01:00
supersync-scenarios-flowchart.md docs(sync): align SYNC_IMPORT scenarios with current gate semantics 2026-04-29 16:17:56 +02:00
supersync-scenarios.md feat(supersync): retry transient web fetch failures and surface them as warnings 2026-05-13 11:20:02 +02:00
vector-clocks.md docs(sync): consolidate sync docs + enforce the contributor model 2026-05-15 16:51:50 +02:00

Operation Log & Sync Documentation

The Operation Log is the single sync system for all providers (SuperSync, WebDAV, Dropbox, LocalFile). It is an event-sourced persistence + sync layer: the log is the source of truth, current state is derived by replaying it, and vector clocks detect concurrent edits.

                         User Action
                              │
                              ▼
                         NgRx Store  (runtime source of truth)
                              │
          ┌───────────────────┼───────────────────┐
          ▼                   │                   ▼
    OpLogEffects              │             Other Effects
          │                   │
          ├──► SUP_OPS ◄───────┘   (local persistence — IndexedDB)
          │
          └──► Sync Providers
               ├── SuperSync   (operation-based, real-time)
               └── WebDAV / Dropbox / LocalFile  (file-based, single sync-data.json)

Start here

You want to… Read
Write an effect/reducer/bulk-dispatch correctly contributor-sync-model.md — the one invariant, enforced by lint
Understand the whole architecture + why it's built this way operation-log-architecture.md — Parts AF + rejected alternatives
See it visually diagrams/ — 8 topic diagrams

Reference docs

Document Scope
operation-log-architecture.md Authoritative architecture: Local Persistence (A), File-Based Sync (B), Server Sync (C), Validation & Repair (D), Smart Archive (E), Atomic State Consistency (F), and Why this architecture: rejected alternatives
contributor-sync-model.md The single sync invariant for contributors (one intent = one op; replayed/remote ops must not re-trigger effects)
operation-rules.md Design rules and guidelines for operations
package-boundaries.md Dependency/ownership boundaries for @sp/sync-core, @sp/sync-providers, app wiring
vector-clocks.md Vector clock implementation, pruning, history
supersync-encryption-architecture.md End-to-end encryption (AES-256-GCM + Argon2id)
diagrams/ Mermaid diagrams split by topic

Scenario catalogs (expected behavior)

Document Scope
supersync-scenarios.md Concrete SuperSync scenarios AG with expected behavior
supersync-scenarios-flowchart.md Visual decision tree for the SuperSync scenarios
file-based-sync-flowchart.md Visual decision tree for file-based providers
Location Content
packages/super-sync-server/ SuperSync server implementation
ARCHITECTURE-DECISIONS.md Load-bearing product/data decisions

Historical design notes and superseded plans are not kept as docs; they live in git history (reference the relevant commit if you need the rationale).