super-productivity/docs/sync-and-op-log
Johannes Millan 05f6bd27d1
fix(op-log): serialize SQLite adapter transactions on the shared connection (#8849)
* fix(op-log): serialize SQLite adapter transactions on shared connection

The SQLite op-log adapter issued raw BEGIN/COMMIT on the shared
connection with no serialization. Once both stores share one SqliteDb
(the staged native rollout), concurrent operations (capture append,
archive write, compaction) would interleave BEGINs: SQLite has no
nested transactions, so a second BEGIN throws and a bare statement
issued mid-transaction silently joins — and rolls back with — the
foreign transaction, corrupting op-log state.

Funnel every adapter entry point through an internal FIFO queue so a
transaction is exclusive on the connection for its whole BEGIN/COMMIT
and no bare operation interleaves. Transaction-internal work runs
directly on the connection (already holding the slot), so there is no
re-entrancy. Document the mutual-exclusion invariant in the port
contract and add a concurrent-transactions contract test that runs on
both the in-memory fake and real sql.js.

* docs: add complete architecture review report (2026-07-07)

Whole-app architecture review synthesized from eight parallel subsystem
reviews and an adversarial verification pass; findings filed as issues
#8832-#8843, with duplicates cross-referenced rather than re-filed.

* fix(op-log): key the SQLite transaction serializer to the connection

Multi-agent review found the serializer was keyed to the adapter
instance. The native rollout hands the op-log store and archive store
two separate SqliteOpLogAdapter instances over one shared SqliteDb, so
per-instance queues left an op-log BEGIN free to interleave with a
concurrent archive BEGIN on the shared connection — the exact hazard the
serializer targets.

Key the FIFO queue to the connection (WeakMap<SqliteDb>) so every adapter
over one SqliteDb shares one queue. Add a contract test that drives two
adapters over one connection concurrently (verified red with per-instance
keying, green with per-connection). Also: document the re-entrancy
precondition as unenforced (a lint rule, not a runtime flag, is the right
guard — a flag cannot distinguish a re-entrant call from a legal
concurrent one) and correct init/getLastSeq/port-contract doc drift.
2026-07-07 18:00:25 +02:00
..
diagrams refactor: retire GET /api/sync/snapshot, re-scope GET /api/sync/status as diagnostic (#8496) 2026-06-20 12:02:58 +02:00
contributor-sync-model.md docs(sync): consolidate sync docs + enforce the contributor model 2026-05-15 16:51:50 +02:00
file-based-sync-flowchart.md refactor(sync): remove unused error classes, dialog, and constructor-time logging (phase 1, #8325) (#8510) 2026-06-20 13:36:56 +02:00
operation-log-architecture.md refactor: remove dead SyncStateCorruptedError, compat exports, and 0 byte sync-providers barrel #8328 (#8396) 2026-06-15 14:16:14 +02:00
operation-rules.md fix(sync): prevent lock-timeout from wedging op capture (#8306, #8318) (#8383) 2026-06-13 16:14:48 +02:00
package-boundaries.md refactor(sync-core): drop shared-schema vector-clock compat re-export anda app enum (#8441) 2026-06-17 16:21:26 +02:00
README.md docs(sync): consolidate sync docs + enforce the contributor model 2026-05-15 16:51:50 +02:00
sqlite-migration-followup.md fix(op-log): serialize SQLite adapter transactions on the shared connection (#8849) 2026-07-07 18:00:25 +02:00
sqlite-migration.md feat(op-log): validate SQLite backend + IDB→SQLite migration + backend-aware init (#7931) (#7954) 2026-06-02 17:26:23 +02:00
supersync-encryption-architecture.md docs(sync): add encryption architecture and scenario documentation 2026-03-01 21:27:08 +01:00
supersync-scenarios-flowchart.md docs(sync): align SYNC_IMPORT scenarios with current gate semantics 2026-04-29 16:17:56 +02:00
supersync-scenarios.md feat(supersync): retry transient web fetch failures and surface them as warnings 2026-05-13 11:20:02 +02:00
vector-clocks.md refactor(sync-core): drop shared-schema vector-clock compat re-export anda app enum (#8441) 2026-06-17 16:21:26 +02:00

Operation Log & Sync Documentation

The Operation Log is the single sync system for all providers (SuperSync, WebDAV, Dropbox, LocalFile). It is an event-sourced persistence + sync layer: the log is the source of truth, current state is derived by replaying it, and vector clocks detect concurrent edits.

                         User Action
                              │
                              ▼
                         NgRx Store  (runtime source of truth)
                              │
          ┌───────────────────┼───────────────────┐
          ▼                   │                   ▼
    OpLogEffects              │             Other Effects
          │                   │
          ├──► SUP_OPS ◄───────┘   (local persistence — IndexedDB)
          │
          └──► Sync Providers
               ├── SuperSync   (operation-based, real-time)
               └── WebDAV / Dropbox / LocalFile  (file-based, single sync-data.json)

Start here

You want to… Read
Write an effect/reducer/bulk-dispatch correctly contributor-sync-model.md — the one invariant, enforced by lint
Understand the whole architecture + why it's built this way operation-log-architecture.md — Parts AF + rejected alternatives
See it visually diagrams/ — 8 topic diagrams

Reference docs

Document Scope
operation-log-architecture.md Authoritative architecture: Local Persistence (A), File-Based Sync (B), Server Sync (C), Validation & Repair (D), Smart Archive (E), Atomic State Consistency (F), and Why this architecture: rejected alternatives
contributor-sync-model.md The single sync invariant for contributors (one intent = one op; replayed/remote ops must not re-trigger effects)
operation-rules.md Design rules and guidelines for operations
package-boundaries.md Dependency/ownership boundaries for @sp/sync-core, @sp/sync-providers, app wiring
vector-clocks.md Vector clock implementation, pruning, history
supersync-encryption-architecture.md End-to-end encryption (AES-256-GCM + Argon2id)
diagrams/ Mermaid diagrams split by topic

Scenario catalogs (expected behavior)

Document Scope
supersync-scenarios.md Concrete SuperSync scenarios AG with expected behavior
supersync-scenarios-flowchart.md Visual decision tree for the SuperSync scenarios
file-based-sync-flowchart.md Visual decision tree for file-based providers
Location Content
packages/super-sync-server/ SuperSync server implementation
ARCHITECTURE-DECISIONS.md Load-bearing product/data decisions

Historical design notes and superseded plans are not kept as docs; they live in git history (reference the relevant commit if you need the rationale).