Mirrors/super-productivity
1
0
Fork 0
mirror of https://github.com/johannesjo/super-productivity.git synced 2026-07-17 16:37:43 +00:00
Code Issues Projects Releases Wiki Activity

docs(sync): consolidate sync docs + enforce the contributor model

Browse source 
Collapse the sprawling, partly-stale docs/sync-and-op-log/ tree into a
small authoritative set and make the sync-correctness invariant
partly lint-enforced instead of convention-only.

Docs:
- Delete superseded/duplicate/provably-stale design, plan, and
  background-research docs (quick-reference, the architecture-diagrams
  monolith, the "Hybrid Manifest" docs describing code that does not
  exist, completed long-term plans, LLM-synthesis analyses).
- Salvage load-bearing decision history into the surviving docs before
  deletion: rejected-alternatives rationale -> operation-log-architecture
  ("Why this architecture"); vector-clock pruning incident history ->
  vector-clocks.md; archive-payload optimization -> architecture E.7.
- Add contributor-sync-model.md as the single-invariant entry point
  (one user intent = one op; replayed/remote ops must not re-trigger
  effects), with a decision table mapping to the enforcing linters.
- Repoint external/internal cross-refs; add CONTRIBUTING.md + CLAUDE.md
  pointers; record the migration in a dated docs/plans/ design doc.

Enforcement (new eslint-local-rules):
- no-actions-in-effects (error): effects must inject LOCAL_ACTIONS /
  ALL_ACTIONS, never the raw @ngrx/effects Actions stream.
- no-multi-entity-effect (warn, heuristic): flags a literal returned
  array of >=2 action-creator calls; docstring + valid-case specs pin
  exactly which shapes are and are not detected.
- run-specs.js runner wired into `npm run lint` via test:lint-rules;
  refuses to run under test-framework globals and counts RuleTester.run
  invocations so a spec that asserts nothing fails instead of passing.
- Correct the ALL_ACTIONS JSDoc in local-actions.token.ts to match
  reality (archive-operation-handler uses LOCAL_ACTIONS).

Reviewed via parallel multi-agent review; findings W1/W2/W4 and a
dangling doc anchor addressed.
This commit is contained in:
Johannes Millan 2026-05-15 15:32:38 +02:00
parent d29c061646
commit 42e6626b76
29 changed files with 931 additions and 5715 deletions
Download patch file Download diff file Expand all files Collapse all files

11
CLAUDE.md
View file

@ -7,6 +7,7 @@ Guidance for Claude Code working in this repository. Super Productivity is a tod
- Styling changes → [`docs/styling-guide.md`](docs/styling-guide.md)
- User-facing functionality changes → [`docs/documentation-guide.md`](docs/documentation-guide.md)
- Sync, op-log, vector clocks → [`docs/sync-and-op-log/`](docs/sync-and-op-log/)
- Effects/reducers/bulk-dispatch touching synced state → [`docs/sync-and-op-log/contributor-sync-model.md`](docs/sync-and-op-log/contributor-sync-model.md)
- E2E tests → [`e2e/CLAUDE.md`](e2e/CLAUDE.md)
- Load-bearing decisions → [`ARCHITECTURE-DECISIONS.md`](ARCHITECTURE-DECISIONS.md)
@ -41,14 +42,14 @@ For SuperSync E2E (docker-compose) and the full E2E reference, see [`e2e/CLAUDE.
## Sync-correctness rules
Touched on most state-related PRs. Read the linked source/doc for full reasoning before editing.
Touched on most state-related PRs. Read the linked source/doc for full reasoning before editing. Rules 1–3 and 6 are one invariant — *one user intent = one op; replayed/remote ops must not re-trigger effects* — fully explained in [`docs/sync-and-op-log/contributor-sync-model.md`](docs/sync-and-op-log/contributor-sync-model.md).
1. **Effects use `inject(LOCAL_ACTIONS)`**, never `inject(Actions)` — effects must not run for remote sync ops. For archive-specific side effects on remote clients (writing/deleting from IndexedDB), use `ArchiveOperationHandler` (called by `OperationApplierService`). → `src/app/util/local-actions.token.ts`, `docs/sync-and-op-log/operation-log-architecture-diagrams.md` §8.
2. **Action-based effects only.** Selector-based effects fire on every store change (including hydration/sync replay) and bypass `LOCAL_ACTIONS` filtering; if unavoidable, wrap with the `skipDuringSyncWindow()` operator (or guard with `HydrationStateService.isApplyingRemoteOps()`). → `src/app/features/tag/store/tag.effects.ts`.
3. **Multi-entity changes use meta-reducers**, not effects — one reducer pass = one sync op (e.g. deleting a tag also removes it from every task). → `src/app/root-store/meta/task-shared-meta-reducers/`.
1. **Effects inject `LOCAL_ACTIONS`**, never `Actions` (`ALL_ACTIONS` only for the op-log capture effect; remote archive side effects → `ArchiveOperationHandler`, not `ALL_ACTIONS`). Lint-enforced (`no-actions-in-effects`). → [contributor-sync-model.md](docs/sync-and-op-log/contributor-sync-model.md), `src/app/util/local-actions.token.ts`.
2. **Prefer action-based effects**; a selector-based effect needs `skipDuringSyncWindow()`. Lint-enforced (`require-hydration-guard`). → [contributor-sync-model.md](docs/sync-and-op-log/contributor-sync-model.md).
3. **Multi-entity change = meta-reducer**, not an effect fan-out (one reducer pass = one op). → [contributor-sync-model.md](docs/sync-and-op-log/contributor-sync-model.md), `src/app/root-store/meta/task-shared-meta-reducers/`.
4. **Logical clock:** route "what day is this?" through `DateService` (`getLogicalTodayDate`, `isToday`, `todayStr`). Pure reducers/selectors take `startOfNextDayDiffMs` as an arg and call `isTodayWithOffset` for replay determinism. The raw `DateService.startOfNextDayDiff` is `private`; use `getStartOfNextDayDiffMs()` at service boundaries.
5. **`TODAY_TAG` (`'TODAY'`) is virtual** — never add to `task.tagIds`; membership comes from `task.dueWithTime` or `task.dueDay`. `TODAY_TAG.taskIds` only stores ordering. → `ARCHITECTURE-DECISIONS.md` Decision #2.
6. **Bulk dispatches:** add `await new Promise(r => setTimeout(r, 0))` after the loop — `store.dispatch()` is non-blocking, and without yielding, 50+ rapid dispatches lose state updates. → `OperationApplierService.applyOperations()`.
6. **Bulk dispatch loop:** `await new Promise(r => setTimeout(r, 0))` after the loop (else 50+ rapid dispatches lose state). → [contributor-sync-model.md](docs/sync-and-op-log/contributor-sync-model.md), `OperationApplierService.applyOperations()`.
7. **`SYNC_IMPORT` / `BACKUP_IMPORT`** replace state and intentionally drop concurrent ops (CONCURRENT or LESS_THAN by vector clock) — by design, not a bug. → `SyncImportFilterService`.
8. **Vector clocks:** `MAX_VECTOR_CLOCK_SIZE = 20`. Server prunes after conflict detection, before storage. → `docs/sync-and-op-log/vector-clocks.md`.
9. **Logging:** `Log.log({ id: task.id })`, never `Log.log(task)` or `Log.log(title)` — log history is exportable, never log user content.

2
CONTRIBUTING.md
View file

@ -14,7 +14,7 @@ In case you want to contribute, but you wouldn't know how, here are some suggest
1. **Spread the word:** More users means more people testing and contributing to the app which in turn means better stability and possibly more and better features. You can vote for Super Productivity on [Slant](https://www.slant.co/topics/14021/viewpoints/7/~productivity-tools-for-linux~super-productivity), [Product Hunt](https://www.producthunt.com/posts/super-productivity), [Softpedia](https://www.softpedia.com/get/Office-tools/Diary-Organizers-Calendar/Super-Productivity.shtml) or on [AlternativeTo](https://alternativeto.net/software/super-productivity/), you can [tweet about it](https://twitter.com/intent/tweet?text=I%20like%20Super%20Productivity%20%20https%3A%2F%2Fsuper-productivity.com), share it on [LinkedIn](http://www.linkedin.com/shareArticle?mini=true&url=https://super-productivity.com&title=I%20like%20Super%20Productivity&), [reddit](http://www.reddit.com/submit?url=https%3A%2F%2Fsuper-productivity.com&title=I%20like%20Super%20Productivity) or any of your favorite social media platforms. Every little bit helps!
2. **Provide a Pull Request:** Here is a list of [the most popular community requests](https://github.com/super-productivity/super-productivity/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc) and here some info on **[how to run the development build](https://github.com/super-productivity/super-productivity/wiki/2.11-Run-the-Development-Server)** (wiki). Please make sure that you're following the commit message format documented in [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md#commit-message-format) and to also include the issue number in your commit message, if you're fixing a particular issue (e.g.: `feat: add nice feature #31`).
2. **Provide a Pull Request:** Here is a list of [the most popular community requests](https://github.com/super-productivity/super-productivity/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc) and here some info on **[how to run the development build](https://github.com/super-productivity/super-productivity/wiki/2.11-Run-the-Development-Server)** (wiki). Please make sure that you're following the commit message format documented in [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md#commit-message-format) and to also include the issue number in your commit message, if you're fixing a particular issue (e.g.: `feat: add nice feature #31`). If your change touches synced state (effects, reducers, bulk dispatches), read the [Contributor Sync Model](docs/sync-and-op-log/contributor-sync-model.md) first — one invariant, partly lint-enforced.
3. **[Answer questions](https://github.com/super-productivity/super-productivity/discussions)**: You know the answer to another user's problem? Share your knowledge!

4
docs/long-term-plans/server-side-entity-versioning.md
View file

@ -323,9 +323,9 @@ Vector clock pruning remains for bandwidth efficiency, but pruning errors no lon
## Relationship to Other Plans
- **Builds on:** [Server-Side Prune-Aware Comparison](./server-side-prune-aware-comparison.md) -- can be implemented in either order; prune-aware comparison improves the vector clock fallback path
- **Builds on:** Server-Side Prune-Aware Comparison (plan doc removed in `985e839747`; view it with `git show 669f2d7874:docs/long-term-plans/server-side-prune-aware-comparison.md`) -- can be implemented in either order; prune-aware comparison improves the vector clock fallback path
- **Related:** Current client-side fix (commit `f9be1c8500`) remains as defense-in-depth for the vector clock fallback path
- **Related:** [E2E Encryption Plan](../sync-and-op-log/long-term-plans/e2e-encryption-plan.md) -- entity versions are not sensitive data and do not need encryption
- **Related:** [SuperSync Encryption Architecture](../sync-and-op-log/supersync-encryption-architecture.md) -- entity versions are not sensitive data and do not need encryption
## Implementation Order

217
docs/plans/2026-05-15-sync-docs-and-contributor-model-design.md Normal file
View file

@ -0,0 +1,217 @@
# Sync Simplification: Docs Consolidation + Enforced Contributor Model
**Date:** 2026-05-15
**Status:** Design — revised after multi-review (gemini + Claude sub-agent; codex/copilot unavailable in this env). Three confirmed blockers from review folded in below.
**Scope:** Tier 1 + Tier 2 only (see "Scope"). No production TypeScript changes; behavior-preserving.
## Context & core finding
Goal: reduce the **maintenance burden** and **conceptual complexity** of the sync
architecture — less to hold in your head, easier to onboard.
Research (four parallel deep-dives, see "Evidence") produced a counterintuitive
conclusion that shapes this whole plan:
> **The sync *code* is not meaningfully over-engineered. The team already did the
> hard simplification (deleted PFAPI ~83 files, removed the vector-clock defense
> layers, unified both transports behind one `OperationSyncCapable` interface +
> a shared `@sp/sync-core` orchestrator). Three prior independent analyses
> rejected every simpler model (delta-sync, LWW, CRDT) for reasons tied to a
> hard, non-negotiable constraint: no silent data loss on concurrent
> multi-device edits, offline-first, with a dumb/E2EE file server.**
Findings per candidate area:
| Area | Verdict | Safe code reduction | Risk |
|---|---|---|---|
| Transport duplication | Already unified; `file-based-sync-adapter` is *necessary server-emulation on dumb storage*, not redundancy | ~50–120 LOC, touches the most fragile code (snapshot-hydration; issues #7339/#7330) | High — **excluded** |
| Validation/repair | Mostly load-bearing; "4522 LOC" includes 471 test-only LOC | ~155–215 LOC (Tier 3, **deferred**) | Low |
| Four contributor rules | **Real win** — all four are one invariant; codebase already 100% compliant | n/a (adds lint) | Very low |
| Doc sprawl | **Biggest win** — ~33 files/~600 KB, one provably-stale doc falsely marked "Completed" | n/a (docs) | Near-zero |
Therefore: the maintenance-burden ceiling for *code* is low and risky. The
**conceptual-complexity** pain has a large, cheap, low-risk fix that lives in
the **docs** and the **scattered/unenforced contributor rules** — that is this plan.
## Scope
**In scope (Tier 1 + 2):**
1. Consolidate `docs/sync-and-op-log/` from ~33 files to a lean authoritative set.
2. Add one new `contributor-sync-model.md` capturing the single sync invariant.
3. Add two ESLint rules to the existing `eslint-local-rules/` plugin to *enforce*
the model instead of relying on memory.
4. Tighten CLAUDE.md sync rules 1–3,6 to one line each + link to the new doc.
**Explicitly out of scope (not in this plan):**
- Tier 3 code cleanup (dead `DataRepairService`, typia-redundant guards,
`providerMode` discriminant). Tracked separately; low payoff, deferred.
- Any change to sync runtime behavior, the op-log core, vector clocks,
conflict resolution, providers, or `super-sync-server`.
- Replacing the engine or dropping providers (rejected earlier in research).
## Tier 1 — Documentation consolidation
### Target active doc set (7 docs)
| Doc | Action |
|---|---|
| `README.md` | Rewrite as a pure navigation index. Drop the historical/status tables (they drift; that drift is part of the problem). |
| `operation-log-architecture.md` | Remains the **one** authoritative architecture doc. Fold in: (a) `quick-reference.md`'s unique cheat-sheet tables as an appendix; (b) a new condensed **"Rejected alternatives & why"** section preserving the load-bearing rationale from `background-info/` (no-silent-data-loss / offline / dumb-E2EE-server constraint; why delta-sync, LWW, CRDT were rejected). |
| `contributor-sync-model.md` | **New.** The single contributor mental model (see Tier 1 §"New doc"). |
| `vector-clocks.md` | Keep as-is (current; cited by CLAUDE.md rule 8). |
| `supersync-encryption-architecture.md` | Keep as-is (current; implemented). |
| `operation-rules.md` | Keep as-is (short, current, lint-aligned). |
| `package-boundaries.md` | Keep as-is (short, current, matches enforced eslint boundaries). |
| `diagrams/` (directory) | Keep as the canonical diagram set. Fold in the 3 stray flowcharts' content where unique. |
### Deletions (hard-delete; git history is the archive)
No `archive/` folder. Delete; if a surviving doc needs the rationale, link the
**git commit** that removed it (`see commit <hash> for historical <topic> design`).
- `long-term-plans/hybrid-manifest-architecture.md` — **provably stale & misleading**: describes a multi-file `manifest.json` + `ops/` scheme with **zero** code references (`OperationLogManifestService` does not exist; the live format is single-file `sync-data.json`), yet self-labels "Completed". Highest-priority removal.
- `long-term-plans/replace-pfapi-with-oplog-plan.md` — completed Jan 2026; outcome captured by current architecture doc.
- `long-term-plans/e2e-encryption-plan.md` — superseded by `supersync-encryption-architecture.md` (its own header says so).
- `operation-payload-optimization-discussion.md` — dated discussion, not a spec.
- `background-info/` (5 files) — historical research/LLM-synthesized analyses. **Note (review R3):** the synthesized reports self-caveat that the models analyzed different/stale artifacts, so their *specifics are unreliable*; only the durable constraint (no-silent-data-loss / offline / dumb-E2EE-server, and why delta-sync/LWW/CRDT were rejected) is load-bearing. `operation-log-architecture.md` currently has **no** rejected-alternatives section (it covers LWW only as the *implemented* strategy at :1365). So the fold is **net-new synthesis written from first principles**, not mechanical extraction — a writing-judgment task, done **before** deletion.
- `quick-reference.md` — unique cheat-sheet tables folded into the architecture doc, then deleted.
- `operation-log-architecture-diagrams.md` (86 KB monolith) — unique **current** diagrams folded into `diagrams/`, then deleted. **Carve-out (review C2): exclude §5 and §6 "Hybrid Manifest ✅ IMPLEMENTED" (lines ~1507–1546) from the fold** — they assert `OperationLogManifestService` is "Complete", the exact false claim driving the hybrid-manifest deletion. They are deleted, not migrated. Sweep the kept `diagrams/*` for any other `HybridManifest`/`OperationLogManifestService` content during step 2.
- `supersync-scenarios.md`, `supersync-scenarios-flowchart.md`, `file-based-sync-flowchart.md` — fold any unique current flow into `diagrams/`, then delete.
Net: ~33 files → **7 active docs + `diagrams/`**.
### Cross-reference fixes (must be done in the same change so no link dangles)
- `CLAUDE.md:46` → currently points at `operation-log-architecture-diagrams.md §8`; repoint to `contributor-sync-model.md`.
- `operation-log-architecture.md:1545` → ref to diagrams monolith Section 2c; repoint to the new `diagrams/` file.
- `operation-log-architecture.md:2340` → ref to `long-term-plans/hybrid-manifest-architecture.md`; remove (or replace with commit-hash note if rationale is wanted).
- `diagrams/README.md:59` → ref to `../quick-reference.md`; remove (folded into architecture doc).
- Inter-flowchart links in `supersync-scenarios-flowchart.md`, `file-based-sync-flowchart.md`, `quick-reference.md:3` → resolved by the merges.
- `README.md:41` (`replace-pfapi-...`), `README.md:42` (`e2e-encryption-plan`), `README.md:167` (`background-info/`) → all removed by the full README rewrite (step 5); listed for checklist completeness.
- **(review C1 — must-fix) External, outside `docs/sync-and-op-log/`:** `docs/long-term-plans/server-side-entity-versioning.md:328` links to `../sync-and-op-log/long-term-plans/e2e-encryption-plan.md` (a deleted doc). Repoint to `../sync-and-op-log/supersync-encryption-architecture.md` (the kept E2EE reference). This file is **not** in the doc set so it must be an explicit change-set item, not left to verify-time discovery.
### New doc: `contributor-sync-model.md`
States **one invariant, two boundaries, one atomicity rule**:
> **One user intent = exactly one operation. Replayed/remote ops must never
> re-trigger effects.**
>
> - **Action boundary** — effects inject `LOCAL_ACTIONS`, not `Actions`.
> *Enforced by `local-rules/no-actions-in-effects` (Tier 2).*
> - **Selector boundary** — selector-driven effects guard with
> `skipDuringSyncWindow()` / `HydrationStateService.isApplyingRemoteOps()`.
> *Enforced by the existing `local-rules/require-hydration-guard`.*
> - **Atomicity** — multi-entity changes are meta-reducers (one reducer pass =
> one op); bulk-dispatch loops yield with
> `await new Promise(r => setTimeout(r, 0))`.
Plus a short decision table ("Writing an effect? → these checks; the linter
enforces two of them") and links to `operation-rules.md` for the deeper "why".
## Tier 2 — Enforce the model (ESLint)
Existing plugin: `eslint-local-rules/` (`eslint-plugin-local-rules` convention),
`rules/require-hydration-guard.js` + `require-entity-registry.js`, registered in
`eslint.config.js:216-222` for `**/*.effects.ts`. Add, following that exact pattern:
- **`eslint-local-rules/rules/no-actions-in-effects.js`** (`error`): bans
`inject(Actions)` and the `Actions` import (incl. aliased
`import { Actions as X } from '@ngrx/effects'`) in `*.effects.ts`;
message/suggestion points to `LOCAL_ACTIONS`/`ALL_ACTIONS`. Codebase is
**already 100% compliant** (verified: 0 `inject(Actions)`, 0 `@ngrx/effects`
`Actions` imports across all 43 real `*.effects.ts`) → zero migration, pure
regression guard. **Correction (review R2):** the existing rules do
CallExpression/selector analysis only and have **no `ImportDeclaration`
handling**; this rule follows the *plugin + spec structure* of the existing
rules but adds new `ImportDeclaration` + `inject()`-call detection. The spec
must cover the aliased-import case.
- **`eslint-local-rules/rules/no-multi-entity-effect.js`** (`warn`): heuristic —
flags an effect whose dispatch arm references >1 feature slice / >1 entity action
creator; message points to `root-store/meta/task-shared-meta-reducers/`. `warn`
(like `require-entity-registry`) because the heuristic has false positives;
inline-disable with a justification comment is allowed.
Each gets a co-located `.spec.js` (ESLint `RuleTester`) modeled on
`require-hydration-guard.spec.js`, registered in `eslint-local-rules/index.js`
and added to `eslint.config.js`. The `no-multi-entity-effect` spec must include a
**positive "blessed path" case** (a multi-entity change routed through a
`task-shared-meta-reducers/` meta-reducer) so the correct pattern is documented
in-test.
**Spec runner (review C3 — must-fix):** `npm test` runs Karma over `*.spec.ts`
only; it does **not** run `.spec.js`. The existing `require-hydration-guard.spec.js`
currently has **no runner and no CI step** (dead coverage). Add a
`"test:lint-rules"` npm script (e.g. `node --test "eslint-local-rules/**/*.spec.js"`)
plus a CI step, which also resurrects the existing orphaned spec. This is the
only addition beyond docs+rules and is in-scope for Tier 2.
No production TypeScript changes. No runtime behavior change.
## CLAUDE.md changes
- Rules **1, 2, 3, 6** (the four facets of the one invariant): tighten each to a
single terse line that still states the guardrail (kept in always-loaded
context) but moves mechanism/why to `contributor-sync-model.md` via link.
- Rule **1**'s doc pointer: `operation-log-architecture-diagrams.md §8` →
`docs/sync-and-op-log/contributor-sync-model.md`.
- Rules **4, 5, 7, 8, 9** are unrelated to this invariant → unchanged
(rule 8 still points at `vector-clocks.md`, which is kept).
## Execution order (so links never dangle mid-migration)
1. Create `contributor-sync-model.md`.
2. Fold `background-info/` rationale (net-new synthesis) + `quick-reference.md`
tables + diagram monolith content into `operation-log-architecture.md` /
`diagrams/` — **excluding** the monolith's stale §5/§6 Hybrid Manifest
sections (C2).
3. Fix all cross-references (table above) to final destinations — **including
the external `docs/long-term-plans/server-side-entity-versioning.md:328`
(C1)**.
4. Delete the stale/superseded/folded source docs.
5. Rewrite `README.md` as an index of the final 7 docs + `diagrams/`; add
`contributor-sync-model.md` to CLAUDE.md "Required reading per task" and link
it from `CONTRIBUTING.md` (visibility — gemini suggestion).
6. Add the two ESLint rules + specs + `index.js`/`eslint.config.js`
registration + the `test:lint-rules` npm script + CI step (C3).
7. Tighten CLAUDE.md rules 1–3,6 + repoint rule 1.
8. Run the full Verification checklist; only then is the change complete.
## Verification
- Markdown link check across **all of `docs/` (incl. `docs/long-term-plans/`) and
CLAUDE.md** → zero dangling links. (The sweep must NOT exclude
`docs/long-term-plans/` — that is where the C1 external ref lives.)
- `grep -rn "hybrid-manifest\|quick-reference\|architecture-diagrams\|background-info\|supersync-scenarios\|file-based-sync-flowchart\|payload-optimization\|replace-pfapi\|e2e-encryption-plan"` over `*.md *.ts *.js` (excluding only `docs/sync-and-op-log/` and `docs/plans/`) → zero hits after migration.
- `npm run lint` clean; `no-actions-in-effects` produces **0** violations on the
current tree (proves it is a pure regression guard, not a migration).
- `npm run test:lint-rules` green (the new runner; also re-covers the
previously-orphaned `require-hydration-guard.spec.js`).
- **Tightened (review R4):** `git grep -E "HybridManifest|OperationLogManifestService"`
over `src/ packages/` → zero. (Do **not** grep bare `manifest.json` — it has
dozens of unrelated plugin/i18n hits and would false-positive.)
## Risks
- **Low overall** — docs + non-bypassable lint + CLAUDE.md text. No production
code path changes; no sync behavior change.
- *Knowledge loss on delete:* mitigated by folding load-bearing rationale into
the architecture doc **before** deletion, plus git history + commit-hash
references.
- *`no-multi-entity-effect` false positives:* mitigated by shipping as `warn`
with an allowed inline-disable + justification.
- *CLAUDE.md too terse:* the guardrail sentence stays in always-loaded context;
only the "why" moves to the linked doc.
## Evidence (research provenance)
- Complexity inventory: op-log ~28 K LOC; transports already unified behind
`OperationSyncCapable` + `@sp/sync-core`.
- Prior analyses (`background-info/`): op-log chosen over delta-sync/LWW/CRDT
due to the no-data-loss/offline/dumb-E2EE-server constraint.
- Stale-doc proof: zero `HybridManifest`/`manifest.json` refs in code; live
format is `sync-data.json` (`file-based-sync-adapter.service.ts`).
- Contributor-rule unification: 0 `inject(Actions)` in any `*.effects.ts`;
`require-hydration-guard` already enforces the selector boundary.

197
docs/sync-and-op-log/README.md
View file

@ -1,182 +1,61 @@
# Operation Log Documentation
# Operation Log & Sync Documentation
**Last Updated:** January 2026
This directory contains the architectural documentation for Super Productivity's Operation Log system - an event-sourced persistence and synchronization layer that handles ALL sync providers (SuperSync, WebDAV, Dropbox, LocalFile).
## Quick Start
| If you want to... | Read this |
| ----------------------------------- | ------------------------------------------------------------------------------ |
| Understand the overall architecture | [operation-log-architecture.md](./operation-log-architecture.md) |
| See visual diagrams | [diagrams/](./diagrams/) (split by topic) |
| Learn the design rules | [operation-rules.md](./operation-rules.md) |
| Understand package boundaries | [package-boundaries.md](./package-boundaries.md) |
| Understand file-based sync | [diagrams/04-file-based-sync.md](./diagrams/04-file-based-sync.md) |
| Understand SuperSync encryption | [supersync-encryption-architecture.md](./supersync-encryption-architecture.md) |
## Documentation Overview
### Core Documentation
| Document | Description | Status |
| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ |
| [operation-log-architecture.md](./operation-log-architecture.md) | Comprehensive architecture reference covering Parts A-F: Local Persistence, File-Based Sync, Server Sync, Validation & Repair, Smart Archive Handling, and Atomic State Consistency | Active |
| [diagrams/](./diagrams/) | Mermaid diagrams split by topic (local persistence, server sync, file-based sync, etc.) | Active |
| [operation-rules.md](./operation-rules.md) | Design rules and guidelines for the operation log store and operations | Active |
| [package-boundaries.md](./package-boundaries.md) | Dependency direction and ownership boundaries for `@sp/sync-core`, `@sp/sync-providers`, and app sync wiring | Active |
### Sync Architecture
| Document | Description | Status |
| ------------------------------------------------------------------------------ | --------------------------------------------------------------------- | ----------- |
| [diagrams/04-file-based-sync.md](./diagrams/04-file-based-sync.md) | File-based sync with single sync-data.json (WebDAV/Dropbox/LocalFile) | Implemented |
| [diagrams/02-server-sync.md](./diagrams/02-server-sync.md) | SuperSync server sync architecture | Implemented |
| [supersync-encryption-architecture.md](./supersync-encryption-architecture.md) | End-to-end encryption for SuperSync (AES-256-GCM + Argon2id) | Implemented |
### Historical / Completed Plans
| Document | Description | Status |
| -------------------------------------------------------------------------------------- | -------------------------------------------------------- | ---------------------- |
| [replace-pfapi-with-oplog-plan.md](./long-term-plans/replace-pfapi-with-oplog-plan.md) | Plan to unify sync by replacing PFAPI with operation log | Completed (Jan 2026) |
| [e2e-encryption-plan.md](./long-term-plans/e2e-encryption-plan.md) | Original E2EE design (see supersync-encryption for impl) | Implemented (Dec 2025) |
## Architecture at a Glance
The Operation Log system is the **single sync system** for all providers:
The Operation Log is the **single sync system** for all providers (SuperSync,
WebDAV, Dropbox, LocalFile). It is an event-sourced persistence + sync layer:
the log is the source of truth, current state is derived by replaying it, and
vector clocks detect concurrent edits.
```
User Action
│
▼
NgRx Store
(Runtime Source of Truth)
NgRx Store (runtime source of truth)
│
┌───────────────────┼───────────────────┐
▼ │ ▼
OpLogEffects │ Other Effects
│ │
├──► SUP_OPS ◄──────┘
│ (Local Persistence - IndexedDB)
├──► SUP_OPS ◄───────┘ (local persistence — IndexedDB)
│
└──► Sync Providers
├── SuperSync (operation-based, real-time)
├── WebDAV (file-based, single-file snapshot)
├── Dropbox (file-based, single-file snapshot)
└── LocalFile (file-based, single-file snapshot)
├── SuperSync (operation-based, real-time)
└── WebDAV / Dropbox / LocalFile (file-based, single sync-data.json)
```
### Sync Provider Types
## Start here
| Provider Type | Providers | How It Works |
| ---------------- | -------------------------- | ------------------------------------------------------------- |
| **Server-based** | SuperSync | Individual operations uploaded/downloaded via HTTP API |
| **File-based** | WebDAV, Dropbox, LocalFile | Single `sync-data.json` file with state snapshot + recent ops |
| You want to… | Read |
| ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Write an effect/reducer/bulk-dispatch correctly | **[contributor-sync-model.md](./contributor-sync-model.md)** — the one invariant, enforced by lint |
| Understand the whole architecture + why it's built this way | [operation-log-architecture.md](./operation-log-architecture.md) — Parts A–F + rejected alternatives |
| See it visually | [diagrams/](./diagrams/) — 8 topic diagrams |
### The Core Parts
## Reference docs
| Part | Purpose | Description |
| -------------------------- | --------------------------- | ----------------------------------------------------------------------------- |
| **A. Local Persistence** | Fast writes, crash recovery | Operations stored in IndexedDB (`SUP_OPS`), with snapshots for fast hydration |
| **B. File-Based Sync** | WebDAV/Dropbox/LocalFile | Single-file sync with state snapshot and embedded operations buffer |
| **C. Server Sync** | Operation-based sync | Upload/download individual operations via SuperSync server |
| **D. Validation & Repair** | Data integrity | Checkpoint validation with automatic repair and REPAIR operations |
| Document | Scope |
| ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [operation-log-architecture.md](./operation-log-architecture.md) | Authoritative architecture: Local Persistence (A), File-Based Sync (B), Server Sync (C), Validation & Repair (D), Smart Archive (E), Atomic State Consistency (F), and **Why this architecture: rejected alternatives** |
| [contributor-sync-model.md](./contributor-sync-model.md) | The single sync invariant for contributors (one intent = one op; replayed/remote ops must not re-trigger effects) |
| [operation-rules.md](./operation-rules.md) | Design rules and guidelines for operations |
| [package-boundaries.md](./package-boundaries.md) | Dependency/ownership boundaries for `@sp/sync-core`, `@sp/sync-providers`, app wiring |
| [vector-clocks.md](./vector-clocks.md) | Vector clock implementation, pruning, history |
| [supersync-encryption-architecture.md](./supersync-encryption-architecture.md) | End-to-end encryption (AES-256-GCM + Argon2id) |
| [diagrams/](./diagrams/) | Mermaid diagrams split by topic |
Additional architectural patterns:
## Scenario catalogs (expected behavior)
| Pattern | Purpose |
| ------------------------------- | ------------------------------------------------------------------ |
| **E. Smart Archive Handling** | Deterministic archive operations synced via instructions, not data |
| **F. Atomic State Consistency** | Meta-reducers ensure multi-entity changes are atomic |
| Document | Scope |
| ---------------------------------------------------------------------- | ------------------------------------------------------- |
| [supersync-scenarios.md](./supersync-scenarios.md) | Concrete SuperSync scenarios A–G with expected behavior |
| [supersync-scenarios-flowchart.md](./supersync-scenarios-flowchart.md) | Visual decision tree for the SuperSync scenarios |
| [file-based-sync-flowchart.md](./file-based-sync-flowchart.md) | Visual decision tree for file-based providers |
## Key Concepts
## Related
### Event Sourcing
| Location | Content |
| ---------------------------------------------------------------- | ----------------------------------- |
| [packages/super-sync-server/](../../packages/super-sync-server/) | SuperSync server implementation |
| [ARCHITECTURE-DECISIONS.md](../../ARCHITECTURE-DECISIONS.md) | Load-bearing product/data decisions |
The Operation Log treats the database as a **timeline of events** rather than mutable state:
- **Source of Truth**: The log is truth; current state is derived by replaying the log
- **Immutability**: Operations are never modified, only appended
- **Snapshots**: Periodic snapshots speed up hydration (replay from snapshot + tail ops)
### Vector Clocks
Vector clocks track causality for conflict detection:
- Each client has its own counter in the vector clock
- Comparison reveals: `EQUAL`, `LESS_THAN`, `GREATER_THAN`, or `CONCURRENT`
- `CONCURRENT` indicates a true conflict requiring resolution
### LOCAL_ACTIONS Token
Effects that perform side effects (snacks, external APIs, UI) must use `LOCAL_ACTIONS` instead of `Actions`:
```typescript
private _actions$ = inject(LOCAL_ACTIONS); // Excludes remote operations
```
This prevents duplicate side effects when syncing operations from other clients.
## Key Files
### Sync Packages
```
packages/sync-core/src/
├── operation.types.ts # Generic operation primitives
├── vector-clock.ts # Compare/merge/prune algorithms
├── conflict-resolution.ts # Pure conflict helpers
├── replay-coordinator.ts # Generic remote replay ordering
└── ports.ts # App-side orchestration contracts
packages/sync-providers/src/
├── super-sync/ # SuperSync provider implementation
├── file-based/dropbox/ # Dropbox provider implementation
├── file-based/webdav/ # WebDAV + Nextcloud providers
├── file-based/local-file/ # LocalFile provider classes
└── provider-types.ts # Provider-neutral contracts
```
### App Sync Wiring
```
src/app/op-log/sync-providers/
├── sync-providers.factory.ts # Composes app deps into package providers
├── credential-store.service.ts # OAuth/credential storage implementation
├── wrapped-provider.service.ts # Provider wrapper with encryption
├── file-based/ # File-based adapter + app shims
└── super-sync/ # SuperSync app shims + validators
```
### Core Operation Log
```
src/app/op-log/
├── core/ # Core types and operations
├── persistence/ # IndexedDB storage
├── sync/ # Sync orchestration
└── validation/ # Data validation and repair
```
## Related Documentation
| Location | Content |
| ---------------------------------------------------------------- | ------------------------------------- |
| [vector-clocks.md](./vector-clocks.md) | Vector clock implementation details |
| [packages/super-sync-server/](../../packages/super-sync-server/) | SuperSync server implementation |
| [background-info/](./background-info/) | Research and best practices documents |
## Implementation Status
| Component | Status |
| ---------------------------- | ------------------------------------------------ |
| Local Persistence (Part A) | Complete |
| File-Based Sync (Part B) | Complete (WebDAV, Dropbox, LocalFile) |
| Server Sync (Part C) | Complete (SuperSync) |
| Validation & Repair (Part D) | Complete |
| End-to-End Encryption | Complete (AES-256-GCM + Argon2id) |
| PFAPI Elimination | Complete (Jan 2026) |
| Cross-version Sync (A.7.11) | Documented (not yet implemented) |
| Schema Migrations | Infrastructure ready (no migrations defined yet) |
See [operation-log-architecture.md#implementation-status](./operation-log-architecture.md#implementation-status) for detailed status.
> Historical design notes and superseded plans are not kept as docs; they live
> in git history (reference the relevant commit if you need the rationale).

468
docs/sync-and-op-log/background-info/operation-log-best-practises2.md
View file

@ -1,468 +0,0 @@
# Operation Log Sync Server: Best Practices Research
**Status:** Research Complete
**Date:** December 2, 2025
**Purpose:** Inform the design of Super Productivity's server sync architecture
---
## Executive Summary
This document synthesizes best practices from industry leaders (Figma, Linear, Replicache) and academic research on operation-based synchronization systems. Key findings inform our architecture decisions for transitioning from file-based to operation-based sync.
---
## 1. Architecture Patterns
### 1.1 Server-Authoritative vs. Peer-to-Peer
| Pattern | Use Case | Examples |
| ------------------------ | ----------------------------- | ------------------ |
| **Server-authoritative** | Single source of truth needed | Linear, Replicache |
| **Peer-to-peer** | No central server required | CRDTs, Local-first |
| **Hybrid** | Server for ordering, peers ok | Figma multiplayer |
**Recommendation for Super Productivity:** Server-authoritative pattern. The server assigns monotonic sequence numbers, providing total ordering while clients handle optimistic updates.
**Source:** [Replicache - How It Works](https://doc.replicache.dev/concepts/how-it-works)
### 1.2 Push/Pull Protocol (Replicache Pattern)
The most robust pattern uses separate push and pull phases:
```
┌─────────┐ ┌─────────┐
│ Client │ │ Server │
└────┬────┘ └────┬────┘
│ │
│ PUSH: mutations[] │
│─────────────────────────────►│
│ │ Execute mutations
│ │ Update lastMutationID
│ │
│ PULL: since cookie │
│─────────────────────────────►│
│ │
│ Response: patch, cookie, │
│ lastMutationIDChanges │
│◄─────────────────────────────│
│ │
│ Rebase local state │
│ │
```
**Key insight:** The server re-executes client mutations rather than simply storing them. This allows server-side validation, side effects, and authoritative conflict resolution.
**Source:** [Replicache Push/Pull Reference](https://doc.replicache.dev/reference/server-push)
### 1.3 Linear Sync Engine Pattern
Linear uses a transaction-based model with delta broadcasting:
1. **Transactions** execute exclusively on the server
2. **Delta packets** broadcast to all clients (including originator)
3. Each action has a **sync ID** (monotonic sequence)
4. Transactions are cached in IndexedDB if offline
5. Automatic retry on reconnection
**Key insight:** Delta packets may differ from original transactions because the server performs side effects (e.g., generating history, enforcing constraints).
**Source:** [Reverse Engineering Linear's Sync Engine](https://github.com/wzhudev/reverse-linear-sync-engine)
---
## 2. Causality & Ordering
### 2.1 Vector Clocks
Vector clocks track causal relationships between events:
```typescript
type VectorClock = Record<string, number>; // { clientId: sequenceNumber }
// Comparison results:
// - BEFORE: a happened-before b
// - AFTER: a happened-after b
// - EQUAL: same logical time
// - CONCURRENT: conflict - neither ordered
```
**Properties:**
- Detect concurrent updates (conflicts)
- Preserve causal chains (answer never before question)
- Partial ordering only (no total order without server)
**Limitation:** Vector clocks grow with the number of clients. Pruning strategies needed for long-lived systems.
**Source:** [Vector Clocks and Conflicting Data](https://www.designgurus.io/course-play/grokking-the-advanced-system-design-interview/doc/vector-clocks-and-conflicting-data)
### 2.2 Hybrid Logical Clocks (HLC)
HLC combines physical and logical clocks, addressing vector clock limitations:
```typescript
interface HLC {
wallTime: number; // Physical clock component
logical: number; // Logical counter for same wallTime
nodeId: string; // For tiebreaking
}
```
**Advantages:**
- Fits in 64 bits (vs. unbounded vector clocks)
- Close to NTP time (human-readable)
- Used by MongoDB, CockroachDB, YugabyteDB
**Best Practices:**
- Sync nodes with NTP
- Set reasonable error bounds (250-500ms default)
- Validate client timestamps aren't too far from server time
- Allow 20s-1min clock drift tolerance in production
**Source:** [Hybrid Logical Clocks in Depth](https://medium.com/geekculture/all-things-clock-time-and-order-in-distributed-systems-hybrid-logical-clock-in-depth-7c645eb03682)
### 2.3 Server-Assigned Sequence Numbers
The simplest approach for server-authoritative systems:
```sql
-- Per-user monotonic sequence
UPDATE user_sync_state
SET last_seq = last_seq + 1
WHERE user_id = ?
RETURNING last_seq;
```
**Trade-off:** Requires server connectivity for total ordering, but simplifies conflict detection to sequence comparison.
---
## 3. Conflict Resolution Strategies
### 3.1 Strategy Matrix
| Strategy | Use Case | Complexity | User Friction |
| --------------------- | --------------------------- | ---------- | ------------- |
| **Last-Write-Wins** | Low-stakes, non-collab | Low | Medium |
| **Object Versioning** | Git-like history needed | Medium | High |
| **CRDTs** | Math-guaranteed convergence | High | Low |
| **Server Aggregate** | Simplified client | Medium | Low |
| **Application Logic** | Custom per-field rules | Medium | Low |
**Source:** [Hasura Offline-First Design Guide](https://hasura.io/blog/design-guide-to-offline-first-apps)
### 3.2 Operation-Based CRDT Requirements
For operation-based CRDTs (relevant to our op-log approach):
1. **Commutativity:** Operations can apply in any order
2. **Idempotency:** Same operation applied twice = same result
3. **Reliable Causal Broadcast (RCB):** All updates eventually reach all nodes
```typescript
// CRDT interface pattern
interface OperationCRDT<State, Op> {
empty(): State;
query(state: State): unknown;
prepare(state: State, command: unknown): Op;
effect(state: State, op: Op): State;
}
```
**Source:** [Operation-Based CRDTs Protocol](https://www.bartoszsypytkowski.com/operation-based-crdts-protocol/)
### 3.3 Replicache's Rebase Strategy
Instead of applying patches directly, Replicache uses git-like rebase:
1. Rewind to last confirmed server state
2. Apply server patch
3. Replay pending local mutations
4. Atomically reveal result
**Key insight:** Mutators are arbitrary code that can express any conflict resolution policy. The application defines merge semantics, not the sync engine.
**Source:** [Replicache Concepts](https://doc.replicache.dev/concepts/how-it-works)
### 3.4 Field-Level Resolution
Modern systems apply different strategies per field:
| Field Type | Strategy | Rationale |
| ------------ | ---------------- | ----------------------- |
| Single value | LWW | User expects latest |
| Set/Tags | Union | Additive, no data loss |
| Counter/Time | Sum deltas | Mathematically correct |
| Ordered list | LCS + interleave | Preserve both orderings |
| Rich text | OT or CRDT | Character-level merge |
---
## 4. Tombstone & Deletion Handling
### 4.1 Why Tombstones Are Required
In eventually consistent systems, deletions must be tracked:
```
Without tombstones:
Node A: DELETE item-123
Node B: (offline, has item-123)
Node B: (comes online) → "Node A is missing item-123, let me sync it!"
Result: Deleted item resurrects
```
**Source:** [Tombstones in Distributed Systems](<https://en.wikipedia.org/wiki/Tombstone_(data_store)>)
### 4.2 Best Practices
| Practice | Recommendation |
| ------------------------ | ---------------------------------------- |
| **Grace period** | 90 days minimum (Cassandra default: 10d) |
| **Repair before expiry** | All nodes must see tombstone before GC |
| **Soft delete flag** | Better for frequent un-deletes |
| **Tombstone table** | Better for audit trails |
| **Avoid mass deletions** | Creates tombstone storms |
**Cleanup safety rule:** Only garbage-collect tombstones after:
1. Grace period expired AND
2. All devices have acknowledged seeing the deletion
**Source:** [Cassandra Tombstones](https://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html)
---
## 5. Log Compaction & Garbage Collection
### 5.1 When to Compact
Operations become garbage when superseded by newer operations on the same entity:
```
Op 1: CREATE task-123 {title: "Buy milk"} → GARBAGE after Op 2
Op 2: UPDATE task-123 {title: "Buy groceries"} → GARBAGE after Op 3
Op 3: DELETE task-123 → LIVE (until tombstone expires)
```
**Source:** [Apache Geode Log Compaction](https://geode.apache.org/docs/guide/114/managing/disk_storage/compacting_disk_stores.html)
### 5.2 Compaction Strategies
**Live Key Identification (FASTER pattern):**
1. Scan segment for keys
2. Check if newer value exists in later segments
3. If no newer value, key is "live" - reinsert at tail
4. Delete segment after reinsertion
**Trigger Conditions:**
- Garbage ratio exceeds threshold (e.g., 50%)
- Total disk usage exceeds limit
- Time-based (e.g., daily)
**Source:** [Microsoft FASTER Compaction](https://github.com/microsoft/FASTER/issues/70)
### 5.3 Safe Compaction Rules
```typescript
interface CompactionConfig {
// Never delete operations that haven't been:
// 1. Acknowledged by all devices
// 2. Past the retention period
minRetentionDays: 90;
requireAllDevicesAcked: true;
// Snapshot before compacting
createSnapshotBeforeCompaction: true;
}
```
---
## 6. Real-Time Delivery
### 6.1 Figma LiveGraph Pattern
Figma uses PostgreSQL's Write-Ahead Log (WAL) for real-time updates:
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ PostgreSQL │─WAL─►│ Kafka │─────►│ LiveGraph │
│ (primary) │ │ │ │ (servers) │
└─────────────┘ └─────────────┘ └──────┬──────┘
│
WebSocket
│
┌───────▼───────┐
│ Clients │
└───────────────┘
```
**Key insight:** Rather than polling, subscribe to the database replication stream. This provides millisecond-level updates without polling overhead.
**Source:** [Figma LiveGraph](https://www.figma.com/blog/livegraph-real-time-data-fetching-at-figma/)
### 6.2 Poke Pattern (Replicache)
Instead of pushing full data, send lightweight "poke" hints:
```typescript
// Server → Client
interface Poke {
type: 'poke';
// No data payload - just a hint to pull
}
// Client receives poke → triggers pull
```
**Benefits:**
- Reduces server complexity (no need to track client state)
- Client always pulls full consistent view
- Works well with CDN caching
**Source:** [Replicache Poke Mechanism](https://doc.replicache.dev/concepts/how-it-works)
---
## 7. Offline Handling
### 7.1 Operation Queue Pattern
```typescript
interface OfflineQueue {
// Operations stored in IndexedDB
pendingOps: Operation[];
// Metadata
lastSyncedAt: number;
lastServerSeq: number;
// On reconnect
async flush(): Promise<void> {
for (const op of this.pendingOps) {
await this.push(op);
}
}
}
```
### 7.2 Extended Offline Recovery
When offline duration is long (days/weeks):
| Pending Ops | Duration | Recommended Action |
| ----------- | -------- | ------------------------------ |
| < 100 | < 1 day | Normal sync |
| 100-500 | 1-7 days | Show warning, proceed |
| 500-2000 | > 7 days | Offer recovery options |
| > 2000 | Any | Force snapshot upload/download |
**Source:** [Hasura Offline-First Guide](https://hasura.io/blog/design-guide-to-offline-first-apps)
---
## 8. Database & Infrastructure
### 8.1 PowerSync/ElectricSQL Pattern
Both use PostgreSQL logical replication:
```
┌─────────────┐ ┌─────────────┐
│ PostgreSQL │───logical repl────►│ Sync Server │
│ (source) │ │ │
└─────────────┘ └──────┬──────┘
│
Delta stream
│
┌──────▼──────┐
│ SQLite │
│ (client) │
└─────────────┘
```
**PowerSync approach:** Writes go through application backend (custom logic, validation)
**ElectricSQL approach:** Direct writes to Postgres with CRDT merge
**Source:** [PowerSync vs ElectricSQL](https://www.powersync.com/blog/electricsql-vs-powersync)
### 8.2 Scaling Considerations
From Figma LiveGraph 100x:
1. **Stateless API servers** - horizontal scaling
2. **Read-through cache** - sharded by query hash
3. **Query decomposition** - break complex queries into subqueries
4. **Invalidation routing** - probabilistic filters for efficient cache invalidation
**Source:** [Figma LiveGraph 100x](https://www.figma.com/blog/livegraph-real-time-data-at-scale/)
---
## 9. Recommendations for Super Productivity
Based on this research, key recommendations:
### Architecture
1. **Use server-authoritative pattern** with monotonic sequence numbers
2. **Implement push/pull protocol** (not just WebSocket streaming)
3. **Support offline queuing** with IndexedDB persistence
4. **Use poke pattern** for real-time hints (client pulls full state)
### Ordering & Causality
1. **Start with server-assigned sequences** (simpler than vector clocks)
2. **Keep vector clocks for conflict detection** between pending local ops
3. **Consider HLC** if we need human-readable timestamps with ordering
### Conflict Resolution
1. **Field-level strategies** (LWW for most, union for tags, sum for time)
2. **Server re-executes mutations** (not just stores them)
3. **Rebase pattern** for pending local changes
### Tombstones
1. **90-day retention** minimum
2. **Track device acknowledgments** before garbage collection
3. **Soft delete with tombstone table** (for audit trail)
### Compaction
1. **Snapshot before compacting**
2. **Only compact acknowledged operations**
3. **Trigger on garbage ratio** (50%) or time (daily)
---
## References
### Industry Sources
- [Figma LiveGraph](https://www.figma.com/blog/livegraph-real-time-data-fetching-at-figma/)
- [Figma Multiplayer](https://www.figma.com/blog/how-figmas-multiplayer-technology-works/)
- [Linear Sync Engine (Reverse Engineered)](https://github.com/wzhudev/reverse-linear-sync-engine)
- [Replicache Documentation](https://doc.replicache.dev/)
- [PowerSync](https://www.powersync.com/)
- [ElectricSQL](https://electric-sql.com/)
### Academic & Technical
- [Operation-Based CRDTs Protocol](https://www.bartoszsypytkowski.com/operation-based-crdts-protocol/)
- [Hybrid Logical Clocks](https://cse.buffalo.edu/tech-reports/2014-04.pdf)
- [Vector Clocks Explained](https://www.waitingforcode.com/big-data-algorithms/conflict-resolution-distributed-applications-vector-clocks/read)
### Design Guides
- [Hasura Offline-First Guide](https://hasura.io/blog/design-guide-to-offline-first-apps)
- [Akka Replicated Event Sourcing](https://doc.akka.io/libraries/akka-core/current/typed/replicated-eventsourcing.html)
- [Cassandra Tombstones](https://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html)

140
docs/sync-and-op-log/background-info/operation-log-sync-best-practices.md
View file

@ -1,140 +0,0 @@
# Best Practices for Operation Log Sync Server
This document outlines industry best practices for designing and implementing a robust, offline-first synchronization server based on an operation log (event sourcing) architecture.
## 1. Core Architectural Principles
### 1.1. Event Sourcing (The "Truth" is the Log)
Instead of mutating the current state directly, the system records every change as an immutable event (operation).
- **Benefit:** Provides a complete audit trail, allows time-travel debugging, and enables different state representations (projections) to be built from the same data.
- **Best Practice:** The server's primary responsibility is to persist this log in a strict linear order. The "current state" is merely a derivative of this log.
### 1.2. Centralized Serialization (The "Star" Topology)
In a client-server model, the server acts as the linearization point.
- **Logical Clocks:** The server assigns a strictly monotonic **Sequence Number** (or Global Revision ID) to every accepted operation.
- **Ordering:** This sequence number defines the absolute order of events. If two clients send operations simultaneously, the server arbitrarily processes one first, assigning it `Seq N`, and the second gets `Seq N+1`.
- **Simplicity:** This avoids the complexity of mesh-topology vector clocks for the global ordering, although clients may still use vector clocks for local branch management.
### 1.3. "Dumb" Server, "Smart" Client
For end-to-end encrypted or complex domain applications, the server should ideally treat operation payloads as opaque blobs (or just minimal validation).
- **Conflict Resolution:** Logic resides on the client. When a client downloads new operations from the server, it re-bases its local pending operations on top of them (similar to `git rebase`).
- **Validation:** Server validates authentication, authorization, and schema structure, but typically not business rules (which might depend on client-side state).
## 2. Data Storage & Database Choice
### 2.1. SQLite vs. PostgreSQL
| Feature | SQLite | PostgreSQL |
| :-------------- | :------------------------------ | :----------------------------------------- |
| **Use Case** | Personal, Self-Hosted, Embedded | Multi-tenant, Enterprise, High Concurrency |
| **Concurrency** | Single-writer (WAL mode helps) | MVCC (Multi-Version Concurrency Control) |
| **Maintenance** | Zero (single file) | Moderate (requires service management) |
- **Recommendation:** For a personal productivity tool (like Super Productivity), **SQLite** is the ideal default for self-hosting due to its zero-maintenance nature. However, the data access layer should use a query builder (e.g., Kysely, Knex) to support **PostgreSQL** for larger deployments or hosted services.
### 2.2. Immutable Log Storage
- **Append-Only:** The `operations` table should be effectively append-only.
- **Idempotency Keys:** Use client-generated UUIDs (v7 recommended for time-sorting) as the primary key to prevent duplicate processing of the same operation during network retries.
### 2.3. Tombstones
- **Deleting Data:** You cannot simply `DELETE` rows in an offline-first system, or other devices won't know an item was removed.
- **Implementation:** Create a `tombstones` table or a specific `DEL` operation type.
- **Cleanup:** Use a "Time-to-Live" (e.g., 90 days) after which tombstones are hard-deleted (the "horizon" for sync).
## 3. Synchronization Protocol
### 3.1. The Push-Pull Cycle
1. **Push:** Client sends a batch of `pending` operations.
- _Payload:_ `[Op1, Op2]`, `lastKnownServerSeq: 100`.
2. **Server Handling:**
- Checks strictly monotonic sequence.
- Assigns new `serverSeq` (101, 102).
- Stores ops.
- _Conflict Detection:_ If `lastKnownServerSeq` < `CurrentServerSeq` (e.g., 105), the server _could_ reject (Optimistic Locking) OR accept and let the client resolve later (Eventual Consistency). For "Dumb Server" models, accepting and letting the client merge is often smoother for UX.
3. **Pull:** Client requests "all ops since `serverSeq: 100`".
4. **Ack:** Client confirms it has processed ops up to `serverSeq: 102`.
### 3.2. Batching & Compression
- **Batching:** Always group operations. Overhead of HTTP/WS setup is high.
- **Compression:** Use `gzip` or `Brotli` for HTTP responses. For the payload itself, if JSON is verbose, consider a binary format (Protobuf/MsgPack) or just compressed JSON blobs if storage space is a concern.
### 3.3. Snapshotting (Bootstrapping)
Replaying 100,000 operations to build the initial state is slow.
- **Snapshots:** Periodically (e.g., every 1,000 ops), generating a full state snapshot.
- **Hybrid Sync:** New devices download the latest Snapshot + any Operations occurring _after_ that snapshot.
## 4. Client-Side Optimization: Coalescing & Batching
### 4.1. Write Coalescing (Squashing)
Coalescing is the practice of merging multiple granular updates to the same entity into a single operation _before_ it is synced to the server.
- **When to use:**
- **Rapid Text Entry:** If a user types "Hello" (5 ops), squash into 1 op ("Hello") after a debounce period (e.g., 500ms).
- **Slider/Drag Adjustments:** If a user drags a progress bar from 0% to 50%, typically only the final value (50%) matters.
- **Benefits:**
- Reduces server storage growth.
- Speeds up replay/initial sync for other clients.
- Reduces network overhead.
- **Risks:**
- **History Loss:** You lose the detailed audit trail of intermediate keystrokes. (Usually acceptable).
- **Conflict Complexity:** If two users edit the same field simultaneously, coalesced ops can make "Character-level" merging (OT/CRDT) harder if you are relying on simple LWW. But for field-level LWW, it is actually helpful.
### 4.2. Logical Batching
A "Batch" operation is a container for multiple distinct actions that should be treated atomically.
- **Use Case:** "Create Task" + "Add Tag" + "Move to Project X".
- **Implementation:** The client sends a single `BATCH` op containing an array of sub-operations.
- **Benefit:** Ensures database consistency. If the network fails, the task isn't created without its tag.
## 5. Conflict Resolution Strategies
### 5.1. Detection
- **Server-side:** "I received an update for Entity X based on v1, but I am already at v2."
- **Client-side:** "I downloaded Op A (ServerSeq 50) which modifies Task 1, but I have a local pending Op B that also modifies Task 1."
### 5.2. Resolution Models
1. **Last-Write-Wins (LWW):** Simple, robust, but data loss is possible. Uses wall-clock timestamps.
2. **Three-Way Merge:** If the payload is diff-able (e.g., JSON Patch), try to merge.
3. **Manual:** Flag the conflict and ask the user (complex UI).
4. **Hybrid:** LWW for simple fields (Title), Merge for collections (Tags), Append for logs (Time tracking).
## 6. Security & Reliability
### 6.1. Authentication
- **Token-Based:** JWTs are standard.
- **Scope:** Tokens should restrict access to specific `user_id` partitions.
### 6.2. Encryption
- **Transport:** HTTPS/WSS is mandatory.
- **At Rest:** If the server is untrusted, clients should encrypt the `payload` field before sending. The server syncs encrypted blobs. (End-to-End Encryption - E2EE).
### 6.3. Rate Limiting
- Essential to prevent a single malfunctioning client from flooding the log.
## 7. Summary Checklist
- [ ] **Database:** SQLite (Start) / Postgres (Scale).
- [ ] **ID Generation:** UUID v7 (Time-ordered).
- [ ] **Protocol:** HTTP (Reliable) + WebSocket (Notify).
- [ ] **Optimization:** Coalesce rapid local edits; Batch atomic actions.
- [ ] **Conflict:** Server assigns order; Client reconciles.
- [ ] **Safety:** Tombstones for deletions, Snapshots for speed.

341
docs/sync-and-op-log/background-info/synthesized-delta-sync-analysis.md
View file

@ -1,341 +0,0 @@
# Delta Sync Root-Cause Analysis: Synthesized Report
**Synthesized From:** Gemini 2.5 Flash, GPT-5, Claude Opus 4.5
**Date:** December 2, 2025
---
## Critical Note: Analysis Scope Discrepancy
**The three AI models analyzed different artifacts:**
| Model | What Was Analyzed | Branch/Source |
| -------------------- | ---------------------------------------- | ------------------------------------------ |
| **Gemini 2.5 Flash** | Current working directory | `feat/delta-sync` (45-line stub) |
| **GPT-5** | Documentation & design docs | Theoretical/planned design |
| **Claude Opus 4.5** | `feat/sync-server` branch via `git show` | 940-line implementation (different branch) |
**Current Reality (as of analysis date):**
- **`feat/delta-sync` branch** (the branch being evaluated): `super-sync.ts` is **45 lines** (WebDAV stub)
- `feat/sync-server` branch (separate, not under evaluation): Contains 940-line implementation
- Server (`packages/super-sync-server`): Auth wrapper only, no delta/changes logic
**Important Clarification:** The correct comparison should be between `feat/delta-sync` and `feat/operation-logs`. The `feat/sync-server` branch is a separate implementation effort and should not be conflated with `feat/delta-sync`.
**Implication:** Gemini's finding that the delta-sync implementation is "vaporware" is correct for `feat/delta-sync`. Opus's detailed code analysis applies to a different branch (`feat/sync-server`) which contains more code but is not the branch under evaluation. GPT-5's analysis is based on documented design.
**The strategic recommendation (abandon delta sync on `feat/delta-sync`) is valid**, because:
1. The `feat/delta-sync` branch has no real implementation—just a stub (Gemini)
2. The documented design has inherent architectural limitations (GPT-5)
3. Even fully-implemented delta sync (as in `feat/sync-server`) has fundamental issues (Opus)
---
## 1. Executive Summary
This document synthesizes analyses from three AI models (Gemini 2.5 Flash, GPT-5, Claude Opus 4.5) examining the delta-sync implementation in Super Productivity. All three models independently identified **fundamental architectural issues** that explain why stabilization has proven difficult—whether analyzing the stub, the design, or the implementation branch.
### Consensus Findings
| Finding | Gemini | GPT-5 | Opus | Confidence |
| ----------------------------------------- | :----: | :---: | :--: | :--------: |
| Shadow state is the core problem | ✅ | ✅ | ✅ | **High** |
| Watermark/revision tracking is unreliable | ✅ | ✅ | ✅ | **High** |
| LWW semantics cause data loss | ✅ | ✅ | ✅ | **High** |
| O(N) diffing doesn't scale | ✅ | ✅ | ✅ | **High** |
| Implementation is incomplete | ✅ | ✅ | ⚠️ | **High** |
| Multi-state consistency is impossible | ✅ | ✅ | ✅ | **High** |
**Key Insight (Consensus):** The delta-sync architecture requires maintaining multiple synchronized states (app data, shadow state, watermarks, vector clocks) without transactional guarantees. This creates a combinatorial explosion of failure modes that are difficult to test and reproduce.
---
## 2. Root Causes: Cross-Model Analysis
### 2.1 The Shadow State Problem
**All three models identify this as the primary root cause.**
#### Gemini's Analysis
> "The Delta Sync architecture requires the client to maintain a **Shadow State**—a perfect local copy of the data as it exists on the server. Without a durable shadow state, the client loses its 'diffing baseline' on every app restart."
#### GPT-5's Analysis
> "Shadow state is easily lost (IDB eviction, encryption key mismatch, cache cleared, new device). When missing, the client interprets 'no shadow' as 'everything changed,' triggering full-entity uploads."
#### Opus's Analysis
> "Dual-Cache Inconsistency: Shadow state exists both in-memory (`lastSyncedState` Map) and in IndexedDB. These can drift apart if app crashes after memory update but before IDB write."
#### Synthesis
| Failure Mode | Gemini | GPT-5 | Opus |
| ---------------------------- | ------ | ----- | ---- |
| IDB eviction/loss | ✅ | ✅ | ✅ |
| Encryption key mismatch | — | ✅ | ✅ |
| Memory/IDB drift | — | — | ✅ |
| No integrity verification | — | — | ✅ |
| Silent recovery masks issues | ✅ | ✅ | ✅ |
**Consolidated Root Cause:** Shadow state has **no durability guarantees**, **no integrity verification**, and **no atomic relationship** with watermarks or server state. Any corruption silently triggers full syncs, which may overwrite local changes.
---
### 2.2 The Watermark/Revision Problem
**All three models identify watermark drift as a critical issue.**
#### Gemini's Analysis
> "If Shadow State corrupts, the client is broken until full reset."
#### GPT-5's Analysis
> "If local watermarks drift (app restart during sync, partial writes), the client may ask the server for changes the server has already compacted or skip ranges entirely. That produces 'empty change set but stale shadow' scenarios."
#### Opus's Analysis
> "Watermark-Shadow Desynchronization: Watermark and shadow state are updated separately. If one update succeeds and the other fails, the client enters an inconsistent state."
#### Synthesis
| Failure Mode | Impact |
| --------------------------- | -------------------------------------------------- |
| Watermark newer than shadow | Client misses changes (thinks it's up-to-date) |
| Watermark older than shadow | Client re-downloads changes (inefficient but safe) |
| Watermark drift during sync | Client and server disagree on sync point |
| No atomic coupling | Crash mid-sync corrupts state permanently |
**Consolidated Root Cause:** Watermarks and shadow state are stored in separate IDB transactions. There is no mechanism to ensure they remain consistent after crashes or partial failures.
---
### 2.3 The LWW (Last-Write-Wins) Problem
**All three models identify shallow merge semantics as a data-loss risk.**
#### Gemini's Analysis
> "'Last Write Wins' blindly overwrites data. No context of _why_ a change happened. A `TaskCompleted` op and `TaskRenamed` op cannot both be applied—one wipes out the other."
#### GPT-5's Analysis
> "Server applies `merged = { ...oldData, ...newData }`, so correctness depends on clients sending full values for every top-level property. When it misses a nested field, the server overwrites the old object with the partial payload, dropping untouched keys."
#### Opus's Analysis
> "BLOB models use last-write-wins (whole object replacement), while ENTITY models use field-level merging. If mode detection is wrong, data is either over-merged or under-merged."
#### Synthesis
```
Scenario: Two devices edit Task A concurrently
Device 1: Renames task to "Important Meeting"
Device 2: Marks task as completed
Delta Sync Result (LWW):
- If Device 2 syncs last → Task renamed but NOT completed (Device 1's change lost)
- If Device 1 syncs last → Task completed but NOT renamed (Device 2's change lost)
Expected Result:
- Task should be BOTH renamed AND completed (independent changes)
```
**Consolidated Root Cause:** Delta sync transmits state diffs, not intent. When two devices modify different properties of the same entity, only one change survives. This is a fundamental limitation of state-based sync with shallow merge.
---
### 2.4 The Performance Problem
**All three models identify O(N) diffing as a scalability bottleneck.**
#### Gemini's Analysis
> "Diffing 10k items freezes UI. Performance degrades linearly with data size."
#### GPT-5's Analysis
> "For thousands of tasks, diffing pegs the UI thread, causing frame drops and timeouts. If data changes mid-diff, the computed delta no longer matches the final state."
#### Opus's Analysis
> "Diff calculation (`createDiff`) is O(N) where N = number of entities. For users with 10,000+ tasks, this can block the main thread."
#### Synthesis
| Dataset Size | Diff Time (estimated) | User Impact |
| ------------ | --------------------- | ------------- |
| 100 tasks | ~10ms | Imperceptible |
| 1,000 tasks | ~100ms | Minor delay |
| 10,000 tasks | ~1,000ms | UI freeze |
| 50,000 tasks | ~5,000ms | Unusable |
**Consolidated Root Cause:** The diff algorithm must compare every entity against its shadow counterpart using `JSON.stringify()`. This is inherently O(N) and cannot be optimized without fundamental architecture changes (dirty tracking, incremental diffing, web workers).
---
### 2.5 The Implementation Completeness Problem
**Gemini and GPT-5 note that the implementation is incomplete. Opus analyzed the more complete `feat/sync-server` branch.**
#### Gemini's Analysis (Most Critical)
> "Code analysis reveals the Server has **NO** delta DB (only Users table) and Client is just a WebDAV wrapper. The implementation is effectively 'vaporware'."
#### GPT-5's Analysis
> "The actual code path (`super-sync.ts`) is currently just a thin WebDAV wrapper—none of the documented delta logic is wired in."
#### Opus's Analysis
> "The `feat/sync-server` branch contains a 940-line `SuperSyncProvider` with IDB shadow state, watermarks, and diff logic. However, multiple TODOs indicate incomplete areas."
#### Synthesis
| Component | Current State | Gap |
| ------------------- | --------------------------------- | ----------------------------------------- |
| Server delta API | Documented but minimal | No changes table in Gemini's analysis |
| Client shadow state | Implemented in `feat/sync-server` | No crash consistency |
| Diff engine | Implemented | No worker offload, O(N) |
| Watermark tracking | Implemented | Not atomic with shadow |
| Encryption | Two modes implemented | Mode switching is implicit |
| Vector clocks | Implemented | Governance gaps (empty clocks = conflict) |
**Note:** The discrepancy between Gemini/GPT-5 and Opus suggests the implementation has evolved. Opus analyzed a more recent state of `feat/sync-server` where delta logic exists but has fundamental issues.
---
## 3. Why Stabilization is Difficult: Consensus View
All three models agree that stabilization faces structural barriers:
### 3.1 State-Derived vs Intent-Derived (GPT-5)
> "The system attempts to infer intent from mutable snapshots, so any shadow corruption forces expensive recomputation and can silently lose semantics."
### 3.2 No Single Source of Truth (All Models)
| State Component | Storage | Can Drift? |
| ---------------------- | ------- | :--------: |
| NgRx in-memory state | Memory | Yes |
| IndexedDB app data | IDB | Yes |
| Shadow state (memory) | Memory | Yes |
| Shadow state (IDB) | IDB | Yes |
| Watermarks | IDB | Yes |
| Server state | Remote | Yes |
| Vector clocks (local) | IDB | Yes |
| Vector clocks (remote) | Remote | Yes |
**Total independent states: 8**
**Possible inconsistent combinations: 2^8 - 1 = 255**
### 3.3 Testing Complexity (All Models)
| Edge Case | Testable? | Currently Tested? |
| -------------------------------- | :-------: | :---------------: |
| Concurrent pushes from 2 devices | Difficult | ❌ |
| Network failure mid-sync | Difficult | ❌ |
| IDB eviction | Difficult | ❌ |
| Vector clock overflow | Medium | ⚠️ (disabled) |
| Encryption key change | Medium | ❌ |
| Large dataset (10k+ tasks) | Difficult | ❌ |
### 3.4 Operational Surface Area (GPT-5)
> "The design spans WebDAV fallbacks, optional encryption, IndexedDB persistence, and REST deltas. Each layer introduces its own failure modes, and they compound."
---
## 4. Model-Specific Unique Insights
### 4.1 Gemini's Unique Insight: "Phase 0" Status
Gemini noted that the `SuperSyncProvider` is currently "Phase 0"—a WebDAV wrapper—indicating the delta sync was never fully implemented. This suggests the project hit a complexity wall during implementation.
### 4.2 GPT-5's Unique Insight: Fallback Path Coupling
> "Switching modes changes merge semantics (snapshot LWW vs. delta patches). After fallback, the shadow state no longer matches server revisions."
This highlights that having two sync modes (delta and WebDAV fallback) creates hybrid states that neither path fully owns.
### 4.3 Opus's Unique Insight: Vector Clock Governance Gaps
Opus identified specific code-level issues in vector clock handling:
- Empty clocks treated as `CONCURRENT` (triggers false conflicts)
- Pruning at 50 clients loses causality history
- Overflow reset to 1 corrupts comparison with old clocks
---
## 5. Estimated Effort to Stabilize
### Gemini's Estimate: 12-16 Weeks (Build from Scratch)
> "Server (4-6 wks): Design changes schema, implement `/api/sync` endpoints. Client (6-8 wks): Implement `ShadowStateStore`, Diff Engine (Worker), Partial Patching. Migration (2 wks)."
### GPT-5's Estimate: 3-5 Weeks (Make It Real)
> "Implement and persist shadow state + per-model watermarks with crash-safe coupling. Add diff+merge pipeline. Add large-dataset perf tests. Harden with soak tests."
### Opus's Estimate: Architectural Rewrite Required
> "Stabilization would require either deep architectural changes to make state transitions atomic and verifiable, OR switching to a different synchronization paradigm."
### Synthesis
| Approach | Effort | Risk | Outcome |
| -------------------------- | :---------: | :----: | ----------------------------- |
| Minimal fixes (bug-by-bug) | 2-4 weeks | High | Whack-a-mole; new bugs emerge |
| Proper delta sync (GPT-5) | 3-5 weeks | Medium | Viable but fragile |
| Full rebuild (Gemini) | 12-16 weeks | Medium | Proper delta sync |
| Architecture switch (Opus) | 4-6 weeks | Low | Operation log approach |
---
## 6. Consolidated Recommendations
### 6.1 Do Not Attempt
- **Bug-by-bug fixes:** The issues are architectural, not isolated bugs
- **Adding more fallback paths:** Increases state space complexity
- **Optimizing diff performance first:** Doesn't address correctness issues
### 6.2 Consider
- **Per-entity versioning without full operation log:** Simpler than operation log, addresses some issues (see `operationlog-critique.md`)
- **Hybrid approach:** Use delta sync for simple models, operation log for complex ones
### 6.3 Recommended
- **Complete the operation log implementation:** All three models agree this is more tractable
- **If delta sync must be used:** Follow GPT-5's plan with crash-safe coupling between shadow and watermarks
---
## 7. Conclusion
**All three AI models independently reached the same conclusion:** The delta-sync implementation has fundamental architectural issues that make stabilization expensive and risky.
### Core Problems (Unanimous)
1. **Shadow state has no durability or integrity guarantees**
2. **Watermarks and shadow state can desynchronize**
3. **LWW merge semantics lose concurrent independent changes**
4. **O(N) diffing doesn't scale to large datasets**
5. **Multiple independent states create combinatorial failure modes**
### Path Forward (Consensus)
The operation-log approach (`feat/operation-logs`) provides a more tractable path because:
- Single source of truth (operation log) eliminates multi-state consistency issues
- Per-entity conflict detection is granular (not whole-file)
- Append-only logs are inherently more robust to corruption
- Performance scales with change frequency, not dataset size
The delta-sync approach is not inherently wrong, but the current implementation would require significant rework to achieve stability—effort that may be better spent completing the operation-log implementation.

285
docs/sync-and-op-log/background-info/synthesized-delta-vs-oplog.md
View file

@ -1,285 +0,0 @@
# Delta Sync vs Operation Log: Synthesized Comparison
**Synthesized From:** Gemini 2.5 Flash, GPT-5, Claude Opus 4.5
**Date:** December 2, 2025
**Branches Compared:** `feat/delta-sync` vs `feat/operation-logs`
---
## Implementation Status
**Both approaches have substantial implementations.**
### Current Implementation Status
| Branch | Key Files | Lines | Status |
| --------------------- | ----------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------- |
| `feat/delta-sync` | `super-sync.ts`, `diff-utils.ts`, `super-sync-api.ts` | ~1,600 | Full implementation: shadow state (IDB), diffing, watermarks, granular encryption |
| `feat/operation-logs` | `operation-log-store.service.ts`, `operation-log-sync.service.ts`, etc. | ~600+ | Core logic implemented: op store, sync service, effects, hydrator |
### Analysis Sources
| Model | Delta Sync Source | Notes |
| ---------- | ------------------------------------------- | ----------------------------------------------------------- |
| **Gemini** | Earlier state of `feat/delta-sync` | Analyzed before branch was updated with full implementation |
| **GPT-5** | Design docs + code | Analyzed architectural design |
| **Opus** | `feat/delta-sync` (940-line implementation) | Full code analysis of implemented delta sync |
**Update:** Gemini's finding that delta sync was "vaporware" is now outdated. The `feat/delta-sync` branch has been updated and now contains the full 940-line `SuperSyncProvider` implementation with IDB shadow state, diff engine, and watermark tracking. Both approaches can now be compared on equal footing as implemented code.
---
## 1. Executive Summary
This document compares two synchronization **architectures** for Super Productivity, both of which are now substantially implemented:
- **Delta Sync:** State-based synchronization with shadow state, diffing, and watermarks
- **Operation Log:** Event-sourced synchronization with append-only operation log
### Approaches Compared
| Approach | Branch | Core Principle | Implementation |
| ----------------- | --------------------- | ---------------------------------------------- | --------------------------------------- |
| **Delta Sync** | `feat/delta-sync` | Compare state snapshots, transmit differences | **Implemented** (~1,600 lines) |
| **Operation Log** | `feat/operation-logs` | Record actions, transmit and replay operations | **Partially implemented** (~600+ lines) |
### Verdict (Unanimous)
All three models recommend **Operation Log** based on architectural analysis:
| Model | Rationale |
| ------ | ------------------------------------------------------------------------------- |
| Gemini | Delta sync has fundamental shadow state consistency issues |
| GPT-5 | Delta sync design has inherent LWW data-loss risk |
| Opus | Code analysis reveals non-atomic state updates and vector clock governance gaps |
---
## 2. Architecture Comparison
### 2.1 Delta Sync (As Implemented)
```
Client Storage (super-sync.ts):
├── App data (IndexedDB)
├── Shadow state (super-sync-shadow IDB)
│ ├── shadow_state: last synced state per model
│ └── watermarks: revision cursors per model
└── Memory cache (lastSyncedState Map)
Sync Flow:
1. Load shadow state from IDB (or memory cache)
2. Compute diff via createDiff() in diff-utils.ts
3. Upload changes to /api/sync/changes
4. Server shallow-merges (LWW)
5. Update shadow + watermark in IDB
```
**Code-Verified Concerns (from Opus analysis of super-sync.ts):**
- Shadow state exists in both memory and IDB (can drift)
- Watermark and shadow saved in separate IDB transactions (not atomic)
- Diff computation is O(N) using JSON.stringify comparison
- LWW merge loses concurrent independent changes
- Empty vector clocks treated as CONCURRENT (false conflicts)
### 2.2 Operation Log (As Implemented)
```
Client Storage (actual):
├── SUP_OPS IndexedDB
│ ├── ops: append-only operation log
│ └── state_cache: periodic snapshots
└── NgRx Store (derived from ops)
Sync Flow:
1. Local action → append to ops log
2. Upload pending ops to remote
3. Download remote ops
4. Detect conflicts via vector clock comparison
5. Apply non-conflicting ops; surface conflicts to user
```
**Code-Verified Characteristics:**
- Single source of truth (operation log)
- Conflict detection is per-entity
- Compaction service bounds log growth
- Replay hydration from snapshot + tail ops
---
## 3. Criteria Comparison
### 3.1 Conflict Handling
This is the most significant differentiator and is **architecture-dependent, not implementation-dependent**.
| Aspect | Delta Sync (Design) | Operation Log (Implemented) |
| -------------- | -------------------------------- | ------------------------------------- |
| Detection | Whole-file vector clock | Per-entity vector clock |
| Merge | Shallow LWW (`{...old, ...new}`) | Semantic (independent ops both apply) |
| User choice | All-or-nothing | Per-entity granular |
| Data loss risk | **High** (silent overwrites) | **Low** (conflicts surfaced) |
**Example:**
```
Device A: Renames task to "Meeting"
Device B: Marks task complete
Delta Sync (LWW): One change lost
Operation Log: Both changes applied (independent operations)
```
**Winner: Operation Log** (unanimous, strong confidence)
### 3.2 Maintainability
| Aspect | Delta Sync (Design) | Operation Log (Implemented) |
| --------------- | ----------------------------------- | --------------------------- |
| Data models | 2 (app state + shadow) | 1 (operation log) |
| Adding features | Update shadow handling + diff rules | Add action to whitelist |
| Debugging | Compare shadow vs actual vs server | Inspect operation sequence |
| Source of truth | Ambiguous (shadow can drift) | Clear (operation log) |
**Winner: Operation Log** (unanimous)
### 3.3 Performance
| Metric | Delta Sync | Operation Log | Basis |
| -------------- | ------------------------------- | ----------------------- | ------------------------------------------------------- |
| Diff cost | O(N) per sync | O(1) append | **Code-verified** (diff-utils.ts iterates all entities) |
| Startup | Fast (load state) | Snapshot + replay tail | **Code-verified** |
| Bandwidth | Smaller patches | Larger payloads | **Code-verified** (delta sends only changed fields) |
| Large datasets | UI freeze risk at 10k+ entities | Scales with change rate | **Needs measurement** |
**Winner: Tie** — Delta sync wins on bandwidth; operation log wins on CPU. Needs benchmarking with real datasets.
### 3.4 Disk Usage
| Metric | Delta Sync (Implemented) | Operation Log (Implemented) |
| ------------ | ----------------------------------------- | --------------------------------------------- |
| Steady-state | ~2X (shadow state in IDB) | ~1.2X (with compaction) |
| Basis | **Code-verified** (super-sync-shadow IDB) | **Code-verified** (compaction service exists) |
**Winner: Operation Log** (if compaction works correctly)
---
## 4. Implementation Effort
### Delta Sync: Stabilize Existing Implementation
`feat/delta-sync` now has full implementation (~1,600 lines). Remaining work to stabilize:
| Component | Effort | Notes |
| ------------------------------- | --------- | ---------------------------------------------- |
| Atomic shadow/watermark updates | 1-2 weeks | Currently separate IDB transactions |
| Web worker for diff | 1 week | Prevent UI freeze on large datasets |
| Server delta API hardening | 1-2 weeks | Pagination, error handling |
| Conflict UI improvements | 1 week | Auto-merge feedback |
| Testing/hardening | 2-4 weeks | Edge cases, concurrent clients, crash recovery |
**Total: 6-10 weeks** (high risk due to multi-state consistency issues identified in code analysis)
### Operation Log: Complete Implementation
`feat/operation-logs` has core logic (~600+ lines); remaining work:
| Component | Effort | Notes |
| ---------------------------- | ---------- | --------------------------------------- |
| Conflict resolution UI | 1-2 weeks | Currently stub |
| Dependency resolver retry | 0.5-1 week | Handle missing parents |
| Smart resolution suggestions | 1 week | LWW fallback for simple conflicts |
| Genesis migration | 1 week | Convert existing data to first op |
| Effect guards | 0.5-1 week | Prevent side effects during replay |
| Testing/hardening | 1-2 weeks | Compaction, large logs, concurrent tabs |
**Total: 4-6 weeks** (medium risk, primarily compaction correctness)
---
## 5. How to Validate Claims
Before committing to either approach, measure:
### For Delta Sync (if implemented)
1. **Diff cost:** Time `createDiff()` with 1k, 10k, 50k entities
2. **Shadow storage:** Measure IDB size with shadow vs without
3. **Watermark consistency:** Test crash recovery mid-sync
4. **LWW data loss:** Simulate concurrent edits, count lost changes
### For Operation Log
1. **Log growth:** Measure ops/day for typical usage; verify compaction bounds it
2. **Replay time:** Time startup with 1k, 10k, 100k operations
3. **Compaction correctness:** Verify no data loss after compaction
4. **Conflict detection accuracy:** Test concurrent edits, verify correct conflicts surfaced
---
## 6. Risk Assessment
### Delta Sync Risks (Code-Verified)
| Risk | Likelihood | Impact | Notes |
| ----------------------- | :--------: | :----: | -------------------------------------------- |
| Shadow/watermark desync | High | High | Non-atomic IDB transactions in super-sync.ts |
| LWW data loss | High | High | Fundamental to architecture |
| O(N) UI freeze | Medium | Medium | Can be mitigated with web workers |
| Vector clock governance | Medium | Medium | Empty clocks = false conflicts |
### Operation Log Risks (Observed)
| Risk | Likelihood | Impact | Notes |
| ---------------------- | :--------: | :----: | ------------------------------------ |
| Compaction data loss | Low | High | Mitigate with conservative retention |
| Replay non-determinism | Medium | Medium | Requires reducer purity |
| Log growth unbounded | Medium | Low | Compaction service exists |
**Assessment:** Operation log has fewer high-likelihood, high-impact risks.
---
## 7. Recommendation
### Consensus
**Complete the Operation Log implementation** (`feat/operation-logs`).
### Rationale
1. **Both implementations exist**, but operation log has fewer code-verified architectural risks
2. **Conflict handling is architecturally superior** in operation log (per-entity vs whole-file)
3. **Risk profile is better** for operation log (no multi-state consistency issues)
4. **Effort is comparable**: Delta sync 6-10 weeks vs Operation log 4-6 weeks
### If Delta Sync Is Chosen
1. Address atomic shadow/watermark updates first (critical bug)
2. Move diff calculation to web worker (performance)
3. Fix vector clock governance (empty clocks = false conflicts)
4. Accept and document LWW limitations
5. Plan for extensive testing of crash recovery scenarios
---
## 8. Conclusion
| Aspect | Delta Sync | Operation Log |
| --------------------- | ------------------------------ | ----------------------------------- |
| Implementation status | **Implemented** (~1,600 lines) | Partially implemented (~600+ lines) |
| Conflict handling | LWW (lossy) | Per-entity (preserves intent) |
| Effort to production | 6-10 weeks | 4-6 weeks |
| Risk level | High (multi-state consistency) | Medium (compaction correctness) |
| Recommendation | Viable but risky | **Recommended** |
Both approaches are now substantially implemented. The recommendation for operation log is based on **code-verified architectural concerns** in the delta sync implementation:
- Non-atomic shadow/watermark updates create crash consistency risks
- LWW merge semantics cause data loss for concurrent independent changes
- Empty vector clocks trigger false conflicts
**Bottom line:** Operation log provides a more robust architecture for Super Productivity's multi-device use case. If delta sync is chosen, significant stabilization work is needed to address the identified issues.

179
docs/sync-and-op-log/background-info/vector-clock-history-and-alternatives.md
View file

@ -1,179 +0,0 @@
# Vector Clock Pruning: Literature Review & Best Practices
Research compiled Feb 2026 to validate and contextualize the pruning strategy used in Super Productivity's sync system.
---
## 1. The Core Problem
Vector clocks grow linearly with the number of participating clients. In a system where users may use 10–20+ devices over time, unbounded clocks waste storage and bandwidth. Pruning reduces clock size at the cost of **false concurrency** — pruned entries can make two causally ordered events appear concurrent.
**Key insight from all sources:** False concurrency is safe (creates conflicts to resolve), while false ordering (incorrectly declaring one clock greater than another) causes silent data loss.
---
## 2. Pruning Strategies in the Literature
### 2.1 Amazon Dynamo (2007) — Size-Based Pruning
- **Strategy:** Drop the oldest entry (by wall-clock timestamp) when the clock exceeds a threshold (~10 entries).
- **Trade-off:** Simple but relies on timestamps for "oldest" determination, which can be inaccurate across nodes.
- **Source:** DeCandia et al., "Dynamo: Amazon's Highly Available Key-value Store," SOSP 2007, Section 4.4.
### 2.2 Riak (Pre-2.0) — Hybrid Size + Time Pruning
- **Strategy:** Four tunable parameters control pruning:
- `small_vclock` / `big_vclock` — size thresholds
- `young_vclock` / `old_vclock` — age thresholds
- **Behavior:**
- Clocks smaller than `small_vclock` are never pruned
- Clocks younger than `young_vclock` are never pruned
- Clocks older than `old_vclock` are aggressively pruned
- Clocks larger than `big_vclock` are pruned regardless of age
- **Known bug (Riak #613):** Pruning before comparison caused siblings to accumulate indefinitely. The fix was identical to ours: prune after comparison, before storage.
- **Source:** Basho Riak documentation; GitHub issue basho/riak_kv#613.
### 2.3 Voldemort (LinkedIn) — Activity-Based Pruning
- **Strategy:** Strip entries for nodes/replicas that have been retired or are no longer active participants.
- **Rationale:** In a fixed-replica system, only active replicas need clock entries.
- **Not applicable to Super Productivity:** Our "replicas" are user devices that come and go unpredictably.
- **Source:** Project Voldemort documentation and source code.
### 2.4 Causal Stability (CRDTs) — Theoretical Approach
- **Strategy:** Compute the component-wise minimum across all replicas' known clocks (the "stable cut"). Events below this threshold are causally stable — all replicas have seen them — and their clock entries can be safely garbage collected.
- **Requirement:** Requires periodic metadata exchange between all replicas to compute the stable cut.
- **Trade-off:** No false concurrency (exact GC), but requires all-to-all communication.
- **Not practical for Super Productivity:** Devices are intermittently connected; computing a stable cut requires all clients to be reachable.
- **Source:** Almeida et al., "Scalable and Accurate Causality Tracking for Eventually Consistent Stores," DISC 2014.
---
## 3. Alternatives That Avoid Pruning Entirely
### 3.1 Dotted Version Vectors (Riak 2.0+)
- **Concept:** Bounds clock size to the number of **replicas** (not clients). Each entry tracks a replica's latest event plus a "dot" (single event marker) for the specific write.
- **Benefit:** Clock size = number of replicas, which is fixed and small (typically 3–5).
- **Limitation:** Requires a fixed, known replica set. Not directly applicable when every user device is a "replica."
- **Source:** Preguiça et al., "Dotted Version Vectors: Logical Clocks for Optimistic Replication," arXiv:1011.5808.
### 3.2 Interval Tree Clocks
- **Concept:** Uses a tree structure that dynamically adapts to the set of active participants. Nodes can fork (new participant) and join (participant retires) without growing unboundedly.
- **Benefit:** Naturally adapts to changing participant sets without pruning.
- **Limitation:** More complex implementation; not widely adopted in production systems.
- **Source:** Almeida et al., "Interval Tree Clocks: A Logical Clock for Dynamic Systems," OPODIS 2008.
### 3.3 Last-Writer-Wins with Timestamps (Cassandra)
- **Concept:** Abandons vector clocks entirely. Uses wall-clock timestamps for conflict resolution — latest timestamp wins.
- **Benefit:** O(1) metadata per entry. No clock growth problem.
- **Limitation:** Clock skew can cause silent data loss. No true conflict detection — concurrent writes are silently resolved by timestamp.
- **Why not for Super Productivity:** User devices have unreliable clocks; silent data loss is unacceptable for a personal productivity app.
---
## 4. The Critical Rule: Compare Before Pruning
This is the single most important finding, confirmed across multiple sources:
> **Never prune a vector clock before using it in a comparison.**
### Why
Pruning removes information. If clock A has entries `{X:1, Y:2, Z:3}` and you prune Z before comparing against `{X:1, Y:2, Z:3}`, clock A appears to lack knowledge of Z — making the comparison return CONCURRENT instead of EQUAL.
### Historical Bugs
1. **Riak #613:** Pruning before comparison caused "sibling explosion" — objects accumulated hundreds of siblings that could never be resolved because the pruned clocks always appeared concurrent.
2. **Super Productivity (Feb 2026):** Server pruning before comparison caused an infinite rejection loop when MAX was 10. Client K merges all clocks + its own ID (11 entries), server prunes to 10, non-shared keys cause CONCURRENT, server rejects, client re-merges, loop repeats. Fixed by increasing MAX to 20 and moving pruning to after comparison.
### The Fix (Both Systems)
```
1. Receive full clock from client
2. Compare full (unpruned) clock against stored clock
3. If accepted: prune THEN store
4. If rejected: return rejection with stored clock for client-side resolution
```
---
## 5. Pruning-Aware Comparison
When both clocks have been pruned, standard comparison is unreliable because missing entries could mean either "never knew about this client" or "entry was pruned." Two approaches exist:
### 5.1 Conservative (Super Productivity's approach)
When both clocks are at MAX size:
- Only compare shared keys
- If non-shared keys exist on the losing side, return CONCURRENT
- Rationale: Non-shared keys might represent unknown causal history
### 5.2 Riak's Approach (Pre-2.0)
- Treat missing entries in pruned clocks as 0 (standard comparison)
- Accept the resulting false concurrency as siblings
- Resolve siblings on read via application-level merge
### Trade-off
Super Productivity's conservative approach generates more conflicts but never produces false ordering. Riak's approach generates fewer conflicts but requires a robust sibling merge mechanism.
---
## 6. Validation of Super Productivity's Design
The project's current architecture uses a simple 2+1 layer approach:
| Layer | Current Mechanism | Precedent |
| --------------------------------- | -------------------------------------------------------- | ------------------------------------- |
| 1. Server prunes after comparison | Full clock comparison, prune before storage | Dynamo, Riak (post-#613 fix) |
| 2. Same-client check | Monotonic counter comparison for import client's own ops | Novel — always mathematically correct |
### What the Literature Validates
- Size-based pruning with a fixed MAX is the standard approach (Dynamo, Riak)
- Compare-before-prune is essential (Riak #613, Dynamo best practices)
- False concurrency from pruning is the safe direction (all sources agree)
- Fixing root causes (too-small MAX) over adding defense layers is standard engineering practice
### What's Novel to This Project
- The same-client check — leveraging monotonic counters for definitive post-import detection (always mathematically correct regardless of MAX size)
### Why MAX=20 Simplified Everything
The original 4-layer defense (protected client IDs, pruning-aware comparison, `isLikelyPruningArtifact`, same-client check) was designed to work around a root cause: MAX=10 was too small, making pruning a frequent occurrence that interacted badly with SYNC_IMPORT operations. Commit `d70f18a94d` increased MAX from 10 to 30, which was later reduced to 20 (a 20-entry clock is ~333 bytes — negligible overhead), and removed the defense layers that were treating symptoms rather than the cause. With MAX=20, pruning requires 21+ unique client IDs — an extremely rare scenario for a personal productivity app. The `isLikelyPruningArtifact` heuristic was also removed since it had known false positives and was unnecessary at MAX=20. Only the same-client check remains as a safety net — it's always mathematically correct (monotonic counters are definitive).
---
## 7. Potential Future Improvements
### 7.1 Dotted Version Vectors (If Server Becomes the Authority)
If the architecture evolves toward a server-centric model where the server is the primary coordinator (rather than a relay), Dotted Version Vectors could bound clock size to the number of active server "vnodes" rather than client devices.
### 7.2 Client Registration with Bounded IDs
Assign clients numeric IDs from a small, bounded set (e.g., 0–15). When a client retires, its ID can be reclaimed. This bounds clock size without pruning but requires a registration/retirement protocol.
### 7.3 Periodic Stable-Cut GC
If clients periodically report their known clock state to the server, the server could compute a stable cut and notify clients which entries are safe to GC. This eliminates false concurrency from pruning but requires all-to-all communication.
---
## 8. Sources
1. DeCandia, G. et al. (2007). "Dynamo: Amazon's Highly Available Key-value Store." SOSP '07.
2. Almeida, P. S. et al. (2008). "Interval Tree Clocks: A Logical Clock for Dynamic Systems." OPODIS '08.
3. Preguiça, N. et al. (2010). "Dotted Version Vectors: Logical Clocks for Optimistic Replication." arXiv:1011.5808.
4. Almeida, P. S. et al. (2014). "Scalable and Accurate Causality Tracking for Eventually Consistent Stores." DISC '14.
5. Basho Riak documentation — Vector Clocks, Dotted Version Vectors.
6. Basho/riak_kv GitHub issue #613 — Sibling explosion from pre-comparison pruning.
7. Project Voldemort documentation — Versioning and conflict resolution.
8. Apache Cassandra documentation — Timestamps and conflict resolution.

106
docs/sync-and-op-log/contributor-sync-model.md Normal file
View file

@ -0,0 +1,106 @@
# The Contributor Sync Model
**The one thing to understand before writing any effect, reducer, or bulk
dispatch that touches synced state.**
Super Productivity syncs by replaying an operation log. Almost every sync
correctness rule you will hit is a facet of a **single invariant**:
> ## One user intent = exactly one operation. Replayed and remote operations must never re-trigger effects.
Reducers **must** run for remote/replayed operations (that is how state is
rebuilt). Effects **must not** — the UI side effect (snack, sound, navigation)
already happened on the originating client, and any cascading change is already
its own entry in the operation log. Re-running effects on replay duplicates side
effects and emits phantom operations that conflict with sync.
Everything below is that invariant applied at three points.
---
## Boundary 1 — The action boundary
**Effects inject `LOCAL_ACTIONS`, never `inject(Actions)`.**
`LOCAL_ACTIONS` is the standard actions stream with `meta.isRemote` filtered
out (`src/app/util/local-actions.token.ts`). Remote/replayed operations are
applied as one `bulkApplyOperations` action; `LOCAL_ACTIONS` ensures your effect
only sees genuine local user intent.
- Default for **all** effects: `private _actions$ = inject(LOCAL_ACTIONS);`
- The only legitimate exception uses `ALL_ACTIONS` and handles `isRemote`
itself: `operation-log.effects.ts` (captures/persists every action). You are
almost certainly not adding a second.
- Remote **archive** side effects are _not_ an `ALL_ACTIONS` case:
`archive-operation-handler.effects.ts` itself uses `LOCAL_ACTIONS`; the
remote-client archive writes/deletes are driven separately by
`OperationApplierService` → `ArchiveOperationHandler`.
✅ **Enforced by `local-rules/no-actions-in-effects`** — you cannot get this
wrong; the linter rejects `inject(Actions)` / `Actions` imports in
`*.effects.ts`.
## Boundary 2 — The selector boundary
**Selector-driven effects must guard with `skipDuringSyncWindow()`.**
An effect that reacts to a _selector_ (store state) instead of a specific
_action_ bypasses Boundary 1 entirely — it fires on every store change,
including hydration and sync replay. Two timing gaps (initial startup before
first sync; the post-sync re-evaluation window) make such effects emit
operations with stale vector clocks that immediately conflict.
- Use `skipDuringSyncWindow()` for selector-based effects that modify
frequently-synced entities or perform "repair"/"consistency" work.
- The narrower `skipWhileApplyingRemoteOps()` /
`HydrationStateService.isApplyingRemoteOps()` exist for finer control.
- **Prefer action-based effects.** A selector-based effect is the
intuitive-but-usually-wrong choice; reach for it only when there is no
action to key off.
✅ **Enforced by `local-rules/require-hydration-guard`** (existing rule).
## The atomicity rule — one intent, one op
**Multi-entity changes are meta-reducers, not effects. Bulk dispatch loops yield.**
- A change that touches more than one entity for a single user intent (e.g.
deleting a tag also removing it from every task) must be **one reducer pass**
so it becomes **one operation**. Put it in
`src/app/root-store/meta/task-shared-meta-reducers/`, not in an effect that
dispatches a fan-out of follow-up actions. An effect-based fan-out emits N
operations for one intent _and_ re-runs on replay (a restatement of Boundary 1).
- `store.dispatch()` is non-blocking. After a loop of 50+ dispatches, add
`await new Promise((r) => setTimeout(r, 0))` so captured operations don't
lose intermediate state.
⚠️ `local-rules/no-multi-entity-effect` (`warn`) flags this heuristically — it
catches the array-literal fan-out shape (`map(() => [a(), b()])`), not every
multi-entity dispatch (e.g. a `of(a(), b())` varargs fan-out slips past). The
blessed pattern is a `task-shared-meta-reducers/` reducer.
---
## Decision table — "I'm writing an effect"
| Question | Answer | Linter |
| ----------------------------------------------------- | --------------------------------------------------------- | ------------------------------------ |
| Does it inject the actions stream? | Use `LOCAL_ACTIONS` (not `Actions`) | ✅ `no-actions-in-effects` (error) |
| Does it react to a **selector** instead of an action? | Add `skipDuringSyncWindow()` | ✅ `require-hydration-guard` (error) |
| Does one user intent change **>1 entity**? | Make it a meta-reducer, not an effect | ⚠️ `no-multi-entity-effect` (warn) |
| Does it dispatch in a **loop of 50+**? | `await new Promise(r => setTimeout(r, 0))` after the loop | — (convention) |
Two of the three are mechanically enforced — you do not need to memorize them,
only understand _why_ (the invariant at the top).
---
## Why (deeper)
- **Mechanism & rules:** [`operation-rules.md`](./operation-rules.md)
- **Architecture:** [`operation-log-architecture.md`](./operation-log-architecture.md)
- **Diagrams:** [`diagrams/05-meta-reducers.md`](./diagrams/05-meta-reducers.md),
[`diagrams/08-sync-flow-explained.md`](./diagrams/08-sync-flow-explained.md)
- **Source of truth:** `src/app/util/local-actions.token.ts`,
`src/app/util/skip-during-sync-window.operator.ts`,
`src/app/op-log/apply/hydration-state.service.ts`

2
docs/sync-and-op-log/diagrams/README.md
View file

@ -56,7 +56,7 @@ This directory contains visual diagrams explaining the Operation Log sync archit
| [../operation-log-architecture.md](../operation-log-architecture.md) | Comprehensive architecture reference |
| [../operation-rules.md](../operation-rules.md) | Design rules and guidelines |
| [../vector-clocks.md](../vector-clocks.md) | Vector clock implementation details |
| [../quick-reference.md](../quick-reference.md) | Quick lookup for common patterns |
| [../contributor-sync-model.md](../contributor-sync-model.md) | The single sync invariant (effects) |
## Diagram Conventions

416
docs/sync-and-op-log/long-term-plans/e2e-encryption-plan.md
View file

@ -1,416 +0,0 @@
# E2E Encryption for SuperSync Server
> **Status: Completed** (December 2025)
>
> This plan has been fully implemented. For the current implementation details, see:
>
> - [supersync-encryption-architecture.md](../supersync-encryption-architecture.md) - Implementation documentation with diagrams
> - `src/app/op-log/sync/operation-encryption.service.ts` - Core encryption service
## Summary
Add end-to-end encryption to SuperSync where the server cannot read operation payloads. Users provide a separate encryption password which is used to derive an encryption key client-side. This is the same approach used by legacy sync providers (Dropbox, WebDAV, Local File).
## Key Decisions
| Decision | Choice |
| -------------------- | -------------------------------------------------------------- |
| Encryption scope | Payload-only (metadata stays plaintext for conflict detection) |
| Key derivation | User-provided encryption password → Argon2id → key |
| Password change | Not supported (would require re-encrypting all data) |
| Server changes | None required |
| Missing key handling | Fail gracefully with dialog to enter password |
---
## Architecture
### Encryption Flow
```
User Encryption Password → Argon2id (64MB, 3 iter) → AES-256 Key → encrypt/decrypt payloads
```
### Data Flow
```
Upload: Operation → encrypt payload with key → upload (metadata plaintext)
Download: Receive ops → decrypt payload with key → apply to state
```
### Why Payload-Only Encryption?
The server needs plaintext metadata for:
- **Conflict detection** - Uses vector clocks to detect concurrent edits
- **Deduplication** - Uses operation IDs to prevent duplicates
- **Ordering** - Uses timestamps and server sequence numbers
- **Tombstone tracking** - Uses entity IDs for delete tracking
The server does NOT need to read:
- **Payloads** - The actual data being created/updated/deleted
This design encrypts payloads while keeping metadata accessible, giving the server enough information to coordinate sync without seeing user data.
---
## Implementation Plan
### Phase 1: Data Model Changes
**File:** `src/app/pfapi/api/sync/sync-provider.interface.ts`
Add encryption flag to `SyncOperation`:
```typescript
export interface SyncOperation {
// ... existing fields ...
isPayloadEncrypted?: boolean; // NEW: true if payload is encrypted string
}
```
**File:** `src/app/pfapi/api/sync/providers/super-sync/super-sync.model.ts`
The `encryptKey` field already exists in `SyncProviderPrivateCfgBase`. Just need to add the enable flag:
```typescript
export interface SuperSyncPrivateCfg extends SyncProviderPrivateCfgBase {
// ... existing fields ...
isEncryptionEnabled?: boolean; // NEW
// encryptKey?: string; // Already inherited from base
}
```
---
### Phase 2: Client-Side Encryption Service
**New file:** `src/app/op-log/sync/operation-encryption.service.ts`
```typescript
import { inject, Injectable } from '@angular/core';
import { encrypt, decrypt } from '../../../../pfapi/api/encryption/encryption';
@Injectable({ providedIn: 'root' })
export class OperationEncryptionService {
/**
* Encrypts the payload of a SyncOperation.
* Returns a new operation with encrypted payload and isPayloadEncrypted=true.
*/
async encryptOperation(op: SyncOperation, encryptKey: string): Promise<SyncOperation> {
const payloadStr = JSON.stringify(op.payload);
const encryptedPayload = await encrypt(payloadStr, encryptKey);
return {
...op,
payload: encryptedPayload,
isPayloadEncrypted: true,
};
}
/**
* Decrypts the payload of a SyncOperation.
* Returns a new operation with decrypted payload.
* Throws DecryptError if decryption fails.
*/
async decryptOperation(op: SyncOperation, encryptKey: string): Promise<SyncOperation> {
if (!op.isPayloadEncrypted) {
return op; // Pass through unencrypted ops
}
const decryptedStr = await decrypt(op.payload as string, encryptKey);
return {
...op,
payload: JSON.parse(decryptedStr),
isPayloadEncrypted: false,
};
}
/**
* Batch encrypt operations for upload.
*/
async encryptOperations(
ops: SyncOperation[],
encryptKey: string,
): Promise<SyncOperation[]> {
return Promise.all(ops.map((op) => this.encryptOperation(op, encryptKey)));
}
/**
* Batch decrypt operations after download.
* Non-encrypted ops pass through unchanged.
*/
async decryptOperations(
ops: SyncOperation[],
encryptKey: string,
): Promise<SyncOperation[]> {
return Promise.all(ops.map((op) => this.decryptOperation(op, encryptKey)));
}
}
```
**Reuses:** Existing `src/app/pfapi/api/encryption/encryption.ts` (AES-GCM, Argon2id)
---
### Phase 3: Upload Integration
**File:** `src/app/op-log/sync/operation-log-upload.service.ts`
Modify `_uploadPendingOpsViaApi()`:
```typescript
// Add injection
private encryptionService = inject(OperationEncryptionService);
// In _uploadPendingOpsViaApi(), after converting to SyncOperation format:
const privateCfg = await syncProvider.privateCfg.load();
let opsToUpload = syncOps;
if (privateCfg?.isEncryptionEnabled && privateCfg?.encryptKey) {
opsToUpload = await this.encryptionService.encryptOperations(syncOps, privateCfg.encryptKey);
}
// Upload opsToUpload instead of syncOps
const response = await syncProvider.uploadOps(opsToUpload, clientId, lastKnownServerSeq);
// Also encrypt piggybacked ops handling needs decryption:
if (response.newOps && response.newOps.length > 0) {
let ops = response.newOps.map((serverOp) => serverOp.op);
if (privateCfg?.encryptKey) {
ops = await this.encryptionService.decryptOperations(ops, privateCfg.encryptKey);
}
const operations = ops.map((op) => syncOpToOperation(op));
piggybackedOps.push(...operations);
}
```
---
### Phase 4: Download Integration
**File:** `src/app/op-log/sync/operation-log-download.service.ts`
Modify `_downloadRemoteOpsViaApi()`:
```typescript
// Add injection
private encryptionService = inject(OperationEncryptionService);
private matDialog = inject(MatDialog);
// After downloading ops, before converting to Operation format:
const privateCfg = await syncProvider.privateCfg.load();
let syncOps = response.ops
.filter((serverOp) => !appliedOpIds.has(serverOp.op.id))
.map((serverOp) => serverOp.op);
// Check if any ops are encrypted
const hasEncryptedOps = syncOps.some(op => op.isPayloadEncrypted);
if (hasEncryptedOps) {
let encryptKey = privateCfg?.encryptKey;
// If no key cached, prompt user
if (!encryptKey) {
encryptKey = await this._promptForEncryptionPassword();
if (!encryptKey) {
// User cancelled - abort sync
return { newOps: [], success: false };
}
}
try {
syncOps = await this.encryptionService.decryptOperations(syncOps, encryptKey);
} catch (e) {
if (e instanceof DecryptError) {
// Wrong password - prompt again
await this._showDecryptionErrorDialog();
return { newOps: [], success: false };
}
throw e;
}
}
const operations = syncOps.map((op) => syncOpToOperation(op));
```
---
### Phase 5: UI Changes
**File:** `src/app/features/config/form-cfgs/sync-form.const.ts`
Add encryption fields to SuperSync provider form:
```typescript
// In SuperSync fieldGroup, add:
{
key: 'isEncryptionEnabled',
type: 'checkbox',
props: {
label: T.F.SYNC.FORM.SUPER_SYNC.L_ENABLE_E2E_ENCRYPTION,
},
},
{
hideExpression: (model: any) => !model.isEncryptionEnabled,
key: 'encryptKey',
type: 'input',
props: {
type: 'password',
label: T.F.SYNC.FORM.L_ENCRYPTION_PASSWORD,
required: true,
},
},
{
hideExpression: (model: any) => !model.isEncryptionEnabled,
type: 'tpl',
props: {
tpl: `<div class="warn-text">{{ T.F.SYNC.FORM.SUPER_SYNC.ENCRYPTION_WARNING | translate }}</div>`,
},
},
```
**Translations:** `src/assets/i18n/en.json`
```json
{
"F": {
"SYNC": {
"FORM": {
"SUPER_SYNC": {
"L_ENABLE_E2E_ENCRYPTION": "Enable end-to-end encryption",
"ENCRYPTION_WARNING": "Warning: If you forget your encryption password, your data cannot be recovered. This password is separate from your login password."
}
},
"S": {
"DECRYPTION_FAILED": "Failed to decrypt synced data. Please check your encryption password.",
"ENCRYPTION_PASSWORD_REQUIRED": "Encryption password required to sync encrypted data."
}
}
}
}
```
**New dialog component:** `src/app/imex/sync/dialog-encryption-password/`
Simple dialog to prompt for encryption password when needed:
- Input field for password
- Cancel and OK buttons
- Used when encrypted ops are received but no password is cached
---
## File Summary
### New Files
| File | Purpose |
| ----------------------------------------------------- | -------------------------- |
| `src/app/op-log/sync/operation-encryption.service.ts` | Encrypt/decrypt operations |
| `src/app/imex/sync/dialog-encryption-password/` | Password prompt dialog |
### Modified Files
| File | Changes |
| ----------------------------------------------------------------- | ----------------------------------------- |
| `src/app/pfapi/api/sync/sync-provider.interface.ts` | Add `isPayloadEncrypted` to SyncOperation |
| `src/app/pfapi/api/sync/providers/super-sync/super-sync.model.ts` | Add `isEncryptionEnabled` flag |
| `src/app/op-log/sync/operation-log-upload.service.ts` | Encrypt before upload |
| `src/app/op-log/sync/operation-log-download.service.ts` | Decrypt after download |
| `src/app/features/config/form-cfgs/sync-form.const.ts` | Add encryption toggle + password field |
| `src/app/t.const.ts` | Add translation keys |
| `src/assets/i18n/en.json` | Add translation strings |
### No Server Changes Required
The server treats encrypted payloads as opaque strings - no modifications needed.
---
## Security Considerations
### What's Protected
1. **Payload content** - All user data (tasks, projects, notes, etc.) is encrypted
2. **Zero-knowledge** - Server never sees encryption password or plaintext data
3. **Strong crypto** - AES-256-GCM with Argon2id key derivation
### What's Exposed (by design)
1. **Operation metadata** - IDs, timestamps, entity types, vector clocks
2. **Traffic patterns** - Server knows when you sync and how many operations
3. **Encryption status** - Server can see `isPayloadEncrypted: true`
### Cryptographic Details
| Component | Algorithm | Parameters |
| ------------------ | ----------- | --------------------------------------------- |
| Key derivation | Argon2id | 64MB memory, 3 iterations |
| Payload encryption | AES-256-GCM | Random 12-byte IV, 16-byte salt per operation |
### Limitations
| Limitation | Reason |
| -------------------- | --------------------------------------------------------- |
| No password change | Would require re-encrypting all operations on all clients |
| No password recovery | True zero-knowledge means no recovery possible |
| Two passwords | Login password + encryption password (by design) |
### Threat Model
| Threat | Mitigated? | Notes |
| -------------------- | ---------- | --------------------------------------------------- |
| Server reads data | Yes | Payloads encrypted client-side |
| Server breach | Yes | Attacker gets encrypted blobs, needs password |
| MITM attack | Yes | HTTPS + authenticated encryption |
| Password brute force | Partially | Argon2id makes attacks expensive (64MB per attempt) |
| Lost password | No | Data unrecoverable without password |
---
## Migration Path
### Enabling Encryption (Existing User)
1. User enables "E2E Encryption" in SuperSync settings
2. User enters encryption password
3. Warning shown about password recovery
4. Password saved to `SuperSyncPrivateCfg.encryptKey`
5. `isEncryptionEnabled` set to true
6. Future operations encrypted; existing operations remain plaintext
7. Other clients prompted for password when they encounter encrypted ops
### Multi-Client Scenario
When encryption is enabled on one client:
1. Other clients download operations normally
2. When they encounter `isPayloadEncrypted: true`, decryption is attempted
3. If no password cached, dialog prompts for password
4. Password cached in local `SuperSyncPrivateCfg` for future syncs
### Disabling Encryption
1. User unchecks encryption toggle
2. `isEncryptionEnabled` set to false
3. Future operations sent without encryption
4. Existing encrypted operations remain encrypted (still readable with password)
---
## Testing Strategy
1. **Unit tests** for `OperationEncryptionService`
- Encrypt/decrypt round-trips with various payload types
- Non-encrypted ops pass through unchanged
- Wrong password throws DecryptError
2. **Integration tests** for upload/download
- Encrypted operations sync correctly
- Mixed encrypted/unencrypted history works
- Piggybacked operations decrypt correctly
3. **E2E tests**
- Two clients with same password sync correctly
- Missing password shows dialog
- Wrong password shows error and retries

624
docs/sync-and-op-log/long-term-plans/hybrid-manifest-architecture.md
View file

@ -1,624 +0,0 @@
# Hybrid Manifest & Snapshot Architecture for File-Based Sync
> **Status: Completed** (December 2025)
**Context:** Optimizing WebDAV/Dropbox sync for the Operation Log architecture.
**Related:** [Operation Log Architecture](../operation-log-architecture.md)
> **Implementation Note:** This architecture is fully implemented in `OperationLogManifestService`, `OperationLogUploadService`, and `OperationLogDownloadService`. The embedded operations buffer, overflow file creation, and snapshot support are all operational.
---
## 1. The Problem
The current `OperationLogSyncService` fallback for file-based providers (WebDAV, Dropbox) is inefficient for frequent, small updates.
**Current Workflow (Naive Fallback):**
1. **Write Operation File:** Upload `ops/ops_CLIENT_TIMESTAMP.json`.
2. **Read Manifest:** Download `ops/manifest.json` to get current list.
3. **Update Manifest:** Upload new `ops/manifest.json` with the new filename added.
**Issues:**
- **High Request Count:** Minimum 3 HTTP requests per sync cycle.
- **File Proliferation:** Rapidly creates thousands of small files, degrading WebDAV directory listing performance.
- **Latency:** On slow connections (standard WebDAV), this makes sync feel sluggish.
---
## 2. Proposed Solution: Hybrid Manifest
Instead of treating the manifest solely as an _index_ of files, we treat it as a **buffer** for recent operations.
### 2.1. Concept
- **Embedded Operations:** Small batches of operations are stored directly inside `manifest.json`.
- **Lazy Flush:** New operation files (`ops_*.json`) are only created when the manifest buffer fills up.
- **Snapshots:** A "base state" file allows us to delete old operation files and clear the manifest history.
### 2.2. Data Structures
**Updated Manifest:**
```typescript
interface HybridManifest {
version: 2;
// The baseline state (snapshot). If present, clients load this first.
lastSnapshot?: SnapshotReference;
// Ops stored directly in the manifest (The Buffer)
// Limit: ~50 ops or 100KB payload size
embeddedOperations: EmbeddedOperation[];
// References to external operation files (The Overflow)
// Older ops that were flushed out of the buffer
operationFiles: OperationFileReference[];
// Merged vector clock from all embedded operations
// Used for quick conflict detection without parsing all ops
frontierClock: VectorClock;
// Last modification timestamp (for ETag-like cache invalidation)
lastModified: number;
}
interface SnapshotReference {
fileName: string; // e.g. "snapshots/snap_1701234567890.json"
schemaVersion: number; // Schema version of the snapshot
vectorClock: VectorClock; // Clock state at snapshot time
timestamp: number; // When snapshot was created
}
interface OperationFileReference {
fileName: string; // e.g. "ops/overflow_1701234567890.json"
opCount: number; // Number of operations in file (for progress estimation)
minSeq: number; // First operation's logical sequence in this file
maxSeq: number; // Last operation's logical sequence
}
// Embedded operations are lightweight - full Operation minus redundant fields
interface EmbeddedOperation {
id: string;
actionType: string;
opType: OpType;
entityType: EntityType;
entityId?: string;
entityIds?: string[];
payload: unknown;
clientId: string;
vectorClock: VectorClock;
timestamp: number;
schemaVersion: number;
}
```
**Snapshot File Format:**
```typescript
interface SnapshotFile {
version: 1;
schemaVersion: number; // App schema version
vectorClock: VectorClock; // Merged clock at snapshot time
timestamp: number;
data: AppDataComplete; // Full application state
checksum?: string; // Optional SHA-256 for integrity verification
}
```
---
## 3. Workflows
### 3.1. Upload (Write Path)
When a client has local pending operations to sync:
```
┌─────────────────────────────────────────────────────────────────┐
│ Upload Flow │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 1. Download manifest.json │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 2. Detect remote changes │
│ (compare frontierClock) │
└───────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
Remote has new ops? No remote changes
│ │
▼ │
Download & apply first ◄───────┘
│
▼
┌───────────────────────────────┐
│ 3. Check buffer capacity │
│ embedded.length + pending │
└───────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
< BUFFER_LIMIT (50) >= BUFFER_LIMIT
│ │
▼ ▼
Append to embedded Flush embedded to file
│ + add pending to empty buffer
│ │
└───────────────┬───────────────┘
▼
┌───────────────────────────────┐
│ 4. Check snapshot trigger │
│ (operationFiles > 50 OR │
│ total ops > 5000) │
└───────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
Trigger snapshot No snapshot needed
│ │
└───────────────┬───────────────┘
▼
┌───────────────────────────────┐
│ 5. Upload manifest.json │
└───────────────────────────────┘
```
**Detailed Steps:**
1. **Download Manifest:** Fetch `manifest.json` (or create empty v2 manifest if not found).
2. **Detect Remote Changes:**
- Compare `manifest.frontierClock` with local `lastSyncedClock`.
- If remote has unseen changes → download and apply before uploading (prevents lost updates).
3. **Evaluate Buffer:**
- `BUFFER_LIMIT = 50` operations (configurable)
- `BUFFER_SIZE_LIMIT = 100KB` payload size (prevents manifest bloat)
4. **Strategy Selection:**
- **Scenario A (Append):** If `embedded.length + pending.length < BUFFER_LIMIT`:
- Append `pendingOps` to `manifest.embeddedOperations`.
- Update `manifest.frontierClock` with merged clocks.
- **Result:** 1 Write (manifest). Fast path.
- **Scenario B (Overflow):** If buffer would exceed limit:
- Upload `manifest.embeddedOperations` to new file `ops/overflow_TIMESTAMP.json`.
- Add file reference to `manifest.operationFiles`.
- Place `pendingOps` into now-empty `manifest.embeddedOperations`.
- **Result:** 1 Upload (overflow file) + 1 Write (manifest).
5. **Upload Manifest:** Write updated `manifest.json`.
### 3.2. Download (Read Path)
When a client checks for updates:
```
┌─────────────────────────────────────────────────────────────────┐
│ Download Flow │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 1. Download manifest.json │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 2. Quick-check: any changes? │
│ Compare frontierClock │
└───────────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
No changes (clocks equal) Changes detected
│ │
▼ ▼
Done ┌────────────────────────┐
│ 3. Need snapshot? │
│ (local behind snapshot)│
└────────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
Download snapshot Skip to ops
+ apply as base │
│ │
└───────────────┬───────────────┘
▼
┌────────────────────────┐
│ 4. Download new op │
│ files (filter seen) │
└────────────────────────┘
│
▼
┌────────────────────────┐
│ 5. Apply embedded ops │
│ (filter by op.id) │
└────────────────────────┘
│
▼
┌────────────────────────┐
│ 6. Update local │
│ lastSyncedClock │
└────────────────────────┘
```
**Detailed Steps:**
1. **Download Manifest:** Fetch `manifest.json`.
2. **Quick-Check Changes:**
- Compare `manifest.frontierClock` against local `lastSyncedClock`.
- If clocks are equal → no changes, done.
3. **Check Snapshot Needed:**
- If local state is older than `manifest.lastSnapshot.vectorClock` → download snapshot first.
- Apply snapshot as base state (replaces local state).
4. **Download Operation Files:**
- Filter `manifest.operationFiles` to only files with `maxSeq > localLastAppliedSeq`.
- Download and parse each file.
- Collect all operations.
5. **Apply Embedded Operations:**
- Filter `manifest.embeddedOperations` by `op.id` (skip already-applied).
- Add to collected operations.
6. **Apply All Operations:**
- Sort by `vectorClock` (causal order).
- Detect conflicts using existing `detectConflicts()` logic.
- Apply non-conflicting ops; present conflicts to user.
7. **Update Tracking:**
- Set `localLastSyncedClock = manifest.frontierClock`.
---
## 4. Snapshotting (Compaction)
To prevent unbounded growth of operation files, any client can trigger a snapshot.
### 4.1. Triggers
| Condition | Threshold | Rationale |
| ------------------------------- | --------- | -------------------------------------- |
| External `operationFiles` count | > 50 | Prevent WebDAV directory bloat |
| Total operations since snapshot | > 5000 | Bound replay time for fresh installs |
| Time since last snapshot | > 7 days | Ensure periodic cleanup |
| Manifest size | > 500KB | Prevent manifest from becoming too big |
### 4.2. Process
```
┌─────────────────────────────────────────────────────────────────┐
│ Snapshot Flow │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 1. Ensure full sync complete │
│ (no pending local/remote) │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 2. Read current state from │
│ NgRx (authoritative) │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 3. Generate snapshot file │
│ + compute checksum │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 4. Upload snapshot file │
│ (atomic, verify success) │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 5. Update manifest │
│ - Set lastSnapshot │
│ - Clear operationFiles │
│ - Clear embeddedOperations │
│ - Reset frontierClock │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 6. Upload manifest │
└───────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ 7. Cleanup (async, best- │
│ effort): delete old files │
└───────────────────────────────┘
```
### 4.3. Snapshot Atomicity
**Problem:** If the client crashes between uploading snapshot and updating manifest, other clients won't see the new snapshot.
**Solution:** Snapshot files are immutable and safe to leave orphaned. The manifest is the source of truth. Cleanup is best-effort.
**Invariant:** Never delete the current `lastSnapshot` file until a new snapshot is confirmed.
---
## 5. Conflict Handling
The hybrid manifest doesn't change conflict detection - it still uses vector clocks. However, the `frontierClock` in the manifest enables **early conflict detection**.
### 5.1. Early Conflict Detection
Before downloading all operations, compare clocks:
```typescript
const comparison = compareVectorClocks(localFrontierClock, manifest.frontierClock);
switch (comparison) {
case VectorClockComparison.LESS_THAN:
// Remote is ahead - safe to download
break;
case VectorClockComparison.GREATER_THAN:
// Local is ahead - upload our changes
break;
case VectorClockComparison.CONCURRENT:
// Potential conflicts - download ops for detailed analysis
break;
case VectorClockComparison.EQUAL:
// No changes - skip download
break;
}
```
### 5.2. Conflict Resolution
When conflicts are detected at the operation level, the existing `ConflictResolutionService` handles them. The hybrid manifest doesn't change this flow.
---
## 6. Edge Cases & Failure Modes
### 6.1. Concurrent Uploads (Race Condition)
**Scenario:** Two clients download the manifest simultaneously, both append ops, both upload.
**Problem:** Second upload overwrites first client's operations.
**Solution:** Use provider-specific mechanisms:
| Provider | Mechanism |
| ----------- | ------------------------------------------- |
| **Dropbox** | Use `update` mode with `rev` parameter |
| **WebDAV** | Use `If-Match` header with ETag |
| **Local** | File locking (already implemented in PFAPI) |
**Implementation:**
```typescript
interface HybridManifest {
// ... existing fields
// Optimistic concurrency control
etag?: string; // Server-assigned revision (Dropbox rev, WebDAV ETag)
}
async uploadManifest(manifest: HybridManifest, expectedEtag?: string): Promise<void> {
// If expectedEtag provided, use conditional upload
// On conflict (412 Precondition Failed), re-download and retry
}
```
### 6.2. Manifest Corruption
**Scenario:** Manifest JSON is invalid (partial write, encoding issue).
**Recovery Strategy:**
1. Attempt to parse manifest.
2. On parse failure, check for backup manifest (`manifest.json.bak`).
3. If no backup, reconstruct from operation files using `listFiles()`.
4. If reconstruction fails, fall back to snapshot-only state.
```typescript
async loadManifestWithRecovery(): Promise<HybridManifest> {
try {
return await this._loadRemoteManifest();
} catch (parseError) {
PFLog.warn('Manifest corrupted, attempting recovery...');
// Try backup
try {
return await this._loadBackupManifest();
} catch {
// Reconstruct from files
return await this._reconstructManifestFromFiles();
}
}
}
```
### 6.3. Snapshot File Missing
**Scenario:** Manifest references a snapshot that doesn't exist on the server.
**Recovery Strategy:**
1. Log error and notify user.
2. Fall back to replaying all available operation files.
3. If operation files also reference missing ops, show data loss warning.
### 6.4. Schema Version Mismatch
**Scenario:** Snapshot was created with schema version 3, but local app is version 2.
**Handling:**
- If `snapshot.schemaVersion > CURRENT_SCHEMA_VERSION + MAX_VERSION_SKIP`:
- Reject snapshot, prompt user to update app.
- If `snapshot.schemaVersion > CURRENT_SCHEMA_VERSION`:
- Load with warning (some fields may be stripped by Typia).
- If `snapshot.schemaVersion < CURRENT_SCHEMA_VERSION`:
- Run migrations on loaded state.
### 6.5. Large Pending Operations
**Scenario:** User was offline for a week, has 500 pending operations.
**Handling:**
- Don't try to embed all 500 in manifest.
- Batch into multiple overflow files (100 ops each).
- Upload files first, then update manifest once.
```typescript
const BATCH_SIZE = 100;
const chunks = chunkArray(pendingOps, BATCH_SIZE);
for (const chunk of chunks) {
await this._uploadOverflowFile(chunk);
}
// Single manifest update at the end
await this._uploadManifest(manifest);
```
---
## 7. Advantages Summary
| Metric | Current (v1) | Hybrid Manifest (v2) |
| :---------------------- | :----------------------------------- | :---------------------------------------------------- |
| **Requests per Sync** | 3 (Upload Op + Read Man + Write Man) | **1-2** (Read Man, optional Write) |
| **Files on Server** | Unbounded growth | **Bounded** (1 Manifest + 0-50 Op Files + 1 Snapshot) |
| **Fresh Install Speed** | O(n) - replay all ops | **O(1)** - load snapshot + small delta |
| **Conflict Detection** | Must parse all ops | **Quick check** via frontierClock |
| **Bandwidth per Sync** | ~2KB (op file) + manifest overhead | **~1KB** (manifest only for small changes) |
| **Offline Resilience** | Good | **Same** (operations buffered locally) |
---
## 8. Implementation Status
All phases have been implemented as of December 2025:
### ✅ Phase 1: Core Infrastructure (Complete)
1. **Types** (`operation.types.ts`):
- `HybridManifest`, `SnapshotReference`, `OperationFileReference` interfaces defined
- Backward compatibility maintained with existing `OperationLogManifest`
2. **Manifest Handling** (`operation-log-manifest.service.ts`):
- `loadManifest()` handles v1 and v2 formats
- Automatic v1 to v2 migration on first write
- Buffer/overflow logic in upload services
3. **FrontierClock Tracking**:
- Vector clocks merged when adding embedded operations
- `lastSyncedFrontierClock` stored locally for quick-check
### ✅ Phase 2: Snapshot Support (Complete)
4. **Snapshot Operations** (in `operation-log-upload.service.ts` and `operation-log-download.service.ts`):
- Snapshot generation with current state serialization
- Upload with retry logic
- Download + validate + apply
5. **Snapshot Triggers**:
- Automatic triggers based on file count and operation count
- Remote file cleanup after 14 days (`REMOTE_OP_FILE_RETENTION_MS`)
### ✅ Phase 3: Robustness (Complete)
6. **Concurrency Control**:
- Provider-specific revision checking (Dropbox rev, WebDAV ETag)
- Retry-on-conflict logic implemented
7. **Recovery Logic**:
- Manifest corruption recovery with file listing fallback
- Missing file handling with graceful degradation
### ✅ Phase 4: Testing (Complete)
8. **Tests**:
- Unit tests in `operation-log-manifest.service.spec.ts`
- Integration tests in `sync-scenarios.integration.spec.ts`
- E2E tests in `supersync.spec.ts`
### Key Implementation Files
| File | Purpose |
| ----------------------------------- | ------------------------------------------- |
| `operation-log-manifest.service.ts` | Manifest loading, saving, buffer management |
| `operation-log-upload.service.ts` | Upload with buffer/overflow logic |
| `operation-log-download.service.ts` | Download with snapshot support |
| `operation.types.ts` | Type definitions |
---
## 9. Configuration Constants
```typescript
// Buffer limits
const EMBEDDED_OP_LIMIT = 50; // Max operations in manifest buffer
const EMBEDDED_SIZE_LIMIT_KB = 100; // Max payload size in KB
// Snapshot triggers
const SNAPSHOT_FILE_THRESHOLD = 50; // Trigger when operationFiles exceeds this
const SNAPSHOT_OP_THRESHOLD = 5000; // Trigger when total ops exceed this
const SNAPSHOT_AGE_DAYS = 7; // Trigger if no snapshot in N days
// Batching
const UPLOAD_BATCH_SIZE = 100; // Ops per overflow file
// Retry
const MAX_UPLOAD_RETRIES = 3;
const RETRY_DELAY_MS = 1000;
```
---
## 10. Resolved Design Questions
The following questions were resolved during implementation:
1. **Encryption:** Snapshots use the same encryption as operation files (via `EncryptAndCompressHandlerService`).
2. **Compression:** Snapshots are compressed using the same compression scheme as other sync files.
3. **Checksum Verification:** Currently using timestamp-based validation; checksums can be added if needed.
4. **Clock Drift:** Vector clocks handle ordering; timestamps are informational only.
---
## 11. File Reference
### Remote Storage Layout (v2)
```
/ (or /DEV/ in development)
├── manifest.json # HybridManifest (buffer + references)
├── ops/
│ ├── ops_CLIENT1_170123.json # Flushed operations
│ └── ops_CLIENT2_170456.json
└── snapshots/
└── snap_170789.json # Full state snapshot (if present)
```
### Code Files
```
src/app/op-log/
├── operation.types.ts # HybridManifest, SnapshotReference types
├── store/
│ └── operation-log-manifest.service.ts # Manifest management
├── sync/
│ ├── operation-log-upload.service.ts # Upload with buffer/overflow
│ └── operation-log-download.service.ts # Download with snapshot support
└── docs/
└── hybrid-manifest-architecture.md # This document
```

80
docs/sync-and-op-log/long-term-plans/replace-pfapi-with-oplog-plan.md
View file

@ -1,80 +0,0 @@
# Plan: Replace PFAPI with Operation Log Sync for All Providers
> **Status: Completed** (January 2026)
>
> This plan has been fully implemented. The entire `src/app/pfapi/` directory has been deleted.
> All sync providers now use the unified operation log system via `FileBasedSyncAdapter`.
>
> **Current Implementation:**
>
> - Sync providers: `src/app/op-log/sync-providers/`
> - File-based adapter: `src/app/op-log/sync-providers/file-based/file-based-sync-adapter.service.ts`
> - Server migration: `src/app/op-log/sync/server-migration.service.ts`
---
## Original Goal
Simplify the codebase by removing PFAPI's model-by-model sync and using operation logs exclusively for **all sync providers** (WebDAV, Dropbox, LocalFile). Migration required for existing users; old PFAPI files kept as backup.
## What Was Implemented
### Phase 1: Enable Operation Log Sync (All Providers) - DONE
All providers now use operation log sync:
- WebDAV: `src/app/op-log/sync-providers/file-based/webdav/`
- Dropbox: `src/app/op-log/sync-providers/file-based/dropbox/`
- LocalFile: `src/app/op-log/sync-providers/file-based/local-file/`
- SuperSync: `src/app/op-log/sync-providers/super-sync/`
### Phase 2: Migration Logic - DONE
Migration from legacy PFAPI format is handled by `ServerMigrationService`:
- Checks for existing PFAPI metadata file on remote
- Downloads full state and creates SYNC_IMPORT operation
- Uploads initial snapshot via operation log
### Phase 3: PFAPI Code Removal - DONE
The entire `src/app/pfapi/` directory has been deleted (~83 files, 2.0 MB).
What was kept (moved to op-log):
- Provider implementations (WebDAV, Dropbox, LocalFile)
- Encryption/compression utilities
- Auth flows
### Phase 4: Testing & Cleanup - DONE
- Multi-device sync scenarios tested via E2E tests
- Migration testing completed
- Large operation log handling verified
- All tests pass
## Final Architecture
```
src/app/op-log/
├── sync-providers/
│ ├── super-sync/ # Server-based sync
│ ├── file-based/ # File-based providers
│ │ ├── file-based-sync-adapter.service.ts
│ │ ├── webdav/
│ │ ├── dropbox/
│ │ └── local-file/
│ ├── provider-manager.service.ts
│ └── wrapped-provider.service.ts
├── sync/
│ ├── operation-log-sync.service.ts
│ └── server-migration.service.ts
└── ...
```
## Key Decisions Made
- Single-file sync format (`sync-data.json`) with state snapshot + recent ops
- Content-based optimistic locking via `syncVersion` counter
- Piggybacking mechanism for concurrent sync handling
- Server migration service handles legacy PFAPI data migration

1993
docs/sync-and-op-log/operation-log-architecture-diagrams.md
View file

File diff suppressed because it is too large Load diff

82
docs/sync-and-op-log/operation-log-architecture.md
View file

@ -116,6 +116,39 @@ This document is structured around these four purposes. Most complexity lives in
---
## Why this architecture: rejected alternatives
The operation log is **not** incidental complexity. It is the minimum design
that satisfies one hard, non-negotiable constraint:
> **No silent data loss on concurrent multi-device edits, offline-first, with a
> "dumb" server that cannot merge** (WebDAV/Dropbox/LocalFile have no server
> logic; SuperSync payloads are end-to-end encrypted and opaque to the server).
Independent prior analyses (three separate model reviews) evaluated every
simpler approach against that constraint and rejected each:
| Alternative | What it is | Why rejected |
| --- | --- | --- |
| **Last-Write-Wins (global timestamp)** | Drop logical clocks; newest wall-clock write wins | User devices have unreliable clocks; concurrent independent field edits silently overwrite each other. Unacceptable for a personal productivity app. (Survives only as a *field-level* tie-break inside conflict resolution.) |
| **Delta / state-diff sync** | Keep a shadow copy, upload changed fields, server shallow-merges | Shadow state has no atomic coupling with the watermark → a crash mid-sync corrupts permanently; LWW shallow-merge loses concurrent independent edits; O(N) `JSON.stringify` diffing freezes the UI at 10k+ tasks; requires server-side merge, incompatible with opaque E2EE payloads. |
| **Full-state / snapshot sync** | Sync whole model files (the old PFAPI model) | Re-transfers everything on every change; no per-entity conflict granularity; cannot reconstruct intent after an offline edit. Retained only as a *bootstrapping* mechanism (snapshot + tail replay), not the sync mechanism. |
| **CRDTs (Yjs/Automerge/etc.)** | Math-guaranteed convergence | High conceptual complexity; most implementations assume a trusted server or relay, clashing with the dumb-file + E2EE constraint. The op-log deliberately *borrows* op-based-CRDT properties (UUID idempotency, causal ordering) without the full machinery. |
| **Server-assigned sequence numbers** | Let the server impose a total order | Requires server connectivity for ordering — incompatible with offline-first and file-based providers that have no server. Used only as a *complement* (SuperSync seq for global order; vector clocks still required for the file-based/offline case). |
**Consequences any future redesign must preserve:** no silent loss from
concurrent independent edits; works without a trusted/merging server and with
opaque E2EE payloads; offline edits rebase cleanly on reconnect; tombstones
with retention; bounded growth via snapshot + compaction; conflict metadata
prefers false-concurrency over false-ordering (compare clocks *before* pruning);
scales to 10k+ active / 20k+ archived tasks without main-thread O(N) work.
The only self-identified over-engineering historically was the vector-clock
pruning *defense layers*, which were since removed (see
[`vector-clocks.md`](./vector-clocks.md)).
---
# Part A: Local Persistence
The operation log is primarily a **Write-Ahead Log (WAL)** for local persistence. It provides:
@ -1289,20 +1322,6 @@ for (const entry of fullStateOps) {
2. **Semantic clarity** - Full state uploads use appropriate endpoint
3. **Server-side optimization** - Server can cache snapshots for faster client bootstrap
## C.4 File-Based Sync Fallback
For providers without API support (WebDAV/Dropbox), operations are synced via files (`OperationLogUploadService` and `OperationLogDownloadService` handle this transparently):
```
ops/
├── manifest.json
├── ops_CLIENT1_1701234567890.json
├── ops_CLIENT1_1701234599999.json
└── ops_CLIENT2_1701234600000.json
```
The manifest tracks which operation files exist. Each file contains a batch of operations. The system supports both API-based sync and this file-based fallback.
## C.4 Conflict Detection
Conflicts are detected using vector clocks at the entity level. **Importantly, a conflict can only
@ -1423,7 +1442,7 @@ When a `moveToArchive` operation conflicts with a field-level update (e.g., rena
**Implementation:** `ConflictResolutionService` checks whether either the local or remote side contains a `TASK_SHARED_MOVE_TO_ARCHIVE` action. If so, the archive side wins automatically, and a new archive operation is created with a merged vector clock (via `_createArchiveWinOp()`).
This is the **first level** of archive resurrection prevention. The **second level** is in the `bulkOperationsMetaReducer` (see [Area 10: Bulk Application](#area-10-bulk-application) in the quick reference), which pre-scans operation batches for archive operations and skips any LWW Update operations targeting entities being archived in the same batch. This two-level defense handles the 3+ client scenario where LWW Updates can arrive before or after archive ops in the same batch.
This is the **first level** of archive resurrection prevention. The **second level** is in the `bulkOperationsMetaReducer` (see [Archive Resurrection Prevention](diagrams/06-archive-operations.md#archive-resurrection-prevention-two-level-defense)), which pre-scans operation batches for archive operations and skips any LWW Update operations targeting entities being archived in the same batch. This two-level defense handles the 3+ client scenario where LWW Updates can arrive before or after archive ops in the same batch.
**Key files:**
@ -1542,7 +1561,7 @@ for (const op of ops) {
**Why Drop CONCURRENT?** An operation from a client that never saw the import may reference entities that no longer exist in the imported state. Dropping ensures the import truly restores all clients to the same point in time.
See [operation-log-architecture-diagrams.md](./operation-log-architecture-diagrams.md) Section 2c for visual diagrams.
See [diagrams/03-conflict-resolution.md](./diagrams/03-conflict-resolution.md) for visual diagrams.
---
@ -1976,6 +1995,34 @@ Fresh clients receive time tracking via SYNC_IMPORT:
| `time-tracking.reducer.ts` | NgRx reducer for syncTimeTracking |
| `archive-operation-handler.service.ts` | Handles flushYoungToOld remotely |
## E.7 Archive Payload: Rejected Optimizations
`moveToArchive` deliberately carries **full task data** (~2 KB/task), not just
IDs. Reason: archive sync is two systems — the operation syncs immediately, but
the archive model file syncs later. A remote client receiving `moveToArchive`
must write the tasks to its local archive _now_, before the archive file
arrives and while the tasks are already deleted from the originating client's
active state. So the operation must be self-sufficient.
Smaller-payload alternatives were explored and rejected:
| Option | Idea | Why rejected |
| ----------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A — private `_tasks` field | strip before storage | remote ops still need the full data for sync |
| B — meta-reducer enrichment | capture tasks from state before deletion | meta-reducers must be pure; awkward async from the sync reducer |
| C — two-phase (write + delete) | split into two ops | same total payload, just more complexity |
| D — operation-derived archive store | archive populated entirely by ID-only ops | migration of years of existing archive data; initial sync must replay 20K+ archive ops; unbounded op-log growth; compaction must preserve archive state; PFAPI transition |
**Decision: keep the full-payload approach** — it works, has no timing edge
cases, is simple, and archiving is infrequent (end-of-day, not constant). The
size reduction does not justify the complexity. For very large archives, chunk
dispatches at `ARCHIVE_CHUNK_SIZE = 25`.
**Escape hatch (preferred first resort if size becomes a real, not theoretical,
problem):** task data is text/JSON and compresses >90% — compress the `_tasks`
payload within the `moveToArchive` operation (LZ-string/GZIP) before sending.
This removes the size concern without the Option-D architectural overhaul.
---
# Part F: Atomic State Consistency
@ -2336,7 +2383,6 @@ e2e/
# References
- [Operation Rules](./operation-rules.md) - Payload and validation rules
- [Contributor Sync Model](./contributor-sync-model.md) - The single invariant for effects, reducers, and bulk dispatch
- [SuperSync Encryption](./supersync-encryption-architecture.md) - End-to-end encryption implementation
- [Hybrid Manifest Architecture](./long-term-plans/hybrid-manifest-architecture.md) - File-based sync optimization
- [Vector Clocks](./vector-clocks.md) - Vector clock implementation details
- [File-Based Sync Implementation](../ai/file-based-oplog-sync-implementation-plan.md) - Historical implementation plan

202
docs/sync-and-op-log/operation-payload-optimization-discussion.md
View file

@ -1,202 +0,0 @@
# Operation Payload Optimization Discussion
**Date:** December 5, 2025
**Context:** Analysis of operation payload sizes and optimization opportunities
---
## Initial Analysis
We analyzed the codebase for occasions when many or very large operations are produced.
### Issues Identified
| Issue | Severity | Impact |
| --------------------------------- | -------- | ---------------------------------- |
| Tag deletion cascade | **High** | Creates N+1 operations for N tasks |
| Full payload storage | **High** | Large payloads stored repeatedly |
| batchUpdateForProject nesting | Medium | Single op contains nested array |
| Archive operations | Medium | One bulk op for many tasks |
| Single operations per bulk entity | Medium | N operations instead of 1 |
### Fixes Implemented
1. **Payload size monitoring** - Added `LARGE_PAYLOAD_WARNING_THRESHOLD_BYTES` (10KB) and logging when exceeded
2. **Bulk task-repeat-cfg operations** - Tag deletion now uses bulk delete instead of N individual operations
3. **Batch operation chunking** - `batchUpdateForProject` now chunks large operations into batches of `MAX_BATCH_OPERATIONS_SIZE` (50)
---
## Archive Operation Deep Dive
The `moveToArchive` action was identified as having large payloads (~2KB per task). We explored multiple optimization approaches.
### The Core Problem
Two sync systems exist:
1. **Operation Log (SuperSync)** - Real-time operation sync
2. **PFAPI** - Model file sync (daily for archive files)
When Client A archives tasks:
- Operation syncs immediately
- `archiveYoung` model file syncs later (daily)
When Client B receives the operation:
- Must write tasks to local archive
- But tasks are deleted from originating client's state
- Archive file hasn't synced yet
**The operation must carry full task data.**
### Solutions Explored
#### Option A: Hybrid Payload with Private Field
```typescript
moveToArchive: {
taskIds: string[], // Persisted
_tasks: TaskWithSubTasks[] // Stripped before storage
}
```
**Problem:** Remote operations won't have `_tasks` - still need full data for sync.
#### Option B: Meta-Reducer Enrichment
Capture tasks from state before deletion, attach to action for effect.
**Why it seemed possible:**
- Dependency resolution ensures `addTask` ops applied before `moveToArchive`
- Tasks exist in remote client's state when operation arrives
- Meta-reducer runs before main reducer
**Problems:**
- Complex action mutation
- Meta-reducers should be pure
- Awkward async queue from sync reducer
#### Option C: Two-Phase Archive
Split into `writeToArchive` (full data) + `deleteTasks` (IDs only).
**Problem:** Same total payload size. Just added complexity without benefit.
#### Option D: Operation-Derived Archive Store
Archive becomes a separate IndexedDB store populated entirely by operations:
```typescript
archiveTask: { taskIds: string[] } // IDs only
```
Meta-reducer moves task data from active state to archive before deletion.
**Benefits:**
- Tiny payloads
- Single source of truth
- No PFAPI archive sync needed
**Drawbacks:**
1. Migration complexity (years of existing archive data)
2. Initial sync must replay ALL archive ops (20K+ for heavy users)
3. Operation log growth (archive ops span years)
4. Compaction complexity (must preserve archive state)
5. Two storage systems to coordinate
6. PFAPI compatibility during transition
7. Query performance for 20K+ tasks
### The Scale Concern
> "There can be more than 20,000 archived tasks"
If archive was in NgRx store:
- Selectors iterate 20K+ entities
- Entity adapter operations slow down
- Memory bloat on app start
- DevTools unusable
This ruled out simple "add isArchived flag" approaches.
### Key Insight: Dependency Resolution
Operations have causal ordering. When remote client receives `moveToArchive`:
1. `addTask` operations already applied (dependency)
2. Task exists in remote client's active state
3. Could theoretically look up from state before deletion
But the effect runs AFTER the reducer deletes entities. The timing makes this approach impractical without complex meta-reducer side effects.
---
## Final Decision
**Keep the current full-payload approach.**
### Rationale
1. **It works correctly** - Already implemented, tested, documented
2. **Sync reliability** - No edge cases or timing issues
3. **Simplicity** - Single action, clear semantics
4. **Acceptable size** - ~100KB for 50 tasks is manageable
5. **Infrequent operation** - Archiving happens at end of day, not constantly
### Mitigation
For very large archives, chunk operations:
```typescript
const ARCHIVE_CHUNK_SIZE = 25;
async moveToArchive(tasks: TaskWithSubTasks[]): Promise<void> {
const chunks = chunkArray(parentTasks, ARCHIVE_CHUNK_SIZE);
for (const chunk of chunks) {
this._store.dispatch(TaskSharedActions.moveToArchive({ tasks: chunk }));
}
await this._archiveService.moveTasksToArchiveAndFlushArchiveIfDue(parentTasks);
}
```
### Trade-off Summary
| Approach | Payload Size | Complexity | Reliability |
| ------------------------- | ----------------- | ---------- | ----------- |
| Full payload (current) | Large (~2KB/task) | Low | High |
| Meta-reducer enrichment | Small | High | Medium |
| Two-phase archive | Same as current | Higher | High |
| Operation-derived archive | Small | Very High | Medium |
**The payload size reduction doesn't justify the added complexity.**
---
## Related Documentation
- `archive-operation-redesign.md` - Detailed analysis of archive options
- `code-audit.md` - Overall operation compliance audit
- `operation-size-analysis.md` - Initial payload size analysis
---
## Future Considerations
If payload size becomes a real problem (not theoretical), revisit Option D (operation-derived archive) with:
1. Proper migration plan for existing PFAPI data
2. Compaction strategy for long-lived archive operations
3. Performance testing with 20K+ tasks
4. PFAPI compatibility during transition
**Alternative Optimization:**
5. **Payload Compression**: Since task data (text/JSON) compresses extremely well (often >90%), we could compress the `_tasks` payload within the `moveToArchive` operation (e.g., using LZ-string or GZIP) before sending. This would solve the size concern without requiring the architectural overhaul of Option D.
Until then, current approach is the right balance.

798
docs/sync-and-op-log/quick-reference.md
View file

@ -1,798 +0,0 @@
# Operation Log & Sync: Quick Reference
This document provides visual summaries of each major component in the operation log and sync system. For detailed documentation, see `operation-log-architecture.md` and `operation-log-architecture-diagrams.md`.
---
## Area 1: Write Path
The write path captures user actions and persists them as operations to IndexedDB.
```
┌─────────────────────────────────────────────────────────────────┐
│ Write Path │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. User Action ──► NgRx Dispatch │
│ │ │
│ 2. Reducers ──► State Updated (optimistic) │
│ │ │
│ 3. Meta-reducer ──► operationCaptureMetaReducer │
│ │ │
│ 4. Queue ──► OperationCaptureService.enqueue() │
│ │ │
│ 5. Effect ──► OperationLogEffects.persistOperation$ │
│ │ │
│ 6. Lock ──► Web Locks API (cross-tab coordination) │
│ │ │
│ 7. Clock ──► incrementVectorClock(clock, clientId) │
│ │ │
│ 8. Validate ──► validateOperationPayload() (Checkpoint A)│
│ │ │
│ 9. Persist ──► SUP_OPS.ops (IndexedDB) │
│ │ │
│ 10. Upload ──► ImmediateUploadService.trigger() │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Key Files:**
- `operation-capture.meta-reducer.ts` - Captures persistent actions
- `operation-capture.service.ts` - FIFO queue
- `operation-log.effects.ts` - Writes to IndexedDB
- `operation-log-store.service.ts` - IndexedDB wrapper
---
## Area 2: Read Path (Hydration)
The read path loads application state at startup by combining a snapshot with tail operations.
```
┌─────────────────────────────────────────────────────────────────┐
│ Read Path │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. App Startup │
│ │ │
│ 2. Parallel Recovery ──► _recoverPendingRemoteOps() │
│ _migrateVectorClockFromPfapi() │
│ hasStateCacheBackup() │
│ │ │
│ 3. Load Snapshot ──► SUP_OPS.state_cache │
│ │ │
│ 4. Schema Migration ──► migrateStateIfNeeded() (if needed) │
│ │ │
│ 5. Validate ──► Checkpoint B │
│ │ │
│ 6. Restore Clock ──► setVectorClock(snapshot.vectorClock)│
│ │ │
│ 7. Load to NgRx ──► loadAllData(snapshot.state) │
│ │ │
│ 8. Load Tail Ops ──► getOpsAfterSeq(lastAppliedOpSeq) │
│ │ │
│ 9. Migrate Tail ──► _migrateTailOps() │
│ │ │
│ 10. Bulk Replay ──► bulkApplyOperations() │
│ │ │
│ 11. Validate ──► Checkpoint C │
│ │ │
│ 12. Save Snapshot ──► saveStateCache() (if many ops) │
│ │ │
│ 13. Deferred Check ──► _scheduleDeferredValidation() (5s) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Key Files:**
- `operation-log-hydrator.service.ts` - Orchestrates hydration
- `schema-migration.service.ts` - Schema migrations
- `bulk-hydration.meta-reducer.ts` - Bulk operation application
---
## Area 3: Server Sync (SuperSync)
SuperSync exchanges individual operations with a centralized server.
```
┌─────────────────────────────────────────────────────────────────┐
│ COMPLETE SYNC CYCLE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. UPLOAD │
│ ├─ Flush pending writes │
│ ├─ Server migration check (inside lock) │
│ ├─ Upload ops in batches of 25 │
│ ├─ Receive piggybacked ops + rejected ops list │
│ ├─ Process piggybacked → triggers conflict detection │
│ └─ Handle rejected → LWW or mark rejected │
│ │
│ 2. DOWNLOAD (if hasMorePiggyback or scheduled) │
│ ├─ GET /ops?sinceSeq=X │
│ ├─ Gap detection → reset to seq 0 if needed │
│ ├─ Filter already-applied ops │
│ ├─ Decrypt if encrypted │
│ ├─ Paginate while hasMore │
│ └─ Return ops for processing │
│ │
│ 3. PROCESS REMOTE OPS │
│ ├─ Schema migration │
│ ├─ Filter ops invalidated by SYNC_IMPORT │
│ ├─ Full-state op? → Apply directly │
│ ├─ Conflict detection via vector clocks │
│ ├─ LWW resolution if conflicts │
│ ├─ Apply to NgRx via operationApplier │
│ ├─ Merge clocks │
│ └─ Validate state (Checkpoint D) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Key Files:**
- `operation-log-sync.service.ts` - Sync orchestration
- `operation-log-upload.service.ts` - Upload logic
- `operation-log-download.service.ts` - Download logic
---
## Area 4: Conflict Detection
Conflict detection uses vector clocks to determine causal relationships.
```
Remote Op Arrives
│
▼
┌───────────────────────┐
│ Get entity IDs from op│
└───────────────────────┘
│
┌───────────────┴───────────────┐
▼ ▼
For each entityId: All checked?
│ │
▼ ▼
┌───────────────────┐ Return { conflicts,
│ Build local │ nonConflicting }
│ frontier clock │
└───────────────────┘
│
▼
┌───────────────────┐
│ Compare clocks │
│ local vs remote │
└───────────────────┘
│
┌───────┼───────┬───────────┬──────────┐
▼ ▼ ▼ ▼ ▼
EQUAL GREATER LESS CONCURRENT
│ _THAN _THAN │
│ │ │ │
▼ ▼ ▼ ▼
Skip Skip Has local TRUE CONFLICT
(dup) (superseded) pending? → Collect
│
┌─────┴─────┐
▼ ▼
NO YES
│ │
▼ ▼
Apply CONFLICT
(needs pending)
```
**Comparison Results:**
| Comparison | Meaning | Has Local Pending? | Action |
| -------------- | ----------------- | ------------------ | ------------------------ |
| `EQUAL` | Same operation | N/A | Skip (duplicate) |
| `GREATER_THAN` | Local is newer | N/A | Skip (superseded remote) |
| `LESS_THAN` | Remote is newer | No | Apply remote |
| `LESS_THAN` | Remote is newer | Yes | Apply (remote dominates) |
| `CONCURRENT` | Neither dominates | No | Apply remote |
| `CONCURRENT` | Neither dominates | Yes | **TRUE CONFLICT** |
**Key Files:**
- `vector-clock.service.ts` - Vector clock management
- `conflict-resolution.service.ts` - Conflict detection logic
- `src/app/sync/util/vector-clock.ts` - Clock comparison
---
## Area 5: Conflict Resolution (LWW)
Last-Write-Wins automatically resolves conflicts using timestamps.
```
┌─────────────────────────────────────────────────────────────────┐
│ LWW RESOLUTION PIPELINE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ 1. CONFLICT DETECTED (vector clocks CONCURRENT) │
│ Local ops: [Op1, Op2] Remote ops: [Op3] │
│ │
│ 2. COMPARE TIMESTAMPS │
│ local_max = max(Op1.ts, Op2.ts) │
│ remote_max = max(Op3.ts) │
│ │
│ 3a. IF local_max > remote_max → LOCAL WINS │
│ ├─ Create new UPDATE op with: │
│ │ • Current state from NgRx store │
│ │ • Merged clock (all ops) + increment │
│ │ • Preserved original timestamp │
│ ├─ Reject old local ops (superseded clocks) │
│ ├─ Store remote ops, mark rejected │
│ └─ New op will sync on next cycle │
│ │
│ 3b. IF remote_max >= local_max → REMOTE WINS │
│ ├─ Apply remote ops to NgRx │
│ ├─ Reject local ops (including ALL pending for entity) │
│ └─ Remote state is now authoritative │
│ │
│ 4. VALIDATE STATE (Checkpoint D) │
│ Run validateAndRepairCurrentState() │
│ │
│ 5. NOTIFY USER │
│ "Auto-resolved X conflicts: Y local, Z remote wins" │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Key Invariants:**
| Invariant | Reason |
| --------------------------------- | -------------------------------------------------------------------------- |
| Archive-wins rule | `moveToArchive` always wins over field-level updates, bypassing timestamps |
| Preserve original timestamp | Prevents unfair advantage in future conflicts |
| Merge ALL clocks | New op dominates everything known |
| Reject ALL pending ops for entity | Prevents superseded ops from being uploaded |
| Mark rejected BEFORE applying | Crash safety |
| Remote wins on tie | Server-authoritative |
**Superseded Operation Special Cases:**
| Operation Type | Superseded Handling |
| --------------- | --------------------------------------------------------------------------------------------- |
| Regular UPDATE | Re-created with current entity state + merged clock |
| DELETE | Re-created with original payload + merged clock (entity gone from store) |
| `moveToArchive` | Re-created with original payload + merged clock (entity removed from NgRx by archive reducer) |
**Key Files:**
- `conflict-resolution.service.ts` - LWW resolution + archive-wins rule
- `superseded-operation-resolver.service.ts` - Superseded op handling (incl. moveToArchive special case)
---
## Area 6: SYNC_IMPORT Filtering
Clean slate semantics ensure imports restore all clients to the same state.
```
┌─────────────────────────────────────────────────────────────────┐
│ SYNC_IMPORT FILTERING PIPELINE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Remote Ops Received: [Op1, Op2, SYNC_IMPORT, Op3, Op4] │
│ │
│ 1. FIND IMPORTS │
│ ├─ Check current batch for SYNC_IMPORT/BACKUP_IMPORT/REPAIR│
│ └─ Check local store for previously downloaded import │
│ │
│ 2. DETERMINE LATEST IMPORT │
│ └─ Compare by UUIDv7 ID (time-ordered) │
│ │
│ 3. FOR EACH OP: │
│ ├─ Is it a full-state op? → ✅ Keep │
│ └─ Compare vectorClock with import's clock: │
│ ├─ GREATER_THAN → ✅ Keep (has knowledge) │
│ ├─ EQUAL → ✅ Keep (same history) │
│ ├─ LESS_THAN → ❌ Drop (dominated) │
│ └─ CONCURRENT → ❌ Drop (clean slate) │
│ │
│ 4. RETURN │
│ ├─ validOps: [SYNC_IMPORT, Op4] │
│ └─ invalidatedOps: [Op1, Op2, Op3] │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Why Vector Clocks, Not UUIDv7 Timestamps?**
| Approach | Problem |
| ----------------- | ------------------------------------------- |
| UUIDv7 Timestamps | Affected by clock drift |
| Vector Clocks | Track **causality** - immune to clock drift |
**Key Files:**
- `sync-import-filter.service.ts` - Filtering logic
---
## Area 7: Archive Handling
Archive data bypasses NgRx and is stored directly in IndexedDB.
```
┌─────────────────────────────────────────────────────────────────┐
│ ARCHIVE HANDLING PIPELINE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ LOCAL OPERATION │
│ ─────────────── │
│ 1. User completes task │
│ 2. ArchiveService writes to archiveYoung (BEFORE dispatch) │
│ 3. Dispatch moveToArchive action │
│ 4. Reducer updates NgRx state │
│ 5. ArchiveOperationHandlerEffects │
│ └─ handleOperation() → SKIP (already written) │
│ 6. Operation logged with payload (no archive data!) │
│ │
│ REMOTE OPERATION │
│ ──────────────── │
│ 1. Download moveToArchive operation │
│ 2. OperationApplierService.applyOperations() │
│ ├─ dispatch(bulkApplyOperations) → Reducer updates state │
│ └─ handleOperation() → Write to archiveYoung │
│ 3. Result: Same archive state as originating client │
│ │
│ KEY INSIGHT: Archive data NOT in operation payload! │
│ Each client executes the SAME deterministic logic locally. │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Why This Architecture?**
| Concern | Solution |
| ------------ | ----------------------------------------------- |
| Archive size | Don't sync archive data in operations |
| Consistency | Deterministic replay produces same result |
| Performance | Local writes, no network overhead |
| Sync safety | Operations carry timestamps for reproducibility |
**Key Files:**
- `archive-operation-handler.service.ts` - Unified handler
- `archive-operation-handler.effects.ts` - Local action routing
- `operation-applier.service.ts` - Calls handler for remote ops
---
## Area 8: Meta-Reducers
Meta-reducers enable atomic multi-entity changes in a single reducer pass.
```
┌─────────────────────────────────────────────────────────────────┐
│ META-REDUCER CHAIN (8 Phases, 15 Entries) │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Action Dispatched │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 1: operationCaptureMetaReducer [MUST BE FIRST] │ │
│ │ Captures original state BEFORE modifications │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 2: bulkOperationsMetaReducer │ │
│ │ Unwraps bulk dispatches for hydration/sync │ │
│ │ Pre-scans for archive ops to prevent │ │
│ │ resurrection via LWW Update │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 3: undoTaskDeleteMetaReducer │ │
│ │ Captures task context before deletion │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 4: Core CRUD Meta-Reducers (dependency order) │ │
│ │ • taskSharedCrudMetaReducer │ │
│ │ • taskBatchUpdateMetaReducer │ │
│ │ • taskSharedLifecycleMetaReducer │ │
│ │ • taskSharedSchedulingMetaReducer │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 5: Entity-Specific Cascades │ │
│ │ • projectSharedMetaReducer │ │
│ │ • tagSharedMetaReducer │ │
│ │ • issueProviderSharedMetaReducer │ │
│ │ • taskRepeatCfgSharedMetaReducer │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 6: plannerSharedMetaReducer │ │
│ │ Syncs with task.dueDay and TODAY_TAG │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 7: Synthetic Multi-Step Operations │ │
│ │ • shortSyntaxSharedMetaReducer │ │
│ │ • lwwUpdateMetaReducer │ │
│ │ (adapter / singleton / unsupported patterns) │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ Phase 8: actionLoggerReducer [MUST BE LAST] │ │
│ │ Pure logging only │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Final State │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Critical Ordering Rules:**
- `operationCaptureMetaReducer` must be at index 0 (captures pre-modification state)
- `bulkOperationsMetaReducer` must be at index 1 (unwraps bulk before other reducers run)
- `actionLoggerReducer` must be last (logs final state)
- Development mode validates these constraints at startup
**Why Meta-Reducers for Multi-Entity Changes?**
| Approach | Problem |
| ------------- | -------------------------------------------------------- |
| Effects | Multiple dispatches = multiple operations = partial sync |
| Meta-reducers | Single pass = single operation = atomic sync |
**Example: Tag Deletion Cascade**
```
deleteTag({ id: 'tag-1' })
│
▼ (tag-shared.reducer.ts)
┌─────────────────────────────────────────┐
│ 1. Remove tag from tag.ids │
│ 2. Remove tag from task.tagIds (all) │
│ 3. Remove tag from planner references │
│ 4. Update TODAY_TAG.taskIds if needed │
└─────────────────────────────────────────┘
│
▼
Single operation logged with all changes
```
**Key Files:**
- `meta-reducer-registry.ts` - Ordering documentation
- `tag-shared.reducer.ts` - Tag deletion cascade
- `project-shared.reducer.ts` - Project deletion cascade
- `planner-shared.reducer.ts` - Planner updates
---
## Area 9: Compaction
Compaction prevents the operation log from growing indefinitely.
```
┌─────────────────────────────────────────────────────────────────┐
│ COMPACTION PIPELINE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ TRIGGER │
│ ─────── │
│ Every 500 operations (COMPACTION_THRESHOLD) │
│ OR Emergency (storage quota exceeded) │
│ │
│ STEPS (all within OPERATION_LOG lock) │
│ ───── │
│ 1. Get current state from NgRx store │
│ │ │
│ 2. Get current vector clock │
│ │ │
│ 3. Get lastSeq IMMEDIATELY before saving │
│ │ │
│ 4. Extract snapshotEntityKeys from state │
│ (for conflict detection post-compaction) │
│ │ │
│ 5. Save snapshot to IndexedDB (state_cache): │
│ { │
│ state: currentState, │
│ lastAppliedOpSeq: lastSeq, │
│ vectorClock: currentVectorClock, │
│ compactedAt: Date.now(), │
│ schemaVersion: CURRENT_SCHEMA_VERSION, │
│ snapshotEntityKeys: [...] │
│ } │
│ │ │
│ 6. Reset compaction counter │
│ │ │
│ 7. Delete old operations WHERE: │
│ ├─ syncedAt IS SET (never drop unsynced ops!) │
│ ├─ appliedAt < cutoff (7 days, or 1 day emergency) │
│ └─ seq <= lastSeq (keep tail for conflict frontier) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Key Constants:**
| Constant | Value | Purpose |
| ----------------------------------- | ---------- | ------------------------- |
| `COMPACTION_THRESHOLD` | 500 ops | Triggers compaction |
| `COMPACTION_RETENTION_MS` | 7 days | Keep synced ops |
| `EMERGENCY_COMPACTION_RETENTION_MS` | 1 day | Aggressive cleanup |
| `COMPACTION_TIMEOUT_MS` | 25 seconds | Abort before lock expires |
**Safety Rules:**
| Rule | Why |
| ------------------------- | ------------------------- |
| Never delete unsynced ops | Would lose user data |
| Get lastSeq BEFORE saving | Race window safety |
| Keep ops within retention | Allow conflict resolution |
**Key Files:**
- `operation-log-compaction.service.ts` - Compaction logic
- `operation-log.effects.ts` - Trigger and counter
- `operation-log.const.ts` - Configuration constants
---
## Area 10: Bulk Application
Bulk application optimizes performance by applying many operations in a single dispatch.
```
┌─────────────────────────────────────────────────────────────────┐
│ BULK APPLICATION FLOW │
├─────────────────────────────────────────────────────────────────┤
│ │
│ WITHOUT BULK (naive): │
│ Op1 → dispatch → update → effects │
│ Op2 → dispatch → update → effects │
│ ... │
│ Op500 → dispatch → update → effects │
│ Result: 500 updates, 500 effect evaluations │
│ │
│ ───────────────────────────────────────────────────────────── │
│ │
│ WITH BULK (optimized): │
│ [Op1, Op2, ..., Op500] │
│ │ │
│ ▼ │
│ dispatch(bulkApplyOperations({ operations })) │
│ │ │
│ ▼ (in bulkOperationsMetaReducer) │
│ ┌────────────────────────────────────────┐ │
│ │ // Pre-scan: collect archived IDs │ │
│ │ archivedIds = findArchiveOps(ops) │ │
│ │ │ │
│ │ for (op of operations) { │ │
│ │ action = convertOpToAction(op) │ │
│ │ // Skip LWW Updates for archived │ │
│ │ if (isLwwUpdate(action) && │ │
│ │ archivedIds.has(entityId)) │ │
│ │ continue; │ │
│ │ state = reducer(state, action) │ │
│ │ } │ │
│ │ return state │ │
│ └────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ Single store update │
│ Effects see only: '[OperationLog] Bulk Apply Operations' │
│ │
│ Result: 1 update, 0 individual effect triggers (10-50x faster) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Why Effects Don't Fire:**
| Effect Type | Behavior |
| -------------- | ----------------------------------------------------------- |
| Action-based | Only see `bulkApplyOperations` (no listener) |
| Selector-based | Suppressed by `HydrationStateService.isApplyingRemoteOps()` |
**Key Files:**
- `bulk-hydration.meta-reducer.ts` - Core loop
- `bulk-hydration.action.ts` - Action definition
- `operation-converter.util.ts` - Op → Action conversion
- `operation-applier.service.ts` - Orchestration
---
## Area 11: Encryption (E2E)
End-to-end encryption ensures the server never sees plaintext data.
```
┌─────────────────────────────────────────────────────────────────┐
│ ENCRYPTION ARCHITECTURE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ CLIENT A SERVER CLIENT B │
│ ───────── ────── ───────── │
│ │
│ { task: "secret" } │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ Argon2id │ ← User password │
│ │ Key Derive │ (64MB memory-hard) │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ AES-256 │ │
│ │ GCM Encrypt│ │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ "base64..." ──────────► [Encrypted blob] ─────────► │
│ (server stores) │ │
│ ▼ │
│ ┌─────────────┐ │
│ │ AES-256 │ │
│ │ GCM Decrypt│ │
│ └─────────────┘ │
│ │ │
│ ▼ │
│ { task: "secret" } │
│ │
│ SERVER SEES: Encrypted base64 blobs only │
│ ZERO KNOWLEDGE: Server cannot read operation payloads │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**Encryption Parameters:**
| Parameter | Value |
| ----------------- | ----------------- |
| Algorithm | AES-256-GCM |
| Key derivation | Argon2id |
| Salt length | 16 bytes (random) |
| IV length | 12 bytes (random) |
| Argon2 memory | 64 MB |
| Argon2 iterations | 3 |
**Encrypted Blob Format:**
```
┌──────────────────────────────────────────────────────────┐
│ Salt (16B) │ IV (12B) │ Ciphertext + GCM Auth Tag │
└──────────────────────────────────────────────────────────┘
→ Encoded as base64 for transport
```
**Key Files:**
- `operation-encryption.service.ts` - High-level API
- `sync/encryption/encryption.ts` - AES-GCM + Argon2id
- `operation-log-upload.service.ts` - Encryption during upload
- `operation-log-download.service.ts` - Decryption during download
---
## Area 12: Unified File-Based Sync
All sync providers (WebDAV, Dropbox, LocalFile, SuperSync) now use the unified operation log system.
```
┌─────────────────────────────────────────────────────────────────┐
│ UNIFIED SYNC ARCHITECTURE │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ OperationLogSyncService │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │ │
│ ┌─────────────┴─────────────┐ │
│ ▼ ▼ │
│ ┌────────────────────┐ ┌────────────────────────────────┐ │
│ │ FileBasedSyncAdapter│ │ SuperSyncProvider │ │
│ │ (OperationSyncable) │ │ (OperationSyncable) │ │
│ │ │ │ │ │
│ │ ├─ uploadOps() │ │ ├─ uploadOps() │ │
│ │ ├─ downloadOps() │ │ ├─ downloadOps() │ │
│ │ └─ uploadSnapshot │ │ └─ uploadSnapshot() │ │
│ └────────────────────┘ └────────────────────────────────┘ │
│ │ │ │
│ ┌───────┬───┴───┬──────────┐ │ │
│ ▼ ▼ ▼ ▼ ▼ │
│ WebDAV Dropbox LocalFile ... SuperSync Server │
│ │
└─────────────────────────────────────────────────────────────────┘
```
**File-Based Sync Model (Unified):**
```
Remote Storage (WebDAV/Dropbox folder):
/superProductivity/
├── sync-data.json ← Single file with:
│ • Full state snapshot
│ • Recent ops buffer (200)
│ • Vector clock
│ • Archive data
└── sync-data.json.bak ← Backup of previous version
```
**All Providers Now Use Same Interface:**
| Aspect | File-Based (WebDAV/Dropbox/LocalFile) | SuperSync |
| ------------- | ------------------------------------- | --------------------- |
| Granularity | Individual operations | Individual operations |
| Conflict Unit | Single entity | Single entity |
| Resolution | Automatic (LWW) | Automatic (LWW) |
| Storage | Single sync-data.json | PostgreSQL |
| History | Recent 200 ops | Full op log |
**Key Files:**
- `op-log/sync-providers/file-based/file-based-sync-adapter.service.ts` - File-based adapter
- `op-log/sync-providers/file-based/file-based-sync.types.ts` - Types for sync-data.json
- `op-log/sync-providers/super-sync/super-sync.ts` - SuperSync provider
- `op-log/sync/operation-log-sync.service.ts` - Main sync orchestration
- `op-log/persistence/pfapi-migration.service.ts` - Legacy PFAPI migration
---
## File Reference
```
src/app/op-log/
├── core/ # Types, constants, errors
│ ├── operation.types.ts # Type definitions
│ ├── operation-log.const.ts # Constants
│ ├── persistent-action.interface.ts # Action interface
│ └── entity-registry.ts # Entity type registry
├── capture/ # Write path: Actions → Operations
│ ├── operation-capture.meta-reducer.ts # Captures persistent actions
│ ├── operation-capture.service.ts # FIFO queue
│ └── operation-log.effects.ts # Writes to IndexedDB
├── apply/ # Read path: Operations → State
│ ├── bulk-hydration.action.ts # Bulk apply action
│ ├── bulk-hydration.meta-reducer.ts # Applies ops in single pass
│ ├── operation-applier.service.ts # Apply ops to NgRx
│ ├── operation-converter.util.ts # Op → Action conversion
│ ├── hydration-state.service.ts # Tracks hydration state
│ └── archive-operation-handler.service.ts # Archive side effects
├── store/ # IndexedDB persistence
│ ├── operation-log-store.service.ts # IndexedDB wrapper
│ ├── operation-log-hydrator.service.ts # Startup hydration
│ ├── operation-log-compaction.service.ts # Snapshot + GC
│ └── schema-migration.service.ts # Schema migrations
├── sync/ # Server sync (SuperSync)
│ ├── operation-log-sync.service.ts # Sync orchestration
│ ├── operation-log-upload.service.ts # Upload logic
│ ├── operation-log-download.service.ts # Download logic
│ ├── conflict-resolution.service.ts # LWW resolution
│ ├── sync-import-filter.service.ts # SYNC_IMPORT filtering
│ ├── vector-clock.service.ts # Clock management
│ └── operation-encryption.service.ts # E2E encryption
├── validation/ # State validation
│ ├── validate-state.service.ts # State consistency checks
│ └── validate-operation-payload.ts # Operation validation
├── util/ # Shared utilities
│ ├── entity-key.util.ts # Entity key helpers
│ └── client-id.provider.ts # Client ID management
└── testing/ # Test infrastructure
├── integration/ # Integration tests
└── benchmarks/ # Performance tests
```

50
docs/sync-and-op-log/vector-clocks.md
View file

@ -360,3 +360,53 @@ Rules that must hold for the system to be correct. Use these to verify implement
| Server: conflict detection + prune after comparison | `packages/super-sync-server/src/sync/sync.service.ts` |
| Server: DoS cap (sanitize, no pruning) | `packages/super-sync-server/src/sync/services/validation.service.ts` |
| Server: snapshot clock pruning during download optimization | `packages/super-sync-server/src/sync/services/operation-download.service.ts` |
---
## 11. History & Rationale (why pruning is the way it is)
Decision-history behind the current pruning design (previously in a separate
research doc, now git-only). Load-bearing context for anyone changing
`MAX_VECTOR_CLOCK_SIZE` or the prune ordering.
### Compare before pruning — and the bugs that proved it
**Never prune a vector clock before using it in a comparison.** Pruning removes
information: a missing entry is ambiguous — "never knew about this client" vs
"entry was pruned" — so a pre-pruned comparison returns CONCURRENT instead of
EQUAL/causal. Two independent incidents established this:
- **Riak #613:** pruning before comparison caused "sibling explosion" — objects
accumulated hundreds of siblings that could never resolve because pruned
clocks always compared CONCURRENT.
- **Super Productivity (Feb 2026):** with `MAX = 10`, server pruning before
comparison caused an infinite rejection loop — a client merges all clocks +
its own ID (11 entries), the server prunes to 10, the non-shared key forces
CONCURRENT, the server rejects, the client re-merges, the loop repeats.
Fix in both systems: compare the **full unpruned** clock, then prune **only
before storage**. This is the invariant in §6 and §9.
### Why MAX = 20 (the 10 → 30 → 20 evolution)
The original defense against the Feb-2026 loop was a 4-layer scheme (protected
client IDs, pruning-aware comparison, an `isLikelyPruningArtifact` heuristic,
the same-client check) — symptom treatment. The root cause was that `MAX = 10`
was too small, making pruning frequent and interacting badly with SYNC_IMPORT.
Commit `d70f18a94d` raised `MAX` 10 → 30 (later reduced to 20 — a 20-entry
clock is ~333 bytes, negligible) and **removed three of the four layers**.
`isLikelyPruningArtifact` was dropped (known false positives, unnecessary at
MAX = 20). Only the **same-client check** remains — always mathematically
correct (monotonic counters are definitive) and independent of MAX. At
MAX = 20, pruning needs **21+ distinct client IDs**, extremely rare for a
personal productivity app, so the pruning path is effectively dormant
(see §5 "Pruning is Rare").
### Future options (only if the server becomes the coordinator)
In a server-authoritative model, clock growth could be bounded without pruning
via **Dotted Version Vectors** (bound to server vnodes, not devices),
**bounded reclaimable client IDs** (needs a registration/retirement protocol),
or **periodic stable-cut GC** (needs all-to-all clock reporting). None apply to
the current dumb-relay model.

2
eslint-local-rules/index.js
View file

@ -12,4 +12,6 @@
module.exports = {
'require-hydration-guard': require('./rules/require-hydration-guard'),
'require-entity-registry': require('./rules/require-entity-registry'),
'no-actions-in-effects': require('./rules/no-actions-in-effects'),
'no-multi-entity-effect': require('./rules/no-multi-entity-effect'),
};

70
eslint-local-rules/rules/no-actions-in-effects.js Normal file
View file

@ -0,0 +1,70 @@
/**
* ESLint rule: no-actions-in-effects
*
* Effects must not use the raw `@ngrx/effects` `Actions` stream. They must
* inject `LOCAL_ACTIONS` (the default — remote/replayed ops filtered out) or,
* only for the op-log/archive effects that handle `isRemote` themselves,
* `ALL_ACTIONS`.
*
* This enforces Boundary 1 of the single sync invariant: replayed and remote
* operations must never re-trigger effects (duplicate side effects + phantom
* operations). See docs/sync-and-op-log/contributor-sync-model.md.
*
* Scoped to every `.effects.ts` file via eslint.config.js. The codebase is
* already 100% compliant — this is a regression guard, not a migration.
*
* Flags:
* - `import { Actions } from '@ngrx/effects'` (including aliased imports)
* - `inject(Actions)` (including the aliased local binding)
*/
module.exports = {
meta: {
type: 'problem',
docs: {
description:
'Effects must inject LOCAL_ACTIONS/ALL_ACTIONS, never the raw @ngrx/effects Actions stream',
category: 'Possible Errors',
recommended: true,
},
messages: {
noActionsImport:
"Do not import `Actions` from '@ngrx/effects' in an effect. Inject `LOCAL_ACTIONS` (default) — or `ALL_ACTIONS` only for op-log/archive effects. See docs/sync-and-op-log/contributor-sync-model.md.",
noActionsInject:
'Do not `inject(Actions)` in an effect. Use `inject(LOCAL_ACTIONS)` (default) — or `inject(ALL_ACTIONS)` only for op-log/archive effects. See docs/sync-and-op-log/contributor-sync-model.md.',
},
schema: [],
},
create(context) {
// Local binding name(s) that `Actions` from '@ngrx/effects' was imported as.
const actionsLocalNames = new Set();
return {
ImportDeclaration(node) {
if (node.source.value !== '@ngrx/effects') return;
for (const spec of node.specifiers) {
if (
spec.type === 'ImportSpecifier' &&
spec.imported &&
spec.imported.name === 'Actions'
) {
actionsLocalNames.add(spec.local.name);
context.report({ node: spec, messageId: 'noActionsImport' });
}
}
},
CallExpression(node) {
if (node.callee.type !== 'Identifier' || node.callee.name !== 'inject') {
return;
}
const arg = node.arguments[0];
if (!arg || arg.type !== 'Identifier') return;
// Catch the literal name and any aliased binding of @ngrx/effects Actions.
if (arg.name === 'Actions' || actionsLocalNames.has(arg.name)) {
context.report({ node, messageId: 'noActionsInject' });
}
},
};
},
};

71
eslint-local-rules/rules/no-actions-in-effects.spec.js Normal file
View file

@ -0,0 +1,71 @@
/**
* Tests for no-actions-in-effects ESLint rule
*/
const { RuleTester } = require('eslint');
const rule = require('./no-actions-in-effects');
const ruleTester = new RuleTester({
languageOptions: {
ecmaVersion: 2020,
sourceType: 'module',
},
});
ruleTester.run('no-actions-in-effects', rule, {
valid: [
// The blessed default: inject(LOCAL_ACTIONS)
{
code: `
import { createEffect, ofType } from '@ngrx/effects';
const actions$ = inject(LOCAL_ACTIONS);
`,
},
// The special-case escape hatch: inject(ALL_ACTIONS)
{
code: `
import { createEffect } from '@ngrx/effects';
const actions$ = inject(ALL_ACTIONS);
`,
},
// A local symbol coincidentally named Actions NOT from @ngrx/effects:
// still flagged by the literal-name guard is acceptable; here we only
// assert the common safe imports are clean.
{
code: `
import { Store } from '@ngrx/store';
const store = inject(Store);
`,
},
],
invalid: [
// Importing Actions from @ngrx/effects
{
code: `import { Actions, ofType } from '@ngrx/effects';`,
errors: [{ messageId: 'noActionsImport' }],
},
// Importing + injecting Actions (two distinct violations)
{
code: `
import { Actions } from '@ngrx/effects';
const a$ = inject(Actions);
`,
errors: [{ messageId: 'noActionsImport' }, { messageId: 'noActionsInject' }],
},
// Aliased import + injecting the alias
{
code: `
import { Actions as Acts } from '@ngrx/effects';
const a$ = inject(Acts);
`,
errors: [{ messageId: 'noActionsImport' }, { messageId: 'noActionsInject' }],
},
// Bare inject(Actions) without a tracked import still flagged by name
{
code: `const a$ = inject(Actions);`,
errors: [{ messageId: 'noActionsInject' }],
},
],
});
// eslint-disable-next-line no-console
console.log('no-actions-in-effects: all RuleTester cases passed');

99
eslint-local-rules/rules/no-multi-entity-effect.js Normal file
View file

@ -0,0 +1,99 @@
/**
* ESLint rule: no-multi-entity-effect (heuristic, warn)
*
* Flags ONE narrow syntactic shape: an effect whose dispatch arm returns a
* *literal array* of >=2 action-creator calls directly as an arrow body or a
* `return` argument, e.g. `map(() => [updateProject(), updateTask()])`. One
* user intent should become exactly ONE operation; a multi-entity change must
* live in a meta-reducer (src/app/root-store/meta/task-shared-meta-reducers/),
* not an effect that emits N follow-up actions — an effect fan-out produces N
* ops for one intent and re-runs on replay.
*
* Deliberately NOT detected (covered by the contributor doc + code review;
* pinned as `valid` cases in the spec so the boundary is explicit and a future
* change that starts catching them trips the spec):
* - varargs `of(a(), b())` — not an ArrayExpression
* - `from([a(), b()])` — array wrapped in a call
* - ternary / conditionally returned arrays
* - imperative `store.dispatch(a()); store.dispatch(b())`
*
* It is a `warn` heuristic, not a guarantee: a clean run does NOT prove an
* effect is single-op. If a flagged multi-dispatch is intentionally not synced
* state, disable the line with a justification comment.
*
* See docs/sync-and-op-log/contributor-sync-model.md (the atomicity rule).
*/
module.exports = {
meta: {
type: 'suggestion',
docs: {
description:
'Effects should not dispatch a multi-action fan-out; multi-entity changes belong in a meta-reducer',
category: 'Best Practices',
recommended: false,
},
messages: {
multiEntityEffect:
'This effect returns a fan-out of {{count}} actions for one trigger. One user intent = one operation: move multi-entity changes into a meta-reducer (src/app/root-store/meta/task-shared-meta-reducers/). If this is intentionally not synced state, disable with a justification. See docs/sync-and-op-log/contributor-sync-model.md.',
},
schema: [],
},
create(context) {
const reported = new Set();
/**
* An array of >=2 call expressions returned from an effect stream is the
* fan-out smell (e.g. `map(() => [actionA(), actionB()])`).
*/
const checkArray = (arrNode) => {
if (!arrNode || arrNode.type !== 'ArrayExpression') return;
const calls = arrNode.elements.filter((el) => el && el.type === 'CallExpression');
if (calls.length >= 2 && !reported.has(arrNode)) {
reported.add(arrNode);
context.report({
node: arrNode,
messageId: 'multiEntityEffect',
data: { count: String(calls.length) },
});
}
};
return {
CallExpression(node) {
if (node.callee.type !== 'Identifier' || node.callee.name !== 'createEffect') {
return;
}
const seen = new Set();
const walk = (n) => {
if (!n || typeof n.type !== 'string' || seen.has(n)) return;
seen.add(n);
if (
n.type === 'ArrowFunctionExpression' &&
n.body &&
n.body.type === 'ArrayExpression'
) {
checkArray(n.body);
}
if (
n.type === 'ReturnStatement' &&
n.argument &&
n.argument.type === 'ArrayExpression'
) {
checkArray(n.argument);
}
for (const key of Object.keys(n)) {
if (key === 'parent') continue;
const child = n[key];
if (Array.isArray(child)) {
child.forEach(walk);
} else if (child && typeof child.type === 'string') {
walk(child);
}
}
};
walk(node);
},
};
},
};

126
eslint-local-rules/rules/no-multi-entity-effect.spec.js Normal file
View file

@ -0,0 +1,126 @@
/**
* Tests for no-multi-entity-effect ESLint rule (heuristic, warn)
*/
const { RuleTester } = require('eslint');
const rule = require('./no-multi-entity-effect');
const ruleTester = new RuleTester({
languageOptions: {
ecmaVersion: 2020,
sourceType: 'module',
},
});
ruleTester.run('no-multi-entity-effect', rule, {
valid: [
// Single dispatched action — fine.
{
code: `
const e$ = createEffect(() =>
actions$.pipe(
ofType(trigger),
map(() => doOneThing())
)
);
`,
},
// BLESSED PATH: a multi-entity change routed through a meta-reducer.
// The effect dispatches ONE action; the fan-out happens in the reducer.
{
code: `
const deleteTag$ = createEffect(() =>
actions$.pipe(
ofType(deleteTag),
map(({ id }) => deleteTagAndRemoveFromTasks({ id }))
)
);
`,
},
// A single-element array is not a fan-out.
{
code: `
const e$ = createEffect(() =>
actions$.pipe(map(() => [onlyOne()]))
);
`,
},
// Not a createEffect call — ignored.
{
code: `const arr = [actionA(), actionB()];`,
},
// --- Documented heuristic blind spots (see the rule's JSDoc and
// contributor-sync-model.md). These ARE multi-entity fan-outs but are
// intentionally NOT detected — pinned here so the boundary is explicit
// and any future change that starts catching them trips this spec.
{
// varargs `of(a(), b())` — not an ArrayExpression
code: `
const e$ = createEffect(() =>
actions$.pipe(switchMap(() => of(updateProject(), updateTask())))
);
`,
},
{
// `from([a(), b()])` — array wrapped in a CallExpression
code: `
const e$ = createEffect(() =>
actions$.pipe(mergeMap(() => from([updateProject(), updateTask()])))
);
`,
},
{
// ternary-returned arrays — the array is not the direct body/return arg
code: `
const e$ = createEffect(() =>
actions$.pipe(map(() => (cond ? [a(), b()] : [c(), d()])))
);
`,
},
{
// imperative store.dispatch fan-out — no returned array at all
code: `
const e$ = createEffect(
() =>
actions$.pipe(
tap(() => {
store.dispatch(a());
store.dispatch(b());
}),
),
{ dispatch: false },
);
`,
},
],
invalid: [
// Classic fan-out: effect returns >=2 action-creator calls.
{
code: `
const e$ = createEffect(() =>
actions$.pipe(
ofType(trigger),
map(() => [updateProject(), updateTask()])
)
);
`,
errors: [{ messageId: 'multiEntityEffect' }],
},
// Fan-out via explicit return statement.
{
code: `
const e$ = createEffect(() =>
actions$.pipe(
mergeMap(() => {
return [removeFromTag(), updateTaskOrder(), reIndex()];
})
)
);
`,
errors: [{ messageId: 'multiEntityEffect' }],
},
],
});
// eslint-disable-next-line no-console
console.log('no-multi-entity-effect: all RuleTester cases passed');

70
eslint-local-rules/run-specs.js Normal file
View file

@ -0,0 +1,70 @@
/**
* Runs every `*.spec.js` in eslint-local-rules/rules via ESLint's RuleTester.
*
* RuleTester.run() executes synchronously and throws on the first failing
* case ONLY when no test-framework globals (describe/it) are present — with
* them, RuleTester defers cases to the framework and merely requiring the spec
* asserts nothing. This runner therefore:
* (a) refuses to run if such globals are present, and
* (b) counts RuleTester.run() invocations per spec and fails any spec that
* required cleanly but never called run() (e.g. run() commented out, or
* wrapped in a never-invoked function) — so a spec can no longer be a
* silent pass that asserts nothing.
*
* Wired as `npm run test:lint-rules` (the unit suite — Karma over *.spec.ts —
* does not run these .spec.js files). Run with bare `node`, never under
* jest/mocha.
*/
const fs = require('fs');
const path = require('path');
const { RuleTester } = require('eslint');
// (a) A test-framework global means RuleTester would NOT throw synchronously,
// so requiring a spec would assert nothing. Fail loudly instead of lying.
if (typeof globalThis.describe === 'function' || typeof globalThis.it === 'function') {
// eslint-disable-next-line no-console
console.error(
'lint-rule specs: REFUSING TO RUN — a test-framework global (describe/it) ' +
'is present, so RuleTester defers instead of throwing and nothing would ' +
'be asserted. Run with bare `node eslint-local-rules/run-specs.js`.',
);
process.exit(1);
}
// (b) Count run() calls so a spec that asserts nothing fails instead of passing.
let runCalls = 0;
const originalRun = RuleTester.prototype.run;
RuleTester.prototype.run = function countedRun(...args) {
runCalls += 1;
return originalRun.apply(this, args);
};
const rulesDir = path.join(__dirname, 'rules');
const specs = fs.readdirSync(rulesDir).filter((f) => f.endsWith('.spec.js'));
let failed = false;
for (const spec of specs) {
const runsBefore = runCalls;
try {
require(path.join(rulesDir, spec));
if (runCalls === runsBefore) {
failed = true;
// eslint-disable-next-line no-console
console.error(
`FAIL ${spec}\n required cleanly but never called RuleTester.run() — asserted nothing`,
);
}
} catch (err) {
failed = true;
// eslint-disable-next-line no-console
console.error(`FAIL ${spec}\n${(err && err.stack) || err}`);
}
}
// eslint-disable-next-line no-console
console.log(
failed
? '\nlint-rule specs: FAILED'
: `\nlint-rule specs: ${specs.length} file(s) passed (${runCalls} RuleTester.run call(s))`,
);
process.exit(failed ? 1 : 0);

2
eslint.config.js
View file

@ -220,6 +220,8 @@ module.exports = tseslint.config(
rules: {
'local-rules/require-hydration-guard': 'error',
'local-rules/require-entity-registry': 'warn',
'local-rules/no-actions-in-effects': 'error',
'local-rules/no-multi-entity-effect': 'warn',
},
},
// HTML files

3
package.json
View file

@ -103,7 +103,8 @@
"int:test": "node ./tools/test-lng-files.js",
"int:unused": "node ./tools/find-unused-translations.js",
"int:watch": "node ./tools/extract-i18n-watch.js",
"lint": "npm run lint:ts && npm run lint:scss",
"lint": "npm run lint:ts && npm run lint:scss && npm run test:lint-rules",
"test:lint-rules": "node eslint-local-rules/run-specs.js",
"lint:ts": "ng lint",
"lint:scss": "stylelint \"**/*.scss\" -- --custom-formatter @csstools/stylelint-formatter-github",
"localInstall": "sudo echo 'Starting local install' && rm -Rf ./.tmp/angular-dist/ && rm -Rf ./.tmp/app-builds/ && npm run buildAllElectron:stage && electron-builder --linux deb && sudo dpkg -i .tmp/app-builds/superProductivity*.deb",

6
src/app/util/local-actions.token.ts
View file

@ -59,9 +59,11 @@ export const LOCAL_ACTIONS = new InjectionToken<Observable<Action>>('LOCAL_ACTIO
*
* Only use this for effects that MUST react to remote operations:
* - operation-log.effects.ts: Captures and persists all actions (handles isRemote internally)
* - archive-operation-handler.effects.ts: Refreshes worklog UI after remote archive ops
*
* For all other effects, use LOCAL_ACTIONS instead.
* For all other effects, use LOCAL_ACTIONS instead. Note: remote-client
* archive side effects are NOT an ALL_ACTIONS case — they are driven by
* OperationApplierService -> ArchiveOperationHandler, and
* archive-operation-handler.effects.ts itself uses LOCAL_ACTIONS.
*/
export const ALL_ACTIONS = new InjectionToken<Observable<Action>>('ALL_ACTIONS', {
providedIn: 'root',