mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-07-17 16:37:43 +00:00
docs(sync): fix stale schema-compat docs, add bump policy, rescope Task 6 (#9119)
* docs(sync): fix stale schema docs, add bump policy, rescope Task 6 * docs(sync): add severity-triage and schema-bump rules to AGENTS.md * docs(sync): fix review findings in schema compat docs
This commit is contained in:
Johannes Millan
GitHub
parent
6756e9998c
commit
99f923ad6c
7 changed files with 130 additions and 47 deletions
36
AGENTS.md
36
AGENTS.md
|
|
@ -71,6 +71,42 @@ Touched on most state-related PRs. Read the linked source/doc for full reasoning
|
|||
7. **`SYNC_IMPORT` / `BACKUP_IMPORT`** replace state and intentionally drop concurrent ops (CONCURRENT or LESS_THAN by vector clock) — by design, not a bug. → `SyncImportFilterService`.
|
||||
8. **Vector clocks:** `MAX_VECTOR_CLOCK_SIZE = 20`. Server prunes after conflict detection, before storage. → `docs/sync-and-op-log/vector-clocks.md`.
|
||||
9. **Logging:** `Log.log({ id: task.id })`, never `Log.log(task)` or `Log.log(title)` — log history is exportable, never log user content.
|
||||
10. **A schema bump never protects the released fleet.** v17.0.0–v18.14.0 clients apply ops up to schema 5 UNMIGRATED (their old `+3` skip band) and, at schema ≥ 6, block them but still advance the server cursor — those ops are skipped permanently, even after updating. Only post-v18.14.0 receivers block newer ops safely. So new op semantics MUST degrade gracefully on older clients (`LwwUpdatePayload` envelope pattern); a change old clients would misapply must not ship behind a bump alone. → `packages/shared-schema/src/schema-version.ts`, [operation-log-architecture.md](docs/sync-and-op-log/operation-log-architecture.md) §A.7.11 "Bump Policy".
|
||||
|
||||
## Judging sync severity
|
||||
|
||||
Rules for triage — how to decide whether a sync bug is real and how bad it is. Each one below is
|
||||
here because getting it wrong already produced a confidently wrong conclusion.
|
||||
|
||||
1. **`master` ships to real users. "It's only on master" never downgrades severity.** Every master
|
||||
push auto-publishes to the Play **internal track** (`.github/workflows/build-android.yml`,
|
||||
`tracks: internal` + `status: completed` → testers' phones auto-update within minutes, on their
|
||||
real data). `ghcr.io/super-productivity/supersync:latest` **is** master and has no
|
||||
release-tagged build at all — it is the default in `packages/super-sync-server/docker-compose.yml`,
|
||||
so self-hosters on `docker compose pull` run master HEAD. Snap `edge` is also published from
|
||||
every master push. Only desktop/web/F-Droid/Play-production/Snap-stable are release-gated.
|
||||
2. **Never infer "shipped" from dates or the latest tag — prove it.** Use
|
||||
`git merge-base --is-ancestor <commit> v<tag>` / `git tag --contains <commit>`. Tags are cut from
|
||||
a point in time, and sync features routinely land just after: **#8874's disjoint-field merge
|
||||
landed ~24h after v18.14.0 was tagged and is in no release**, so whole-entity-LWW field loss
|
||||
(rename dies when another device marks the task done, #9095) is live in **every shipped version**.
|
||||
3. **"Restores released behavior" ≠ safe. The released behavior can be the bug.** #9061 froze the
|
||||
disjoint merge on exactly that reasoning and silently re-armed shipped data loss (#9095).
|
||||
A freeze/revert needs the same "what breaks for users?" analysis as a feature.
|
||||
4. **Users do report sync bugs — in non-technical words. There is no `sync` label.** Keyword-grepping
|
||||
`sync`/`op-log`/`conflict` undercounts by ~50×. Search what users actually write: _lost,
|
||||
disappeared, gone, missing, duplicate, reverted, old version, overwritten, reset, not syncing_
|
||||
(#7892 "all data deleted overnight"; #8107 user rebuilt lost projects from memory; #7549 done
|
||||
tasks resurrecting). ~53 user-reported sync/data-loss issues from 44 authors in 90 days ≈ one
|
||||
every 2 days. And silent data loss is structurally under-reported — absence of reports is never
|
||||
evidence of absence.
|
||||
5. **Audit-generated findings are low-precision, not low-yield — verify them, don't dismiss them.**
|
||||
~89% of sync fixes since v18.14.0 repaired code present in the release, yet ~97% of the self-filed
|
||||
sync issues carried no reproduction. So both failure modes are live: **do not close an unreproduced
|
||||
finding as speculation** (#8960/#9073/#8751/#9040 had no repro and were all real and shipped), and
|
||||
**do not fix one blind** — the _fix_ must carry a test that fails without it, and you must confirm
|
||||
the fix actually fires on a real op (#9045 shipped an `entityIds` security check that **never fired**;
|
||||
#9025 was self-retracted as "not a live data-loss bug"). The reproduction gates the _fix_, not belief.
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Sync Simplification Plan
|
||||
|
||||
**Status:** Re-aligned to `origin/master` 7ef7e69e96 on 2026-07-16. The 2026-07-15 re-align (baseline 6fefd741c5) covered 15 commits of sync fixes landing in/around every task surface — #9023 (REPAIR concurrency guard) and #9040 (immutable split snapshots) in Task 2's adapter, #9028 (WS local-win re-upload) in Task 4's shell, #9045 (decrypt-path footprint auth + journal privacy), #9048 (notes/sections/repeat-cfg cascade recovery) and #9035 (clientId LWW tiebreak) in Task 6's territory, #9026 (non-interactive in-lock repair) in the critical section. Master has since advanced 3 further commits, of which exactly one touches a task surface: **#9054** (authenticated LWW project-move footprint, GHSA-8pxh-mgc7-gp3g) lands in `conflict-resolution.service.ts` and is folded into contract 14 and §4's fence list below. The other two are out of scope (#9052 orphan-task navigation heal, #9056 locale date names). Claims below verified against `origin/master`; #9044 was dialog-layer only and did not touch the file adapter.
|
||||
**Status:** Re-aligned to `origin/master` 7ef7e69e96 on 2026-07-16. The 2026-07-15 re-align (baseline 6fefd741c5) covered 15 commits of sync fixes landing in/around every task surface — #9023 (REPAIR concurrency guard) and #9040 (immutable split snapshots) in Task 2's adapter, #9028 (WS local-win re-upload) in Task 4's shell, #9045 (decrypt-path footprint auth + journal privacy), #9048 (notes/sections/repeat-cfg cascade recovery) and #9035 (clientId LWW tiebreak) in Task 6's territory, #9026 (non-interactive in-lock repair) in the critical section. Master has since advanced 3 further commits, of which exactly one touches a task surface: **#9054** (authenticated LWW project-move footprint, GHSA-8pxh-mgc7-gp3g) lands in `conflict-resolution.service.ts` and is folded into contract 14 and §4's fence list below. The other two are out of scope (#9052 orphan-task navigation heal, #9056 locale date names). Claims below verified against `origin/master`; #9044 was dialog-layer only and did not touch the file adapter. **Revised 2026-07-17:** Task 6 rescoped — the disjoint-field merge is load-bearing against whole-entity LWW data loss (#9095, producer unfrozen in #9101) and is now excluded from the rollback; the rollback covers the conflict journal and review UI only. Journal removal ≠ merge removal.
|
||||
|
||||
**Date:** 2026-07-16
|
||||
|
||||
|
|
@ -22,7 +22,7 @@ The smallest safe order is:
|
|||
6. remove only the conflict-review producers and UI that the deployment/persisted-data audit proves disposable;
|
||||
7. correct current sync documentation after behavior settles.
|
||||
|
||||
**Timing constraint:** the conflict-review feature (962c5bbeb1, merged 2026-07-11) is on master but in no release tag, and releases ship every one to two weeks. Every stable release cut before Task 6 expands the persisted-data obligation from edge/dogfood cohorts to the whole fleet. If the Task 1 audit authorizes deletion, land a minimal producer freeze — stop conflict-journal writes and disable the disjoint-merge producer — before the next release cut, ahead of Phase 1. The disjoint-merge gate (`disableDisjointMerge`) already exists but is currently a per-call option set only by the internal failed-merge fallback, so the freeze is a small wiring change at the `autoResolveConflictsLWW` call sites (or a global flag feeding it), not a ready toggle. The freeze is a small reversible diff; the full rollback (Task 6) then proceeds on its own schedule.
|
||||
**Timing constraint:** the conflict-review feature (962c5bbeb1, merged 2026-07-11) is on master but in no release tag, and releases ship every one to two weeks. The journal half of the producer freeze is already in place on master (`disableConflictJournal: true` at the production entry point in `RemoteOpsProcessingService`), so stable releases do not expand the persisted-data obligation. The disjoint-merge half of the original #9061 freeze was deliberately reverted by #9101 and MUST NOT be re-applied: with the merge disabled, concurrent edits to different fields of one entity resolve by whole-entity LWW and the earlier side's edit is silently and permanently lost on every client (#9095 — live in every release up to v18.14.0). The merge producer ships enabled; only the journal stays frozen for stable.
|
||||
|
||||
Atomic browser startup and replacement of flag-and-poll maintenance exclusion remain worthwhile correctness projects, but they are not prerequisites for deleting the two partial SuperSync pipelines. Keeping them separate avoids turning a bounded simplification into a cross-tab/bootstrap and every-import-owner rewrite.
|
||||
|
||||
|
|
@ -31,7 +31,7 @@ Atomic browser startup and replacement of flag-and-poll maintenance exclusion re
|
|||
| Surface | Verdict | Cost-benefit conclusion |
|
||||
| --------------------------------------------------------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Schema v3/v4, v2-to-v3 and v3-to-v4 barriers, IndexedDB version 10, and compatibility readers | **Keep** | Removing them can reject stored operations, misresolve marked project deletion, or silently diverge. Their maintenance cost is tiny relative to data-loss risk. |
|
||||
| Disjoint-merge planner/writer | **Conditional removal; high benefit** | Removing the experimental producer reduces conflict states. Keep both payload readers because deployed non-stable builds and persisted providers may already contain patch operations. Preserve the separate schema-v4 project-delete-wins planner/classifier and historical losing-delete recovery. |
|
||||
| Disjoint-merge planner/writer | **Keep (rescoped 2026-07-17)** | Removal re-introduces #9095: without the merge, concurrent edits to different fields of one entity resolve by whole-entity LWW and the earlier side's edit is silently lost on every client. The producer was unfrozen in #9101 and ships enabled. Keep both payload readers; preserve the separate schema-v4 project-delete-wins planner/classifier and historical losing-delete recovery. |
|
||||
| Conflict journal writer and review UI/route/banner/badge | **Conditional removal; high benefit** | No stable tag contains the feature, but master edge artifacts, Android internal builds, and previews may contain device-local discarded values. Task 1 decides the supported-data obligation. |
|
||||
| Conflict journal database/reader | **Keep until the data obligation expires** | It may hold the only copy of discarded values. A temporary read/export path is cheaper than silently stranding supported data. |
|
||||
| Flip-specific capture/replay handling | **Nothing to remove** | Flip dispatches an ordinary synced entity update. Remove UI references only, never generic update capture/replay. |
|
||||
|
|
@ -61,7 +61,7 @@ The previous broad “remove roughly 6,000 lines together” decision is rejecte
|
|||
11. Archive read-modify-write operations and remote/import replacements serialize through the `TASK_ARCHIVE` boundary. (Known residual: `TimeTrackingService` project/tag cleanup runs outside this mutex — #8941; do not treat that gap as resolved.)
|
||||
12. Raw rebuild resumes before ordinary sync; reducer-pending work blocks sync.
|
||||
13. Import/rebuild retain preflight-before-mutation, backup, atomic replacement, cursor, and crash-resume ordering.
|
||||
14. Encryption fails closed. `isPayloadEncrypted` is not a trusted policy input (enforced at the download boundary). On the decrypt path an encrypted LWW op's `entityId` (bound to authenticated `payload.id`) and, when the payload carries `projectMoveSubTaskIds`, its multi-task project-move footprint `entityIds` (bound to `{entityId} ∪ projectMoveSubTaskIds`) are validated against the GCM-authenticated payload and rejected on mismatch (`verify-decrypted-op-integrity.ts`, #9045). Synthetic conflict-resolution LWW ops (carrying `entityIds` without an authenticated `projectMoveSubTaskIds`) are intentionally exempt from that check; that exemption must survive the disjoint-merge removal. The exemption is safe only because no relocation decision ever reads a plaintext envelope (#9054, GHSA-8pxh-mgc7-gp3g): `getTaskProjectMoveEntityIds` (`conflict-resolution.service.ts`) re-derives a footprint from the authenticated `payload.projectMoveFootprint` for LWW ops, and for raw `TASK_SHARED_UPDATE` ops from `actionPayload.projectMoveSubTaskIds` plus a root of `actionPayload.task.id` — never `op.entityId`/`op.entityIds` — so a tampered remote envelope cannot be laundered into the freshly-authenticated merged op. That single choke point serves the disjoint-merge, local-win, and superseded-op callers, and both LWW-TASK reducers consume only `meta.projectMoveFootprint` (via `parseMoveFootprint`), with footprint-less legacy ops falling back to receiving-state repair. **Task 6 must keep this choke point intact and must not reintroduce an `op.entityIds` read when disjoint-merge goes** — doing so reopens the vector one merge removed. Ops minted in conflict resolution deliberately write the footprint to both `payload.projectMoveFootprint` and `op.entityIds` (the server needs plaintext ids for its indexed conflict detection and cannot read the ciphertext). The surviving envelope `entityIds` readers (`bulk-archive-filter.util.ts`, `get-op-entity-ids.util.ts`) are suppression-only, not relocation, and remain pending the durable AAD hardening.
|
||||
14. Encryption fails closed. `isPayloadEncrypted` is not a trusted policy input (enforced at the download boundary). On the decrypt path an encrypted LWW op's `entityId` (bound to authenticated `payload.id`) and, when the payload carries `projectMoveSubTaskIds`, its multi-task project-move footprint `entityIds` (bound to `{entityId} ∪ projectMoveSubTaskIds`) are validated against the GCM-authenticated payload and rejected on mismatch (`verify-decrypted-op-integrity.ts`, #9045). Synthetic conflict-resolution LWW ops (carrying `entityIds` without an authenticated `projectMoveSubTaskIds`) are intentionally exempt from that check; that exemption must survive the Task 6 rollback (the disjoint merge itself now stays). The exemption is safe only because no relocation decision ever reads a plaintext envelope (#9054, GHSA-8pxh-mgc7-gp3g): `getTaskProjectMoveEntityIds` (`conflict-resolution.service.ts`) re-derives a footprint from the authenticated `payload.projectMoveFootprint` for LWW ops, and for raw `TASK_SHARED_UPDATE` ops from `actionPayload.projectMoveSubTaskIds` plus a root of `actionPayload.task.id` — never `op.entityId`/`op.entityIds` — so a tampered remote envelope cannot be laundered into the freshly-authenticated merged op. That single choke point serves the disjoint-merge, local-win, and superseded-op callers, and both LWW-TASK reducers consume only `meta.projectMoveFootprint` (via `parseMoveFootprint`), with footprint-less legacy ops falling back to receiving-state repair. **Task 6 must keep this choke point intact and must not reintroduce an `op.entityIds` read** — doing so reopens the vector one merge removed. Ops minted in conflict resolution deliberately write the footprint to both `payload.projectMoveFootprint` and `op.entityIds` (the server needs plaintext ids for its indexed conflict detection and cannot read the ciphertext). The surviving envelope `entityIds` readers (`bulk-archive-filter.util.ts`, `get-op-entity-ids.util.ts`) are suppression-only, not relocation, and remain pending the durable AAD hardening.
|
||||
15. File revisions, expected sync versions, and vector-clock baselines remain staged until durable apply and cursor commit. Delete-all ordering across split, tombstone, and backup artifacts remains. Known residual: `_deleteAllData` does not enumerate/remove `STATE_GEN_FILE_PREFIX` immutable snapshots (no `listFiles`; `listFiles`-prune is the documented backstop) — the Task 2 reset helper must account for this remote-file gap.
|
||||
16. Pending local writes flush before the conflict frontier is read. Conflict planning/execution remain inside the operation-log critical section. In-lock validation/repair runs non-interactively — no native `confirm()`/`alert()` blocks the `sp_op_log` critical section during background sync (#9026); only the pre-lock, foreground USE_REMOTE recovery opts into interactive dialogs.
|
||||
17. SuperSync conflict prefetch and per-user lastSeq reservation stay in one transaction; clean-slate replacement preserves monotonic lastSeq.
|
||||
|
|
@ -81,7 +81,7 @@ The previous broad “remove roughly 6,000 lines together” decision is rejecte
|
|||
- [ ] File snapshot baseline atomicity, post-snapshot suffix ordering, concurrent-compaction / immutable-snapshot resolution (#9040), hydration-time local edits, and archive serialization retain their focused regression coverage.
|
||||
- [ ] The partial WebSocket download pipeline and ImmediateUploadService disappear only after focused replacement tests and the request-cost gate pass.
|
||||
- [ ] Conflict-review producers/UI disappear only to the extent authorized by the deployed-build and persisted-data audit.
|
||||
- [ ] Conflict-journal and disjoint-merge producers stop writing before the first stable release that would otherwise ship them, or the audit explicitly accepts shipping and the expanded data obligation.
|
||||
- [ ] The conflict-journal producer stays frozen for stable releases until the rollback decision; the disjoint-merge producer stays ENABLED — disabling it re-introduces #9095 data loss (see Task 6).
|
||||
- [ ] No public provider protocol, server contract, persistence repository layer, cross-tab lease, or general error taxonomy is introduced.
|
||||
- [ ] Each deletion slice removes more production state/policy than its replacement adds.
|
||||
|
||||
|
|
@ -321,13 +321,13 @@ Bundle these with the matching task; re-verify references at deletion time:
|
|||
|
||||
Remove only what the supported-build/persisted-data decision authorizes:
|
||||
|
||||
- disjoint-merge planning/writing;
|
||||
- conflict journal writing;
|
||||
- review route/UI, banner, badge, navigation, and translations;
|
||||
- journal persistence only after its read/export/expiry obligation ends.
|
||||
|
||||
Preserve:
|
||||
|
||||
- **the disjoint-field merge itself — planner (`conflict-disjoint-merge.util.ts`), patch-op writing, and the failed-merge fallback. Removing it re-introduces #9095: whole-entity LWW silently and permanently destroys the losing side's concurrent field edits (a rename dies when another device marks the task done). Journal removal ≠ merge removal — this task rolls back the _review/journal_ feature, not the merge;**
|
||||
- schema version 3 replace/patch compatibility and its v2-to-v3 migration/barrier;
|
||||
- schema version 4 and its v3-to-v4 migration/barrier, `PROJECT_DELETE_WINS_MARKER`, the authenticated project-ID gate (`isProjectDeleteWinsOperation` requiring `op.entityId === payload.projectId` — distinct from the decrypt-path footprint authentication in contract 14), the shared-package (`@sp/sync-core`) delete-wins classification consumed by the client planner (the server does not classify delete-wins — the marker lives inside the E2EE auth tag and is unreadable server-side), local replacement-op construction, and union of cascaded task/note IDs across concurrent marked deletes;
|
||||
- IndexedDB version 10 downgrade protection;
|
||||
|
|
@ -340,12 +340,13 @@ Preserve:
|
|||
|
||||
There is no Flip-specific operation handler to remove. UI deletion must not touch generic entity-update capture/replay.
|
||||
|
||||
Known seams for the deletion diff: inside ConflictResolutionService the merge producer shares the recovery apply batch, checkpoint-exempt op IDs, the atomic mixed-source append, and the failed-merge fallback re-entry with the multi-entity recovery fixes (#8990, #9007). Its whole-entity-win guard now covers both archive wins and schema-v4 marked project deletes (#9009); delete only the disjoint branch, not the guard or delete-win plan. Recreation markers and follow-up builders are consumed by superseded-operation recovery and the replay-safe task/project/section reducers (#9001). Two post-baseline additions expand that shared surface: the #9048 cascade-recreation helpers (`_createCascadeRecreationOpsForWinningProject`, `_collectDeletedEntityIds`, `_getCurrentEntitiesOfType`) are recovery code sharing the recreation markers and clock-merge helpers with the delete-win family — not the merge producer; and the whole-entity LWW `localWinOperationKind:'update'` local-win path (with its delete-recreation branch) is now also reached via the #9035 deterministic clientId tiebreak in `@sp/sync-core` — keep it. The verified disjoint branch remains cleanly separable (none of #9045/#9048/#9035/#9025 touched it). Review the diff against those seams specifically. The journal clearAll() hooks in OperationLogSyncService and BackupService stay wired while the reader/store stays and leave only with the store; since #9045 clearAll() also writes the `SUP_CONFLICT_JOURNAL_CLEARED_BEFORE` localStorage privacy marker, so store removal must also drop that key and the marker-writing in both hooks. The corruption-classification WeakSet side channel exists only for journal taxonomy and goes with the writer. Disjoint-merge eligibility is coupled to the recreate-fallback constants; the review-UI translations exist in en.json only.
|
||||
Known seams for the deletion diff: inside ConflictResolutionService the merge producer shares the recovery apply batch, checkpoint-exempt op IDs, the atomic mixed-source append, and the failed-merge fallback re-entry with the multi-entity recovery fixes (#8990, #9007). Its whole-entity-win guard now covers both archive wins and schema-v4 marked project deletes (#9009); the disjoint branch itself STAYS (see Preserve) — the deletion here is journal/UI only, and must touch neither the merge branch nor the guard or delete-win plan. Recreation markers and follow-up builders are consumed by superseded-operation recovery and the replay-safe task/project/section reducers (#9001). Two post-baseline additions expand that shared surface: the #9048 cascade-recreation helpers (`_createCascadeRecreationOpsForWinningProject`, `_collectDeletedEntityIds`, `_getCurrentEntitiesOfType`) are recovery code sharing the recreation markers and clock-merge helpers with the delete-win family — not the merge producer; and the whole-entity LWW `localWinOperationKind:'update'` local-win path (with its delete-recreation branch) is now also reached via the #9035 deterministic clientId tiebreak in `@sp/sync-core` — keep it. The verified disjoint branch remains cleanly separable (none of #9045/#9048/#9035/#9025 touched it). Review the diff against those seams specifically. The journal clearAll() hooks in OperationLogSyncService and BackupService stay wired while the reader/store stays and leave only with the store; since #9045 clearAll() also writes the `SUP_CONFLICT_JOURNAL_CLEARED_BEFORE` localStorage privacy marker, so store removal must also drop that key and the marker-writing in both hooks. The corruption-classification WeakSet side channel exists only for journal taxonomy and goes with the writer. Disjoint-merge eligibility is coupled to the recreate-fallback constants; the review-UI translations exist in en.json only.
|
||||
|
||||
If supported users can still have journal rows, first stop new writes and retain a read-only review/export path for the decided support window. Do not orphan or silently delete the only discarded values.
|
||||
|
||||
**Acceptance criteria:**
|
||||
|
||||
- [ ] The disjoint-field merge remains active and #9095's rename-vs-done regression spec stays green.
|
||||
- [ ] Stored v3 replace/patch operations and v4 marked/unmarked project deletions still download/replay with their original semantics from every supported backend.
|
||||
- [ ] No schema downgrade, DB downgrade, or generic update removal appears in the diff.
|
||||
- [ ] Marked project delete-wins, historical losing-delete recovery, relationship follow-ups, same-batch delete exclusion, recovery-row replacement, and replay-atomic project-move tests remain green.
|
||||
|
|
@ -449,14 +450,14 @@ Measured against baseline `6fefd741c5`, to anchor the "removes more than it adds
|
|||
| -------------------------------------------- | --------------------------------------------------------------------- | ---------------------------------- |
|
||||
| Task 4 — WsTriggeredDownloadService (330) | ~240 (thin adapter retained) | ~40 watermark/retry adapter |
|
||||
| Task 5 — ImmediateUploadService (394) | ~370 | ~30 predicate→request wiring |
|
||||
| Task 6 — disjoint-merge planner+in-service | ~330 util + ~380–450 in conflict service | 0 |
|
||||
| Task 6 — disjoint-merge planner+in-service | 0 — KEPT (rescoped 2026-07-17; removal re-introduces #9095) | 0 |
|
||||
| Task 6 — conflict-review UI/banner/util/i18n | ~1,020 (page 636 + banner 130 + util 202 + en.json ~50) | 0 |
|
||||
| Task 6 — journal writer + emission util | ~300 now; +~610 (store 426 + model 185) when the data obligation ends | 0 |
|
||||
| Phase-1 incidentals | ~50 | ~0 |
|
||||
| Task 2 (correctness, not simplification) | 0 | ~120–200 isolation/generation code |
|
||||
| Task 3 (background scheduler) | 0 | ~150–250 new service |
|
||||
|
||||
Net production reduction is roughly **2,350–2,850 lines** with the journal store retained, rising toward **~3,000–3,450** once its retention/export obligation ends and the store+model go. This is deliberately far below the rejected "~6,000 lines together" figure: the difference is the compatibility readers, delete-wins/recovery follow-ups, and thin adapters that stay. Test-file churn is larger still (disjoint-merge and review specs ~2,700+ lines removed) but is not counted here — it roughly nets out against new scheduler/adapter/isolation specs and is not the success measure. The number a phase must clear is qualitative (contract 19 and §4's last line): it removes behavioral states and failure paths, not just lines.
|
||||
Net production reduction is roughly **1,600–2,100 lines** with the journal store retained (down from the pre-rescope 2,350–2,850 — the ~710–780-line disjoint-merge tranche now stays, #9095), rising toward **~2,250–2,750** once its retention/export obligation ends and the store+model go. This is deliberately far below the rejected "~6,000 lines together" figure: the difference is the kept merge, compatibility readers, delete-wins/recovery follow-ups, and thin adapters that stay. Test-file churn is larger still (review specs removed; disjoint-merge specs stay) but is not counted here — it roughly nets out against new scheduler/adapter/isolation specs and is not the success measure. The number a phase must clear is qualitative (contract 19 and §4's last line): it removes behavioral states and failure paths, not just lines.
|
||||
|
||||
Approve work in reviewable tranches:
|
||||
|
||||
|
|
|
|||
|
|
@ -88,9 +88,9 @@ The Operation Log serves **four distinct purposes**:
|
|||
| **C. Server Sync** | Upload/download individual operations (SuperSync) | Complete ✅ (single-version)¹ |
|
||||
| **D. Validation & Repair** | Prevent corruption, auto-repair invalid state | Complete ✅ |
|
||||
|
||||
> ¹ **Cross-version sync limitation**: Part C is complete for clients on the same schema version. Cross-version sync (A.7.11) is not yet implemented—see [A.7.11 Conflict-Aware Migration](#a711-conflict-aware-migration-strategy) for guardrails.
|
||||
> ¹ **Cross-version sync**: receiver-side op migration (A.7.11) runs before conflict detection. The remaining caveat is the released fleet: v17.0.0–v18.14.0 clients apply newer-schema ops (up to schema 5) unmigrated — see the [A.7.11 Bump Policy](#bump-policy--a-bump-does-not-protect-the-released-fleet).
|
||||
|
||||
> **✅ Migration Ready**: Migration safety (A.7.12), tail ops consistency (A.7.13), and unified migration interface (A.7.15) are now implemented. The system is ready for schema migrations when `CURRENT_SCHEMA_VERSION > 1`.
|
||||
> **✅ Migrations Active**: Migration safety (A.7.12), tail ops consistency (A.7.13), and unified migration interface (A.7.15) are implemented, and real migrations exist — `CURRENT_SCHEMA_VERSION = 4` (v1→v2 misc-to-tasks-settings split; v2→v3 and v3→v4 are semantic compatibility barriers). See A.7 and the A.7.11 Bump Policy.
|
||||
|
||||
This document is structured around these four purposes. Most complexity lives in **Part A** (local persistence). **Part B** handles file-based sync via the `FileBasedSyncAdapter`. **Part C** handles operation-based sync with SuperSync server. **Part D** integrates validation and automatic repair.
|
||||
|
||||
|
|
@ -647,16 +647,19 @@ private async attemptRecovery(): Promise<void> {
|
|||
|
||||
When Super Productivity's data model changes (new fields, renamed properties, restructured entities), schema migrations ensure existing data remains usable after app updates.
|
||||
|
||||
> **Current Status:** Migration infrastructure is implemented, but no actual migrations exist yet. The `MIGRATIONS` array is empty and `CURRENT_SCHEMA_VERSION = 1`. This section documents the designed behavior for when migrations are needed.
|
||||
> **Current Status (2026-07):** `CURRENT_SCHEMA_VERSION = 4`. Three migrations exist: v1→v2 (misc-to-tasks-settings split, a real payload transformation) and two no-op semantic barriers — v2→v3 (replacement-mode LWW envelopes) and v3→v4 (marked project delete-wins). The barriers change no stored shapes; they gate conflict semantics for receivers that understand them. Read the [A.7.11 Bump Policy](#bump-policy--a-bump-does-not-protect-the-released-fleet) before adding version 5.
|
||||
|
||||
### Configuration
|
||||
|
||||
`CURRENT_SCHEMA_VERSION` is defined in `src/app/op-log/store/schema-migration.service.ts`:
|
||||
`CURRENT_SCHEMA_VERSION` is defined in `packages/shared-schema/src/schema-version.ts` (shared by client and server) and re-exported through `src/app/op-log/persistence/schema-migration.service.ts`:
|
||||
|
||||
```typescript
|
||||
export const CURRENT_SCHEMA_VERSION = 1;
|
||||
export const PROJECT_DELETE_WINS_SCHEMA_VERSION = 4;
|
||||
export const CURRENT_SCHEMA_VERSION = PROJECT_DELETE_WINS_SCHEMA_VERSION;
|
||||
export const MIN_SUPPORTED_SCHEMA_VERSION = 1;
|
||||
export const MAX_VERSION_SKIP = 5; // Max versions ahead we'll attempt to load
|
||||
// MAX_VERSION_SKIP was removed: current receivers block any newer-schema op
|
||||
// outright. Released v17.0.0–v18.14.0 clients still ship the old +3 band and
|
||||
// apply ops up to schema 5 UNMIGRATED — see the A.7.11 Bump Policy.
|
||||
```
|
||||
|
||||
### Core Concepts
|
||||
|
|
@ -759,6 +762,10 @@ Client v2 ◄─── ops from v1 client
|
|||
|
||||
Scenario 2: Older client receives newer ops
|
||||
──────────────────────────────────────────
|
||||
[OUTDATED design sketch — actual behavior differs. Current (post-v18.14.0)
|
||||
receivers BLOCK any newer-schema op outright (cursor frozen, update prompt).
|
||||
Released v17–v18.14 receivers apply newer ops UNMIGRATED — unknown fields are
|
||||
NOT ignored; reducers write them into state. See the A.7.11 Bump Policy.]
|
||||
Client v1 ◄─── ops from v2 client
|
||||
│
|
||||
├── Individual ops: Unknown fields ignored (graceful degradation)
|
||||
|
|
@ -780,6 +787,8 @@ Client v1 conflicts with Client v2
|
|||
|
||||
When receiving full state from remote (e.g., SYNC_IMPORT from another client):
|
||||
|
||||
> **Note (2026-07):** the pseudo-code below predates the removal of the forward-compat band (`MAX_VERSION_SKIP` no longer exists; current receivers block any newer-versioned data outright). Kept as design context only.
|
||||
|
||||
```typescript
|
||||
async handleFullStateImport(payload: { appDataComplete: AppDataComplete }): Promise<void> {
|
||||
const { appDataComplete } = payload;
|
||||
|
|
@ -825,7 +834,7 @@ async handleFullStateImport(payload: { appDataComplete: AppDataComplete }): Prom
|
|||
|
||||
### A.7.5 Migration Implementation
|
||||
|
||||
Migrations are defined in `src/app/op-log/store/schema-migration.service.ts`.
|
||||
Migrations are defined in `packages/shared-schema/src/migrations/` (registered in its `index.ts`); the Angular wrapper is `src/app/op-log/persistence/schema-migration.service.ts`.
|
||||
|
||||
**How to Create a New Migration:**
|
||||
|
||||
|
|
@ -890,22 +899,20 @@ All future schema changes should use the **Schema Migration** system (A.7) descr
|
|||
|
||||
**Rule of thumb:** Additive changes (new optional fields, new entities) don't need operation migration. Field renames/removals require it.
|
||||
|
||||
### A.7.8 Cross-Version Sync (Not Yet Implemented)
|
||||
### A.7.8 Cross-Version Sync
|
||||
|
||||
**Status:** Design ready, not implemented. Safe while `CURRENT_SCHEMA_VERSION = 1`.
|
||||
**Status:** Implemented receiver-side: every remote op passes `SchemaMigrationService.migrateOperation()` before conflict detection (`RemoteOpsProcessingService`, STEP 1). Sender uploads ops as-is.
|
||||
|
||||
**Strategy:** Receiver migrates incoming ops before conflict detection. Sender uploads ops as-is.
|
||||
**Guardrails for newer-schema ops:**
|
||||
|
||||
**Interim guardrails:**
|
||||
|
||||
- Reject ops with `schemaVersion > CURRENT + MAX_VERSION_SKIP`
|
||||
- Prompt user to update app when receiving newer-version ops
|
||||
- Current receivers (post-v18.14.0): block any op with `schemaVersion > CURRENT_SCHEMA_VERSION` outright, freeze the download cursor, and prompt for an app update.
|
||||
- Released receivers (v17.0.0–v18.14.0): tolerate up to `CURRENT + 3` (their `MAX_VERSION_SKIP`) and apply those ops UNMIGRATED after a once-per-session warning — and they advance the cursor even when blocking, permanently skipping blocked ops. This fleet reality drives the A.7.11 Bump Policy.
|
||||
|
||||
**Required before:** Any schema migration that renames/removes fields.
|
||||
|
||||
### A.7.11 Cross-Version Sync Implementation Guide
|
||||
|
||||
> **Status:** Not yet implemented. This section documents the design for when `CURRENT_SCHEMA_VERSION > 1`.
|
||||
> **Status:** The receiver-side migration pipeline is implemented and `CURRENT_SCHEMA_VERSION = 4` (two of the three existing migrations are no-op semantic barriers). The Bump Policy below is normative.
|
||||
|
||||
This guide provides the implementation roadmap for supporting sync between clients on different schema versions.
|
||||
|
||||
|
|
@ -925,6 +932,18 @@ Bump the schema version when:
|
|||
|
||||
**Decision rule:** If the change affects how `state_cache` snapshots or operation payloads are structured, bump the version.
|
||||
|
||||
#### Bump Policy — a bump does NOT protect the released fleet
|
||||
|
||||
A version bump only fences receivers that ship AFTER the bump. As of 2026-07:
|
||||
|
||||
- Every released client from v17.0.0 through v18.14.0 runs schema 2 with a forward-compat band (`MAX_VERSION_SKIP = 3`): it APPLIES ops up to schema 5 unmigrated after a once-per-session warning snack, and blocks schema ≥ 6 — but these clients advance the server cursor even while blocking, permanently skipping the blocked ops (loss that survives the later app update).
|
||||
- Post-v18.14.0 receivers block any newer-schema op outright and freeze the cursor (loud and lossless).
|
||||
|
||||
Therefore:
|
||||
|
||||
1. New op semantics MUST degrade gracefully on older clients — see the `LwwUpdatePayload` envelope pattern in `packages/sync-core` ('patch' ops apply correctly on pre-v3 clients via `updateOne`; the v4 delete-wins marker is inert for them). If they degrade, bumping is safe at any fleet share: the stamp is a fence for future receivers, not a protection for current ones.
|
||||
2. A change that older clients would MISAPPLY must not ship behind a bump alone. No fleet percentage makes it safe while released v17–v18.14 clients still sync: one lagging device silently misapplies the ops for its whole account and writes the result back with dominating clocks. Treat such changes as blocked until the v17–v18.14 sync fleet is effectively extinct — or redesign them to degrade (option 1).
|
||||
|
||||
#### Operation Transformation Strategy
|
||||
|
||||
When receiving operations from older versions:
|
||||
|
|
@ -989,13 +1008,14 @@ Remote Op (v1) Local Op (v2)
|
|||
#### Backward Compatibility Guarantees
|
||||
|
||||
| Scenario | Behavior | User Experience |
|
||||
| ------------------------------------------ | ---------------------------------------------------- | ------------------------- |
|
||||
| Newer client → Older client | Ops uploaded as-is; older client migrates on receive | Seamless |
|
||||
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- |
|
||||
| Older client → Newer client | Newer client migrates incoming ops | Seamless |
|
||||
| Client too old (> MAX_VERSION_SKIP behind) | Reject ops, prompt update | "Please update app" modal |
|
||||
| Client too new (server rejects) | N/A - server doesn't validate schema | No issue |
|
||||
| Newer client → post-v18.14.0 receiver | Receiver blocks the op outright, freezes the cursor, prompts update | Sync pauses loudly, lossless |
|
||||
| Newer client → released receiver (v17–v18.14) | Applies ops up to schema 5 UNMIGRATED (once-per-session warning); at ≥ 6 blocks but advances the cursor, permanently skipping the blocked ops | Silent misapply / silent loss — see Bump Policy |
|
||||
| Op below `MIN_SUPPORTED_SCHEMA_VERSION` | Reject ops, prompt update | "Please update app" error |
|
||||
| Client too new (server rejects) | N/A - server doesn't validate schema semantics (bounds check only) | No issue |
|
||||
|
||||
**MAX_VERSION_SKIP = 5**: Clients more than 5 versions behind cannot sync until updated. This bounds the migration chain complexity.
|
||||
There is no forward-migration path: an older client can never migrate a newer op. Cross-version safety toward older clients rests entirely on payload-level graceful degradation (see Bump Policy above). `MAX_VERSION_SKIP` no longer exists in current code; released v17–v18.14 clients shipped it as `3`.
|
||||
|
||||
#### Migration Rollout Strategy
|
||||
|
||||
|
|
@ -2273,7 +2293,7 @@ When adding new entities or relationships:
|
|||
| ------------------------------- | ------- | ---------------------------------------- | ------------------------------------------------------- |
|
||||
| **Conflict-aware op migration** | A.7.11 | Conflicts may compare mismatched schemas | Before any schema migration that renames/removes fields |
|
||||
|
||||
> **Note**: A.7.11 is required for cross-version sync. Currently safe because `CURRENT_SCHEMA_VERSION = 1` (all clients on same version). See [A.7.11 Interim Guardrails](#interim-guardrails-until-implementation) for pre-release checklist.
|
||||
> **Note**: receiver-side op migration (A.7.11) runs before conflict detection. `CURRENT_SCHEMA_VERSION = 4`; released v17–v18.14 clients apply v3/v4 ops unmigrated and rely on payload-level graceful degradation — see the A.7.11 Bump Policy. What remains unimplemented is a _transforming_ migration for field renames/removals.
|
||||
|
||||
## Part B: Legacy Sync Bridge
|
||||
|
||||
|
|
@ -2322,7 +2342,7 @@ When adding new entities or relationships:
|
|||
- Batch cleanup queries (replaced N+1 pattern)
|
||||
- Database index on `(user_id, received_at)` for cleanup queries
|
||||
|
||||
> **Cross-version limitation**: Part C is complete for clients on the same schema version. When `CURRENT_SCHEMA_VERSION > 1` and clients run different versions, A.7.11 (conflict-aware op migration) is required to ensure correct conflict detection.
|
||||
> **Cross-version note**: incoming ops are migrated (A.7.11) before conflict detection. `CURRENT_SCHEMA_VERSION = 4` since 2026-07; cross-version safety against released v17–v18.14 clients rests on payload-level graceful degradation, not on the version fence — see the A.7.11 Bump Policy.
|
||||
|
||||
## Part D: Validation & Repair
|
||||
|
||||
|
|
@ -2339,11 +2359,11 @@ When adding new entities or relationships:
|
|||
## Future Enhancements 🔮
|
||||
|
||||
| Component | Description | Priority | Notes |
|
||||
| ---------- | ------------------------------------------ | -------- | ------------------------------------------------------------------------------ |
|
||||
| ---------- | ------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------- |
|
||||
| Auto-merge | Automatic merge for non-conflicting fields | Low | |
|
||||
| Undo/Redo | Leverage op-log for undo history | Low | |
|
||||
| Tombstones | Soft delete with retention window | Medium | Deferred Dec 2025 - current safeguards sufficient (see todo.md for evaluation) |
|
||||
| A.7.11 | Conflict-aware operation migration | High | Required before `CURRENT_SCHEMA_VERSION > 1` for cross-version sync |
|
||||
| A.7.11 | Transforming operation migrations | High | Receiver-side pipeline exists; a rename/remove-field migration must follow the A.7.11 Bump Policy |
|
||||
|
||||
> **Recently Completed (December 2025):**
|
||||
>
|
||||
|
|
@ -2381,11 +2401,10 @@ src/app/op-log/
|
|||
├── operation-converter.util.ts # Op ↔ Action conversion
|
||||
├── persistent-action.interface.ts # PersistentAction type + isPersistentAction guard
|
||||
├── entity-key.util.ts # Entity key generation utilities
|
||||
├── store/
|
||||
├── persistence/
|
||||
│ ├── operation-log-store.service.ts # SUP_OPS IndexedDB wrapper
|
||||
│ ├── operation-log-hydrator.service.ts # Startup hydration + crash recovery
|
||||
│ ├── operation-log-compaction.service.ts # Snapshot + cleanup + emergency mode
|
||||
│ ├── operation-log-manifest.service.ts # File-based sync manifest management
|
||||
│ ├── operation-log-migration.service.ts # Genesis migration from legacy
|
||||
│ └── schema-migration.service.ts # State schema migrations
|
||||
├── sync/
|
||||
|
|
|
|||
|
|
@ -78,8 +78,13 @@ This document establishes the core rules and principles for designing the Operat
|
|||
- **Purpose:** To allow future versions of the app to migrate or interpret old operations correctly.
|
||||
- **Default:** Use `CURRENT_SCHEMA_VERSION` from `SchemaMigrationService` at the time of creation.
|
||||
- **Compatibility barrier:** Bump the schema version when an old reducer would
|
||||
silently misinterpret a new payload semantic. The existing newer-schema gate
|
||||
must stop the old client instead of allowing divergent replay.
|
||||
silently misinterpret a new payload semantic — but a bump alone does NOT stop
|
||||
released clients: v17.0.0–v18.14.0 receivers apply ops up to schema 5
|
||||
unmigrated, and at ≥ 6 block them while still advancing the server cursor
|
||||
(permanent silent skip). Only post-v18.14.0 receivers block newer ops safely.
|
||||
New payload semantics must therefore degrade gracefully on old clients; see
|
||||
the Bump Policy in
|
||||
[operation-log-architecture.md](operation-log-architecture.md) §A.7.11.
|
||||
|
||||
### 2.6 Explicit Intent (OpType)
|
||||
|
||||
|
|
|
|||
|
|
@ -4,10 +4,14 @@ import type { SchemaMigration } from '../migration.types';
|
|||
* Compatibility barrier for synthetic LWW replacement payloads.
|
||||
*
|
||||
* Schema v3 introduces replacement-mode LWW envelopes whose omitted fields
|
||||
* are intentionally cleared. A v2 client ignores the mode and applies those
|
||||
* payloads as patches, which can silently diverge. No stored state shape needs
|
||||
* transformation, but stamping all newly produced operations as v3 makes v2
|
||||
* clients stop at the existing newer-schema gate instead of misapplying them.
|
||||
* are intentionally cleared. NOTE: this stamp does NOT stop released clients.
|
||||
* Every v17.0.0–v18.14.0 receiver tolerates ops up to schema 5 (old
|
||||
* MAX_VERSION_SKIP band) and applies them unmigrated: it ignores
|
||||
* `lwwUpdateMode` and applies the envelope's actionPayload via updateOne —
|
||||
* correct for 'patch' ops, and a near-replacement for 'replace' ops where
|
||||
* only omitted-to-clear fields fail to propagate. The stamp is a fence for
|
||||
* post-v18.14.0 receivers only, which block any newer-schema op outright. See the
|
||||
* bump policy in schema-version.ts before relying on a version fence.
|
||||
* Historical v2 operations retain their patch semantics when migrated.
|
||||
*/
|
||||
export const LwwReplacementBarrierMigration_v2v3: SchemaMigration = {
|
||||
|
|
|
|||
|
|
@ -7,6 +7,12 @@ import type { SchemaMigration } from '../migration.types';
|
|||
* deleteProject operations carry an explicit payload marker; leaving older
|
||||
* operations untouched ensures they retain their original timestamp-based LWW
|
||||
* semantics after receiver-side migration.
|
||||
*
|
||||
* NOTE: released pre-v4 clients (v17.0.0–v18.14.0) apply v4 ops unmigrated
|
||||
* and treat the marker as an inert actionPayload key — they keep resolving
|
||||
* project-delete conflicts by timestamp LWW. The version stamp fences only
|
||||
* post-v18.14.0 receivers, which block newer-schema ops outright. See the bump
|
||||
* policy in schema-version.ts.
|
||||
*/
|
||||
export const ProjectDeleteWinsBarrierMigration_v3v4: SchemaMigration = {
|
||||
fromVersion: 3,
|
||||
|
|
|
|||
|
|
@ -1,6 +1,18 @@
|
|||
/**
|
||||
* Current schema version for all operations and state snapshots.
|
||||
* Increment this BEFORE adding a new migration.
|
||||
*
|
||||
* BUMP POLICY — a bump does NOT protect the released fleet. Every released
|
||||
* client from v17.0.0 through v18.14.0 tolerates ops up to schema 5 (its
|
||||
* version 2 plus the old MAX_VERSION_SKIP band of 3) and applies them
|
||||
* UNMIGRATED after one warning snack per session; at schema >= 6 it blocks,
|
||||
* but still advances the server cursor, permanently skipping the blocked ops
|
||||
* even after the user updates. Only post-v18.14.0 receivers block newer ops
|
||||
* safely (cursor frozen). Therefore new op semantics must degrade gracefully on older
|
||||
* clients (see the LwwUpdatePayload envelope pattern in packages/sync-core);
|
||||
* a change old clients would MISAPPLY must not ship behind a bump alone.
|
||||
* Full policy: docs/sync-and-op-log/operation-log-architecture.md, A.7.11
|
||||
* "Bump Policy".
|
||||
*/
|
||||
export const PROJECT_DELETE_WINS_SCHEMA_VERSION = 4;
|
||||
export const CURRENT_SCHEMA_VERSION = PROJECT_DELETE_WINS_SCHEMA_VERSION;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue