From 582929e375d791dfde366b6289be4f42b925deda Mon Sep 17 00:00:00 2001 From: Johannes Millan Date: Sat, 11 Jul 2026 18:06:20 +0200 Subject: [PATCH] docs(sync): document atomic checkpoint, db v8 barrier and rebuild undo Update the archive-operation lifecycle docs to the atomic reducer-checkpoint + clock-merge design, record the IndexedDB v8 downgrade barrier, drop the removed PENDING_OPERATION_EXPIRY_MS constant, and describe the durable Use-Server-Data Undo across reloads. --- .../diagrams/06-archive-operations.md | 8 ++++---- .../operation-log-architecture.md | 12 ++++++------ docs/sync-and-op-log/operation-rules.md | 17 ++++++++--------- docs/sync-and-op-log/supersync-scenarios.md | 7 ++++--- docs/wiki/3.06-User-Data.md | 6 +++--- 5 files changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/sync-and-op-log/diagrams/06-archive-operations.md b/docs/sync-and-op-log/diagrams/06-archive-operations.md index d6b25e71ec..0e27773472 100644 --- a/docs/sync-and-op-log/diagrams/06-archive-operations.md +++ b/docs/sync-and-op-log/diagrams/06-archive-operations.md @@ -88,11 +88,11 @@ flowchart TD subgraph RemoteOp["REMOTE Operation (Sync)"] R1[Download operation
from sync] --> R2["Append remote row
status: pending"] R2 --> R3["Bulk reducer dispatch
meta.isRemote=true"] - R3 --> R4["Reducer-commit checkpoint:
status archive_pending
merge vector clocks"] + R3 --> R4["Atomic reducer checkpoint:
archive_pending + vector clocks"] R4 --> R5["ArchiveOperationHandler
.handleOperation"] R5 --> R6{"Archive side effect
succeeded?"} R6 -->|Yes| R7["Mark applied"] - R6 -->|No| R8["Mark attempted row failed;
successors stay archive_pending"] + R6 -->|No| R8["Bump attempted row retryCount;
successors stay quarantined"] NoEffect["❌ Regular effects DON'T run
(action has meta.isRemote=true)"] end @@ -116,14 +116,14 @@ flowchart TD ## ArchiveOperationHandler Integration -The `OperationApplierService` applies operations in the order they arrive from the sync server, which preserves causal ordering (each client uploads its ops in causal order, and the server assigns sequence numbers in upload order). Incoming rows are first stored as `pending`. Immediately after the single bulk reducer dispatch, the reducer-commit callback checkpoints the entire batch as `archive_pending` and merges its vector clocks; only then do archive side effects run. Successful rows become `applied`. The attempted row becomes `failed` if an archive side effect throws, while unattempted successors remain `archive_pending`. On startup, hydration restores reducer state and retries `archive_pending`/`failed` rows with reducer dispatch disabled, so additive reducers are never applied twice. +The `OperationApplierService` applies operations in the order they arrive from the sync server, which preserves causal ordering (each client uploads its ops in causal order, and the server assigns sequence numbers in upload order). Incoming rows are first stored as `pending`. Immediately after the single bulk reducer dispatch, one transaction changes the whole reducer-committed batch to `archive_pending` and merges its vector clocks; only then do archive side effects run. Successful rows become `applied`. If an archive side effect throws, only the attempted row becomes `failed` and consumes `retryCount`; unattempted successors remain `archive_pending`. On startup, hydration restores reducer state and retries `archive_pending`/`failed` rows with reducer dispatch disabled, so additive reducers are never applied twice. ```mermaid flowchart TD subgraph OperationApplierService["OperationApplierService (Bulk Dispatch)"] OA1["Receive rows already stored
as pending"] --> OA3[convertOpToAction] OA3 --> OA4["Single bulk dispatch
with meta.isRemote=true"] - OA4 --> OA4b["Reducer-commit callback:
mark archive_pending
merge vector clocks"] + OA4 --> OA4b["Atomic reducer checkpoint:
archive_pending + vector clocks"] OA4b --> OA5["archiveOperationHandler
.handleOperation(action)"] end diff --git a/docs/sync-and-op-log/operation-log-architecture.md b/docs/sync-and-op-log/operation-log-architecture.md index 57592e02b3..0c7d7b2461 100644 --- a/docs/sync-and-op-log/operation-log-architecture.md +++ b/docs/sync-and-op-log/operation-log-architecture.md @@ -170,7 +170,7 @@ interface OperationLogEntry { source: 'local' | 'remote'; syncedAt?: number; // For server sync (Part C) rejectedAt?: number; // When rejected during conflict resolution - applicationStatus?: 'pending' | 'archive_pending' | 'applied' | 'failed'; + applicationStatus?: 'pending' | 'archive_pending' | 'failed' | 'applied'; } // state_cache table - periodic snapshots @@ -211,11 +211,11 @@ interface StateCache { Downloaded operations use a durable status transition so reducer state, archive IndexedDB side effects, vector clocks, and the server cursor cannot disagree after a crash: 1. `pending` — the remote op is stored, but no reducer-commit checkpoint exists yet. -2. `archive_pending` — the complete batch was bulk-dispatched to reducers and its vector clocks were merged; archive side effects are still outstanding. -3. `applied` — reducer and archive work both completed. -4. `failed` — an attempted archive side effect failed. Later operations in the same batch remain `archive_pending` without consuming retry budget. +2. `archive_pending` — reducers committed and the op's vector clock was merged atomically; archive side effects have not yet completed. +3. `failed` — an archive side-effect attempt failed. `retryCount` is charged only to the attempted row, so later rows in the same batch do not consume retry budget. +4. `applied` — reducer and archive work both completed. -Startup hydration replays the persisted state history, converts surviving `pending` rows to the archive checkpoint, and retries `archive_pending`/`failed` rows with reducer dispatch disabled. Ordinary sync refuses to download, upload, or advance its cursor while any of these incomplete rows remain. +Startup hydration replays persisted state history, quarantines surviving `pending` rows, and retries `archive_pending`/`failed` rows with reducer dispatch disabled. Ordinary sync refuses to download, upload, or advance its cursor while any incomplete rows remain. Database version 8 is a downgrade barrier: released readers that do not understand the distinct reducer checkpoint cannot open the newer store and silently overlook outstanding archive work. Local actions buffered during a remote-apply window stay ordered until each operation is durable. Transient persistence failures keep the failed suffix queued and block the current sync so a later sync can retry. A deterministically invalid buffered action also remains queued, but requires reload: its reducer already changed live state, so discarding it would let live state diverge from the durable operation log. @@ -2324,7 +2324,7 @@ When adding new entities or relationships: > - **Archive validation**: archiveOld tasks now validated for project/tag references, null-safety added > - **Lock service robustness**: Handle NaN timestamps and invalid lock formats in fallback lock > - **Array payload rejection**: Explicit check to reject arrays (which bypass `typeof === 'object'`) -> - **Pending operation expiry**: Operations pending >24h are quarantined as failed; hydration still restores their reducer history, while archive recovery and the sync gate remain active (`PENDING_OPERATION_EXPIRY_MS`). +> - **Remote apply checkpoints**: reducer commit and vector-clock merge are atomic; `archive_pending` distinguishes unattempted archive work from attempted `failed` rows, while hydration and the sync gate keep incomplete work visible. --- diff --git a/docs/sync-and-op-log/operation-rules.md b/docs/sync-and-op-log/operation-rules.md index 455cf1d3c2..3fa400feda 100644 --- a/docs/sync-and-op-log/operation-rules.md +++ b/docs/sync-and-op-log/operation-rules.md @@ -193,15 +193,14 @@ export class MyEffects { See `operation-log.const.ts` for all configurable values: -| Constant | Value | Description | -| ----------------------------------- | -------- | -------------------------------------------------------------------------- | -| `COMPACTION_TRIGGER` | 500 ops | Operations before automatic compaction | -| `COMPACTION_RETENTION_MS` | 7 days | Synced ops older than this may be deleted | -| `EMERGENCY_COMPACTION_RETENTION_MS` | 1 day | Shorter retention for quota exceeded | -| `MAX_COMPACTION_FAILURES` | 3 | Failures before user notification | -| `MAX_DOWNLOAD_OPS_IN_MEMORY` | 50,000 | Bounds memory during API download | -| `REMOTE_OP_FILE_RETENTION_MS` | 14 days | Server-side operation file retention | -| `PENDING_OPERATION_EXPIRY_MS` | 24 hours | Pending ops older than this are quarantined as failed for archive recovery | +| Constant | Value | Description | +| ----------------------------------- | ------- | ----------------------------------------- | +| `COMPACTION_TRIGGER` | 500 ops | Operations before automatic compaction | +| `COMPACTION_RETENTION_MS` | 7 days | Synced ops older than this may be deleted | +| `EMERGENCY_COMPACTION_RETENTION_MS` | 1 day | Shorter retention for quota exceeded | +| `MAX_COMPACTION_FAILURES` | 3 | Failures before user notification | +| `MAX_DOWNLOAD_OPS_IN_MEMORY` | 50,000 | Bounds memory during API download | +| `REMOTE_OP_FILE_RETENTION_MS` | 14 days | Server-side operation file retention | ## 7. Quick Reference Checklist diff --git a/docs/sync-and-op-log/supersync-scenarios.md b/docs/sync-and-op-log/supersync-scenarios.md index 5bb3d45ebf..7b7340b3c4 100644 --- a/docs/sync-and-op-log/supersync-scenarios.md +++ b/docs/sync-and-op-log/supersync-scenarios.md @@ -154,7 +154,8 @@ Comprehensive spec of all scenarios that can occur during SuperSync synchronizat 3. Throw `LocalDataConflictError` 4. Show full conflict dialog: USE_LOCAL / USE_REMOTE / CANCEL 5. USE_LOCAL → `forceUploadLocalState()` (creates SYNC_IMPORT) -6. USE_REMOTE → `forceDownloadRemoteState()` (clears local ops) +6. USE_REMOTE → capture a single-slot pre-replace backup, then `forceDownloadRemoteState()` (clears local ops) +7. The replacement transaction sets a raw-rebuild resume marker. Completion atomically replaces it with a durable Undo provenance token; startup re-offers Undo after reload while the same backup remains. **User sees:** Full conflict resolution dialog. @@ -201,10 +202,10 @@ Comprehensive spec of all scenarios that can occur during SuperSync synchronizat 2. Check pending local ops. Any pending work triggers the dialog except onboarding example-task creates and the never-synced `GLOBAL_CONFIG:sync` provider-setup write. Other GLOBAL_CONFIG sections remain protected. 3. **Show conflict dialog BEFORE processing** with `scenario: 'INCOMING_IMPORT'` and `syncImportReason` 4. USE_LOCAL → `forceUploadLocalState()` (overrides remote with local data) -5. USE_REMOTE → `forceDownloadRemoteState()` (clears local ops, downloads from seq 0) +5. USE_REMOTE → capture a pre-replace backup, then `forceDownloadRemoteState()` (clears local ops, downloads from seq 0) 6. CANCEL → return with `cancelled: true`, skip upload phase -**User sees:** Conflict dialog explaining remote import detected with local changes at risk. "Use Server Data" recommended. +**User sees:** Conflict dialog explaining remote import detected with local changes at risk. "Use Server Data" recommended. After replacement, a persistent Undo action remains recoverable across reloads while its matching backup exists. ### D.3: Remote Ops Filtered by Stored Local SYNC_IMPORT ✓ diff --git a/docs/wiki/3.06-User-Data.md b/docs/wiki/3.06-User-Data.md index 1baea56684..a51620d446 100644 --- a/docs/wiki/3.06-User-Data.md +++ b/docs/wiki/3.06-User-Data.md @@ -77,7 +77,7 @@ The Windows Store build uses a different path that includes the Windows Store pa ### IndexedDB Databases -- **SUP_OPS** (current, version 4): Main database. Object stores: `ops`, `state_cache`, `import_backup`, `vector_clock`, `archive_young`, `archive_old`. Holds operations log, state snapshots, vector clocks, and archives. +- **SUP_OPS** (current, version 8): Main database. Object stores: `ops`, `state_cache`, `import_backup`, `vector_clock`, `archive_young`, `archive_old`. Holds operations log, state snapshots, vector clocks, and archives. - **pf:** Legacy database used for migration and recovery. - **SUPPluginCache:** Plugin cache. @@ -107,7 +107,7 @@ On mobile (Android/iOS), if the app launches with no local data but a usable on- **Manual backups:** (1) **Create manual backup** in Settings → Sync & Export creates a backup using the same mechanism as automatic backups (platform-dependent location). (2) **Safety backups** are created automatically before certain sync operations; the app keeps a limited number of slots (e.g. two most recent, one first from today, one first from day before). (3) **Export data** downloads a complete backup JSON file to a path you choose (or the browser download folder on web). -**What is included:** All application data is included in backups and exports; nothing is excluded. When archives are included (e.g. when using "Export data" with full backup), both `archive_young` and `archive_old` are part of the file. A separate, single-slot **import backup** is stored locally in IndexedDB (`import_backup`). The app captures it before replacing local state through a file import or the sync conflict action **Use Server Data**. A successful Use Server Data replacement shows a persistent **Undo** action that restores this backup; interrupted replacements offer the same recovery action when sync resumes. The slot is internal recovery state and is not included in exports or the user-visible automatic/manual backup lists. +**What is included:** All application data is included in backups and exports; nothing is excluded. When archives are included (e.g. when using "Export data" with full backup), both `archive_young` and `archive_old` are part of the file. A separate, single-slot **import backup** is stored locally in IndexedDB (`import_backup`). The app captures it before replacing local state through a file import or the sync conflict action **Use Server Data**. A successful Use Server Data replacement shows a persistent **Undo** action that restores this backup; interrupted replacements offer the same recovery action when sync resumes. The Undo remains available after an app reload while that same backup is still present. The slot is internal recovery state and is not included in exports or the user-visible automatic/manual backup lists. ## Import and Export @@ -176,7 +176,7 @@ On mobile (Android/iOS), if the app launches with no local data but a usable on- ## Configuration and Versioning - There are no separate config files in the User Data Folder. Configuration is stored in IndexedDB (`SUP_OPS`, `state_cache`) and in localStorage (`SUP_*` and related keys). -- **Database version:** `SUP_OPS` is at schema version 4. +- **Database version:** `SUP_OPS` is at schema version 8. - **Application version:** Defined in the build (e.g. `src/environments/versions.ts`), not stored in the User Data Folder. - **Backup files:** Each JSON file carries full state; desktop automatic backup naming includes date and time.