diff --git a/docs/sync-and-op-log/diagrams/03-conflict-resolution.md b/docs/sync-and-op-log/diagrams/03-conflict-resolution.md index 1393dc65a6..c88a544e1b 100644 --- a/docs/sync-and-op-log/diagrams/03-conflict-resolution.md +++ b/docs/sync-and-op-log/diagrams/03-conflict-resolution.md @@ -166,12 +166,34 @@ flowchart LR style BackupCreated fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px ``` +### Archive-Wins Exception + +When a `moveToArchive` operation conflicts with a field-level update (rename, time tracking, etc.), the **archive operation always wins** regardless of timestamps. This bypasses the normal LWW timestamp comparison because archiving represents explicit user intent that should not be reversed. + +```mermaid +flowchart TD + subgraph ArchiveCheck["Archive-Wins Check (Before LWW)"] + Conflict["Conflict detected"] --> IsArchive{"Does either side
contain moveToArchive?"} + + IsArchive -->|"Yes"| ArchiveWins["🏆 ARCHIVE WINS
Regardless of timestamps"] + IsArchive -->|"No"| NormalLWW["Proceed to normal
LWW timestamp comparison"] + end + + ArchiveWins --> CreateOp["Create new archive op
with merged vector clock
via _createArchiveWinOp()"] + + style ArchiveWins fill:#fff3e0,stroke:#ef6c00,stroke-width:2px + style NormalLWW fill:#e3f2fd,stroke:#1565c0,stroke-width:2px +``` + +**Why?** Without this rule, a concurrent field-level LWW Update could "resurrect" an archived task by replacing its state in the active store. The archive-wins rule is the first level of defense; the `bulkOperationsMetaReducer` provides a second level by pre-scanning batches for archive ops (see [06-archive-operations.md](./06-archive-operations.md)). + ### Key Implementation Details | Aspect | Implementation | | ---------------------- | --------------------------------------------------------------------------- | | **Timestamp Source** | `Math.max(...Object.values(vectorClock))` - max timestamp from vector clock | | **Tie Breaker** | Remote wins (ensures convergence across all clients) | +| **Archive Exception** | `moveToArchive` always wins over field-level updates, bypassing timestamps | | **Safety Backup** | Created via `BackupService` before any resolution | | **Local Win Update** | New `OpType.UPD` operation created with merged vector clock | | **Vector Clock Merge** | `mergeVectorClocks(localClock, remoteClock)` for local-win ops | diff --git a/docs/sync-and-op-log/diagrams/05-meta-reducers.md b/docs/sync-and-op-log/diagrams/05-meta-reducers.md index 90c5888746..8c21f74501 100644 --- a/docs/sync-and-op-log/diagrams/05-meta-reducers.md +++ b/docs/sync-and-op-log/diagrams/05-meta-reducers.md @@ -182,10 +182,63 @@ sequenceDiagram Note over State: Either ALL changes applied
or NONE (transaction semantics) ``` +## LWW Update Meta-Reducer: Entity Type Handling + +The `lwwUpdateMetaReducer` handles LWW Update actions (created when the local side wins a conflict). It distinguishes between three entity storage patterns: + +```mermaid +flowchart TD + subgraph Input["LWW Update Action"] + Action["[TASK] LWW Update
entityType + entityId + winningData"] + end + + subgraph Lookup["Entity Registry Lookup"] + Registry["Look up entity storage pattern
in entity registry"] + end + + subgraph Patterns["Storage Pattern Handling"] + Adapter["ADAPTER ENTITIES
━━━━━━━━━━━━━━━
TASK, PROJECT, TAG, NOTE,
TASK_REPEAT_CFG, ISSUE_PROVIDER,
SIMPLE_COUNTER, BOARD, METRIC,
REMINDER, PLUGIN_USER_DATA,
PLUGIN_METADATA
━━━━━━━━━━━━━━━
adapter.updateOne() or addOne()
+ relationship syncing"] + + Singleton["SINGLETON ENTITIES
━━━━━━━━━━━━━━━
GLOBAL_CONFIG,
TIME_TRACKING,
MENU_TREE,
WORK_CONTEXT
━━━━━━━━━━━━━━━
Entire feature state replaced
with winning data"] + + Unsupported["UNSUPPORTED
━━━━━━━━━━━━━━━
Map, array, virtual
━━━━━━━━━━━━━━━
Warning logged,
no action taken"] + end + + Input --> Lookup + Lookup --> Patterns + + style Adapter fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px + style Singleton fill:#e3f2fd,stroke:#1565c0,stroke-width:2px + style Unsupported fill:#fff3e0,stroke:#ef6c00,stroke-width:2px +``` + +### Adapter Entity Details + +For adapter-backed entities, the meta-reducer handles two sub-cases: + +| Condition | Behavior | Why | +| ---------------------- | --------------------------------------------------------- | ---------------------------------------------------------------------------------- | +| Entity exists in store | `adapter.updateOne()` — replaces entity with winning data | Normal conflict resolution | +| Entity NOT in store | `adapter.addOne()` — recreates entity | Handles DELETE vs UPDATE race (entity was deleted locally but update won remotely) | + +### Relationship Syncing for Tasks + +After updating a task via LWW, the meta-reducer syncs related entity references: + +| Field Changed | Relationship Synced | +| ------------- | ------------------------------------------------------------------ | +| `projectId` | `project.taskIds` updated to reflect new/old project membership | +| `tagIds` | `tag.taskIds` updated for each added/removed tag | +| `dueDay` | `TODAY_TAG.taskIds` updated (virtual tag, membership via `dueDay`) | +| `parentId` | `parent.subTaskIds` updated for new/old parent task | + +**Key file:** `src/app/root-store/meta/task-shared-meta-reducers/lww-update.meta-reducer.ts` + ## Key Files -| File | Purpose | -| ----------------------------------------------------------------------------- | --------------------------------- | -| `src/app/root-store/meta/task-shared-meta-reducers/` | Task-related multi-entity changes | -| `src/app/root-store/meta/task-shared-meta-reducers/tag-shared.reducer.ts` | Tag deletion with cleanup | -| `src/app/root-store/meta/task-shared-meta-reducers/project-shared.reducer.ts` | Project deletion with cleanup | +| File | Purpose | +| ------------------------------------------------------------------------------ | --------------------------------------------------------- | +| `src/app/root-store/meta/task-shared-meta-reducers/` | Task-related multi-entity changes | +| `src/app/root-store/meta/task-shared-meta-reducers/tag-shared.reducer.ts` | Tag deletion with cleanup | +| `src/app/root-store/meta/task-shared-meta-reducers/project-shared.reducer.ts` | Project deletion with cleanup | +| `src/app/root-store/meta/task-shared-meta-reducers/lww-update.meta-reducer.ts` | LWW Update handling (adapter/singleton/relationship sync) | diff --git a/docs/sync-and-op-log/diagrams/06-archive-operations.md b/docs/sync-and-op-log/diagrams/06-archive-operations.md index 80dcb05eb7..0de1431b87 100644 --- a/docs/sync-and-op-log/diagrams/06-archive-operations.md +++ b/docs/sync-and-op-log/diagrams/06-archive-operations.md @@ -161,6 +161,61 @@ The server guarantees operations arrive in sequence order, and delete operations | `deleteTaskRepeatCfg` | ArchiveOperationHandlerEffects → ArchiveOperationHandler | ArchiveOperationHandler removes repeatCfgId from tasks | | `deleteIssueProvider` | ArchiveOperationHandlerEffects → ArchiveOperationHandler | ArchiveOperationHandler unlinks issue data | +## Archive Resurrection Prevention (Two-Level Defense) + +When multiple clients are syncing concurrently, a race condition can cause archived tasks to "resurrect" — reappearing in the active store after being archived. This happens when a field-level LWW Update (e.g., rename, time tracking) arrives for a task that was concurrently archived. + +The system uses a **two-level defense** to prevent this: + +### Level 1: ConflictResolutionService (Archive-Wins Rule) + +During LWW conflict resolution, if a `moveToArchive` operation conflicts with a field-level update, the **archive always wins** regardless of timestamps. This prevents the LWW update from overriding the archive intent. + +```mermaid +flowchart TD + subgraph Level1["Level 1: Conflict Resolution"] + C1["Conflict: moveToArchive vs field update"] + C1 --> C2{"Archive-Wins
Rule"} + C2 -->|"Archive wins"| C3["Create new archive op
with merged vector clock"] + C2 -->|"No archive involved"| C4["Normal LWW
timestamp comparison"] + end + + style Level1 fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px +``` + +**Key file:** `src/app/op-log/sync/conflict-resolution.service.ts` + +### Level 2: bulkOperationsMetaReducer (Pre-Scan Filtering) + +During bulk operation application (sync/hydration), the meta-reducer **pre-scans** the entire batch for `TASK_SHARED_MOVE_TO_ARCHIVE` operations. It collects all entity IDs being archived, then **skips** any `[TASK] LWW Update` operations targeting those entities. + +This handles the **3+ client scenario** where LWW Updates can appear before or after archive ops in the same batch, bypassing Level 1 conflict resolution. + +```mermaid +flowchart TD + subgraph Level2["Level 2: Bulk Operations Meta-Reducer"] + B1["Receive batch of operations
[LWW Update, moveToArchive, ...]"] + B1 --> B2["PRE-SCAN: Collect all
entity IDs being archived"] + B2 --> B3["For each operation in batch:"] + B3 --> B4{"Is this an LWW Update
for an archived entity?"} + B4 -->|"Yes"| B5["⛔ SKIP
(prevents resurrection)"] + B4 -->|"No"| B6["✅ Apply normally"] + end + + style Level2 fill:#e3f2fd,stroke:#1565c0,stroke-width:2px + style B5 fill:#ffcdd2,stroke:#c62828,stroke-width:2px +``` + +**Key file:** `src/app/op-log/apply/bulk-hydration.meta-reducer.ts` + +### Why Two Levels? + +| Scenario | Level 1 (Conflict Resolution) | Level 2 (Bulk Pre-Scan) | +| ------------------------------------------------------- | ------------------------------------------ | ----------------------- | +| 2 clients: archive vs field update | ✅ Catches in LWW resolution | N/A (not in same batch) | +| 3+ clients: LWW Update arrives in same batch as archive | May not detect (already resolved upstream) | ✅ Catches via pre-scan | +| Hydration replay with mixed ops | N/A (not conflict resolution) | ✅ Catches via pre-scan | + ## Key Files | File | Purpose | @@ -168,5 +223,7 @@ The server guarantees operations arrive in sequence order, and delete operations | `src/app/op-log/apply/archive-operation-handler.service.ts` | **Unified** handler for all archive side effects (local AND remote) | | `src/app/op-log/apply/archive-operation-handler.effects.ts` | Routes local actions to ArchiveOperationHandler via LOCAL_ACTIONS | | `src/app/op-log/apply/operation-applier.service.ts` | Calls ArchiveOperationHandler after dispatching remote operations | +| `src/app/op-log/sync/conflict-resolution.service.ts` | Archive-wins rule during LWW conflict resolution | +| `src/app/op-log/apply/bulk-hydration.meta-reducer.ts` | Pre-scan archive filtering during bulk application | | `src/app/features/archive/archive.service.ts` | Local archive write logic (moveToArchive writes BEFORE dispatch) | | `src/app/features/archive/task-archive.service.ts` | Archive CRUD operations | diff --git a/docs/sync-and-op-log/operation-log-architecture.md b/docs/sync-and-op-log/operation-log-architecture.md index 8bafae0eeb..11af63a22f 100644 --- a/docs/sync-and-op-log/operation-log-architecture.md +++ b/docs/sync-and-op-log/operation-log-architecture.md @@ -1180,8 +1180,8 @@ interface OperationSyncCapable { async uploadPendingOps(syncProvider: OperationSyncCapable): Promise { const pendingOps = await this.opLogStore.getUnsynced(); - // Upload in batches (up to 100 ops per request) - for (const chunk of chunkArray(pendingOps, 100)) { + // Upload in batches (up to 25 ops per request) + for (const chunk of chunkArray(pendingOps, 25)) { const response = await syncProvider.uploadOps( chunk.map(entry => toSyncOperation(entry.op)), clientId, @@ -1415,6 +1415,45 @@ When operations are rejected (either local or remote): - `getUnsynced()` excludes rejected ops (won't re-upload) - Compaction may eventually delete old rejected ops +### Archive-Wins Rule + +When a `moveToArchive` operation conflicts with a field-level update (e.g., rename, time tracking changes), the archive operation **always wins** regardless of timestamps. This bypasses the normal LWW timestamp comparison because archiving represents explicit user intent that should not be reversed by a concurrent field update. + +**Rationale:** If Client A archives a task and Client B concurrently renames it, the archive must win — otherwise, the LWW update would "resurrect" the archived task back into the active store by replacing its state. + +**Implementation:** `ConflictResolutionService` checks whether either the local or remote side contains a `TASK_SHARED_MOVE_TO_ARCHIVE` action. If so, the archive side wins automatically, and a new archive operation is created with a merged vector clock (via `_createArchiveWinOp()`). + +This is the **first level** of archive resurrection prevention. The **second level** is in the `bulkOperationsMetaReducer` (see [Area 10: Bulk Application](#area-10-bulk-application) in the quick reference), which pre-scans operation batches for archive operations and skips any LWW Update operations targeting entities being archived in the same batch. This two-level defense handles the 3+ client scenario where LWW Updates can arrive before or after archive ops in the same batch. + +**Key files:** + +- `src/app/op-log/sync/conflict-resolution.service.ts` — Archive-wins check and `_createArchiveWinOp()` +- `src/app/op-log/apply/bulk-hydration.meta-reducer.ts` — Pre-scan archive filtering + +### Stale Operation Handling for moveToArchive + +The `StaleOperationResolverService` treats `moveToArchive` as a special case alongside DELETE operations. When a `moveToArchive` op is rejected by the server due to concurrent conflicts, it is **re-created with a merged vector clock** instead of being discarded. + +This is necessary because `moveToArchive` removes entities from the NgRx store (via the archive reducer), so `getCurrentEntityState()` returns `undefined` for archived entities. Without this special handling, the stale operation resolver would be unable to re-create the operation, and archived tasks would be lost. + +**Implementation:** Before entity-by-entity processing, `StaleOperationResolverService` identifies bulk semantic operations like `moveToArchive` and re-creates them with the original payload and a merged vector clock, preserving the full task data in `MultiEntityPayload` format. + +**Key file:** `src/app/op-log/sync/stale-operation-resolver.service.ts` + +### Singleton Entity LWW Updates + +The `lwwUpdateMetaReducer` handles LWW Update actions (created when the local side wins a conflict) differently depending on the entity's storage pattern: + +| Storage Pattern | Entity Types | LWW Update Behavior | +| --------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------- | +| **Adapter** | TASK, PROJECT, TAG, NOTE, TASK_REPEAT_CFG, etc. | Individual entity replacement via NgRx entity adapter (`updateOne` or `addOne`) | +| **Singleton** | GLOBAL_CONFIG, TIME_TRACKING, MENU_TREE, WORK_CONTEXT | Entire feature state replaced with the winning data | +| **Unsupported** | Map, array, virtual patterns | Logged as warning; not supported for LWW | + +For **adapter entities**, the meta-reducer also syncs relationships (e.g., `project.taskIds` when `projectId` changes, `tag.taskIds` when `tagIds` changes, `TODAY_TAG.taskIds` when `dueDay` changes, `parent.subTaskIds` when `parentId` changes). + +**Key file:** `src/app/root-store/meta/task-shared-meta-reducers/lww-update.meta-reducer.ts` + ### User Notification A non-blocking snack notification is shown after auto-resolution: diff --git a/docs/sync-and-op-log/operation-rules.md b/docs/sync-and-op-log/operation-rules.md index 546e08ce2c..0abd382abf 100644 --- a/docs/sync-and-op-log/operation-rules.md +++ b/docs/sync-and-op-log/operation-rules.md @@ -133,7 +133,7 @@ This document establishes the core rules and principles for designing the Operat - **Rule:** Normal operations should be batched with reasonable limits. - **Limits:** - - **Max batch size:** 100 operations per batch for normal sync uploads. + - **Max batch size:** 25 operations per batch for normal sync uploads. - **Max payload size:** 1 MB per batch to prevent timeout issues. - **Exception:** `SYNC_IMPORT` and `BACKUP_IMPORT` bypass these limits but must be clearly marked as bulk operations and trigger immediate snapshot creation afterward. diff --git a/docs/sync-and-op-log/quick-reference.md b/docs/sync-and-op-log/quick-reference.md index a3c39856e6..c1f7c9974e 100644 --- a/docs/sync-and-op-log/quick-reference.md +++ b/docs/sync-and-op-log/quick-reference.md @@ -245,18 +245,27 @@ Last-Write-Wins automatically resolves conflicts using timestamps. **Key Invariants:** -| Invariant | Reason | -| --------------------------------- | --------------------------------------------- | -| Preserve original timestamp | Prevents unfair advantage in future conflicts | -| Merge ALL clocks | New op dominates everything known | -| Reject ALL pending ops for entity | Prevents stale ops from being uploaded | -| Mark rejected BEFORE applying | Crash safety | -| Remote wins on tie | Server-authoritative | +| Invariant | Reason | +| --------------------------------- | -------------------------------------------------------------------------- | +| Archive-wins rule | `moveToArchive` always wins over field-level updates, bypassing timestamps | +| Preserve original timestamp | Prevents unfair advantage in future conflicts | +| Merge ALL clocks | New op dominates everything known | +| Reject ALL pending ops for entity | Prevents stale ops from being uploaded | +| Mark rejected BEFORE applying | Crash safety | +| Remote wins on tie | Server-authoritative | + +**Stale Operation Special Cases:** + +| Operation Type | Stale Handling | +| --------------- | --------------------------------------------------------------------------------------------- | +| Regular UPDATE | Re-created with current entity state + merged clock | +| DELETE | Re-created with original payload + merged clock (entity gone from store) | +| `moveToArchive` | Re-created with original payload + merged clock (entity removed from NgRx by archive reducer) | **Key Files:** -- `conflict-resolution.service.ts` - LWW resolution -- `stale-operation-resolver.service.ts` - Stale op handling +- `conflict-resolution.service.ts` - LWW resolution + archive-wins rule +- `stale-operation-resolver.service.ts` - Stale op handling (incl. moveToArchive special case) --- @@ -362,51 +371,67 @@ Meta-reducers enable atomic multi-entity changes in a single reducer pass. ``` ┌─────────────────────────────────────────────────────────────────┐ -│ META-REDUCER CHAIN │ +│ META-REDUCER CHAIN (8 Phases, 15 Entries) │ ├─────────────────────────────────────────────────────────────────┤ │ │ │ Action Dispatched │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Phase 1: operationCaptureMetaReducer │ │ -│ │ Captures persistent actions → FIFO queue │ │ +│ │ Phase 1: operationCaptureMetaReducer [MUST BE FIRST] │ │ +│ │ Captures original state BEFORE modifications │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Phase 2: bulkOperationsMetaReducer │ │ -│ │ Applies bulk ops during sync/hydration │ │ +│ │ Unwraps bulk dispatches for hydration/sync │ │ +│ │ Pre-scans for archive ops to prevent │ │ +│ │ resurrection via LWW Update │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Phase 3: lwwUpdateMetaReducer │ │ -│ │ Handles LWW Update actions (conflict wins) │ │ +│ │ Phase 3: undoTaskDeleteMetaReducer │ │ +│ │ Captures task context before deletion │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Phase 4: Tag/Project/IssueProvider Shared Reducers │ │ -│ │ Cascade deletions across entities │ │ +│ │ Phase 4: Core CRUD Meta-Reducers (dependency order) │ │ +│ │ • taskSharedCrudMetaReducer │ │ +│ │ • taskBatchUpdateMetaReducer │ │ +│ │ • taskSharedLifecycleMetaReducer │ │ +│ │ • taskSharedSchedulingMetaReducer │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Phase 5: Planner/ShortSyntax Shared Reducers │ │ -│ │ Handle task movements and parsing │ │ +│ │ Phase 5: Entity-Specific Cascades │ │ +│ │ • projectSharedMetaReducer │ │ +│ │ • tagSharedMetaReducer │ │ +│ │ • issueProviderSharedMetaReducer │ │ +│ │ • taskRepeatCfgSharedMetaReducer │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Phase 6: Feature Reducers (task, project, tag, etc.) │ │ -│ │ Core state updates │ │ +│ │ Phase 6: plannerSharedMetaReducer │ │ +│ │ Syncs with task.dueDay and TODAY_TAG │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ -│ │ Phase 7: validateAndFixDataConsistency │ │ -│ │ Repair any inconsistencies │ │ +│ │ Phase 7: Synthetic Multi-Step Operations │ │ +│ │ • shortSyntaxSharedMetaReducer │ │ +│ │ • lwwUpdateMetaReducer │ │ +│ │ (adapter / singleton / unsupported patterns) │ │ +│ └─────────────────────────────────────────────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────────────────────────────────────────────────┐ │ +│ │ Phase 8: actionLoggerReducer [MUST BE LAST] │ │ +│ │ Pure logging only │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ @@ -415,6 +440,13 @@ Meta-reducers enable atomic multi-entity changes in a single reducer pass. └─────────────────────────────────────────────────────────────────┘ ``` +**Critical Ordering Rules:** + +- `operationCaptureMetaReducer` must be at index 0 (captures pre-modification state) +- `bulkOperationsMetaReducer` must be at index 1 (unwraps bulk before other reducers run) +- `actionLoggerReducer` must be last (logs final state) +- Development mode validates these constraints at startup + **Why Meta-Reducers for Multi-Entity Changes?** | Approach | Problem | @@ -544,8 +576,15 @@ Bulk application optimizes performance by applying many operations in a single d │ │ │ │ ▼ (in bulkOperationsMetaReducer) │ │ ┌────────────────────────────────────────┐ │ +│ │ // Pre-scan: collect archived IDs │ │ +│ │ archivedIds = findArchiveOps(ops) │ │ +│ │ │ │ │ │ for (op of operations) { │ │ │ │ action = convertOpToAction(op) │ │ +│ │ // Skip LWW Updates for archived │ │ +│ │ if (isLwwUpdate(action) && │ │ +│ │ archivedIds.has(entityId)) │ │ +│ │ continue; │ │ │ │ state = reducer(state, action) │ │ │ │ } │ │ │ │ return state │ │