From f8e0160801ebf0bb415ffbf65c66b48c3fdf50da Mon Sep 17 00:00:00 2001 From: Johannes Millan Date: Tue, 2 Dec 2025 14:54:46 +0100 Subject: [PATCH] docs: plan operation logs again 1 --- CLAUDE.md | 10 +- docs/ai/per-entity-delta-sync-plan.md | 187 ----- docs/ai/sync/operation-log-execution-plan.md | 691 ++++++++++++++++++ docs/ai/{ => sync}/operation-log-sync.md | 20 +- docs/ai/{ => sync}/operationlog-critique.md | 0 .../ai/webdav-non-etag-implementation-plan.md | 220 ------ docs/sync/SYNC-PLAN.md | 58 -- 7 files changed, 704 insertions(+), 482 deletions(-) delete mode 100644 docs/ai/per-entity-delta-sync-plan.md create mode 100644 docs/ai/sync/operation-log-execution-plan.md rename docs/ai/{ => sync}/operation-log-sync.md (98%) rename docs/ai/{ => sync}/operationlog-critique.md (100%) delete mode 100644 docs/ai/webdav-non-etag-implementation-plan.md delete mode 100644 docs/sync/SYNC-PLAN.md diff --git a/CLAUDE.md b/CLAUDE.md index b260a27f2..3826ecfb0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,14 +2,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. -## General Guidelines +## Project-Specific Guidelines -1. Prefer functional programming patterns: Use pure functions, immutability, and avoid side effects where possible. -2. KISS (Keep It Simple, Stupid): Aim for simplicity and clarity in code. Avoid unnecessary complexity and abstractions. -3. DRY (Don't Repeat Yourself): Reuse code where possible. Create utility functions or services for common logic, but avoid unnecessary abstractions. -4. Confirm understanding before making changes: If you're unsure about the purpose of a piece of code, ask for clarification rather than making assumptions. -5. **ALWAYS** use `npm run checkFile ` on each file you modify to ensure proper formatting and linting. This runs both prettier and lint checks on individual files. Unless you want to lint and format multiple files, then use `npm run prettier` and `npm run lint` instead. -6. When creating html templates, prefer plain html like `` and `
`. Keep CSS styles to a minimum. Keep nesting to a minimum. Keep css classes to a minimum. Use Angular Material components where appropriate, but avoid overusing them. +1. **ALWAYS** use `npm run checkFile ` on each `ts` or `scss` file you modify to ensure proper formatting and linting. Unless you want to lint and format multiple files, then use `npm run prettier` and `npm run lint` instead. +2. When creating HTML templates, prefer plain HTML (`
`, `
`). Keep CSS, nesting, and classes to a minimum. Use Angular Material components where appropriate but sparingly. ## Project Overview diff --git a/docs/ai/per-entity-delta-sync-plan.md b/docs/ai/per-entity-delta-sync-plan.md deleted file mode 100644 index 1db58294b..000000000 --- a/docs/ai/per-entity-delta-sync-plan.md +++ /dev/null @@ -1,187 +0,0 @@ -# Per-Entity Delta Sync: Planning Document - -## 1. Executive Summary - -This document outlines the implementation plan for a **Per-Entity Delta Sync** mechanism. This approach is proposed as a lightweight, robust alternative to the complex "Operation Log" architecture. It addresses the core requirements of **data loss prevention**, **conflict minimization**, and **bandwidth efficiency** without the overhead of event sourcing, replay logic, or dual data models. - -**Core Philosophy:** Keep the current snapshot-based model but make synchronization "entity-aware" rather than "file-aware". Use **Batched Deltas** to solve the "many small files" performance problem. - -## 2. Problem Statement - -The current "whole-file" sync approach suffers from: - -1. **Data Loss/Overwrites:** If two devices edit different tasks, the last write wins, overwriting the other device's changes to unrelated tasks. -2. **Bandwidth Inefficiency:** Changing one character in a task requires re-uploading the entire database. -3. **Conflict Rigidity:** Merging is difficult because the granularity is too coarse (the whole file). - -The "Operation Log" proposal solves these but introduces complexity (replaying events) and features not currently required (audit trails). - -## 3. Proposed Architecture: Per-Entity Delta Sync - -### 3.1. Data Model Extensions - -We track version metadata on every entity. - -#### 3.1.1. Entity Metadata - -Every syncable entity (Task, Project, Tag, Note) will implement a versioning interface: - -```typescript -interface EntitySyncMetadata { - id: string; - // Monotonic counter, incremented on every local update - version: number; - // Timestamp of last update (wall clock, for informational/conflict heuristics) - lastModifiedAt: number; - // ID of the device that made the last change - lastModifiedBy: string; - // Deleted flag (soft delete is REQUIRED for delta sync) - isDeleted?: boolean; -} -``` - -#### 3.1.2. Global Sync Metadata (`meta.json`) - -Clients first check a lightweight `meta.json` on the remote provider. - -```typescript -interface RemoteSyncMeta { - // Map of EntityID -> Version. - entityVersions: { - [entityId: string]: number; - }; - // Timestamp of the last successful sync - lastSyncTimestamp: number; - // Pointer to the "Base State" file (full backup) - baseStateFile: string; -} -``` - -### 3.2. Batched Storage Strategy (The "Middle Way") - -To avoid the performance penalty of thousands of small files (WebDAV latency) while maintaining delta capabilities: - -1. **Base State:** `main.json` (or `snapshot_TIMESTAMP.json`) exists as a fallback/bootstrap. -2. **Delta Batches:** When a client syncs, it uploads a **single** JSON file containing _only_ the entities that changed. - - Filename: `delta_{deviceId}_{timestamp}.json` - - Content: `{ tasks: [TaskA, TaskB], projects: [ProjectX] }` -3. **Compaction:** Periodically (e.g., every 10 deltas or once a week), a client downloads the Base + all Deltas, merges them, and uploads a new Base State, deleting old deltas. - -### 3.3. Synchronization Algorithm - -1. **Local Change**: - - - User updates Task A. - - App increments `TaskA.version`. - - App updates local `meta.entityVersions['TaskA']`. - -2. **Sync Process (Initiator)**: - - **Lock**: Acquire remote lock. - - **Read Remote Meta**: Download `meta.json`. - - **Diff**: Identify entities where `Remote.ver > Local.ver` (Incoming) and `Local.ver > Remote.ver` (Outgoing). - - **Download Step (Incoming)**: - - Identify which `delta_*.json` files contain the needed incoming entities (requires `meta.json` to map ID->File, or just download all new delta files since last sync). - - **Apply Incoming**: Merge entities into local DB. **Apply in Topological Order** (Projects first, then Tasks) to satisfy dependencies. - - **Upload Step (Outgoing)**: - - Bundle all local dirty entities into **one** `delta_{myId}_{now}.json`. - - Upload the file. - - Update `meta.json` with new versions and add the new delta file to the list. - - **Unlock**. - -## 4. Cross-Model Relationships & Integrity - -Unlike strict SQL databases, our client-side logic must handle referential integrity manually during the sync merge. - -### 4.1. Dependency Order (Topological Apply) - -When applying a batch of incoming changes, order matters. - -1. **Projects** (Must exist before Tasks assigned to them) -2. **Tags** (Must exist before being referenced) -3. **Tasks** (Parent tasks first, then Subtasks) -4. **Notes** (Attached to Projects/Tasks) - -_Implementation:_ The `applyDelta()` function must sort the incoming entities by type before upserting to IndexedDB. - -### 4.2. Cascading Deletes (Client-Side Responsibility) - -In an Operation Log, a "Delete Project" event triggers a reducer that deletes tasks. In Delta Sync, the **deleting client** acts as the reducer. - -- **Scenario:** User deletes "Project A". -- **Client A Logic:** - 1. Mark "Project A" as `isDeleted: true`. - 2. Find all Tasks in "Project A". - 3. **Update them:** Set `projectId = null` (Inbox) or mark `isDeleted: true` (depending on app policy). - 4. Sync: Uploads a Delta containing { ProjectA (deleted), Task1 (updated), Task2 (updated) }. -- **Client B Logic:** - 1. Receives the Delta. - 2. Upserts the entities. "Project A" is removed/hidden. "Task 1" moves to Inbox. - 3. _Benefit:_ Client B doesn't need complex cascade logic; it just trusts the state. - -### 4.3. Orphan Handling - -If Client B receives a Task pointing to "Project Z" which it doesn't have (e.g., partial sync failure or out-of-order arrival): - -- **Soft Dependency (Tags):** Ignore the missing tag reference. -- **Hard Dependency (Parent Task):** - - _Option 1:_ Queue the orphan until parent arrives. - - _Option 2 (Simpler):_ Display as a top-level task until parent arrives (self-healing). - -## 5. Conflict Resolution - -Since we track versions per entity, conflicts are scoped to a single entity. - -### 5.1. Entity-Level Conflict - -- **Scenario:** Device A renames Task 1. Device B completes Task 1. -- **Resolution:** **Field-Level Merge**. - - Result: Task 1 is renamed AND completed. - - Version: `max(vA, vB) + 1`. - -### 5.2. Relational Conflict (The "Zombie" Case) - -- **Scenario:** Device A edits "Task 1" (in "Project X"). Device B deletes "Project X". -- **Result:** - - Device A syncs: "Task 1" (v2) points to "Project X". - - Device B syncs: "Project X" (deleted). - - **Outcome:** "Task 1" is now an orphan (points to non-existent project). -- **Fix:** The App's "Consistency Check" (run on startup/sync completion) detects tasks pointing to missing projects and moves them to **Inbox**. - -## 6. Implementation Plan - -### Phase 0: Preparation - -1. **Data Audit:** Ensure `id` is stable. -2. **Schema Update:** Add `version`, `lastModifiedAt`, `isDeleted` to `Task`, `Project`, `Tag`. - -### Phase 1: The "Batched Delta" Storage - -1. **Delta Writer:** Create service to dump "dirty" entities to a JSON file. -2. **Delta Reader:** Create service to read a JSON file and upsert entities to NgRx/IndexedDB. -3. **Ordering Logic:** Ensure `DeltaReader` applies `Projects` -> `Tags` -> `Tasks`. - -### Phase 2: The Sync Loop - -1. **Meta Sync:** Read/Write `meta.json` containing `entityVersions`. -2. **Integration:** Hook into existing `sync.service.ts`. Replace "Overwrite All" with "Upload Delta + Update Meta". - -### Phase 3: Integrity & Cleanup - -1. **Cascade Logic:** Ensure UI "Delete" actions explicitly update related entities (moving tasks to Inbox) before syncing. -2. **Orphan Sweeper:** Implement a startup check to move orphaned tasks to Inbox. -3. **Compaction:** Logic to merge `base.json` + `delta_1..10.json` -> new `base.json`. - -## 7. Comparison vs Operation Log - -| Feature | Per-Entity Delta Sync | Operation Log | -| :------------- | :---------------------------------- | :------------------------------------- | -| **State** | **Smart Snapshots** (Current State) | **Events** (History of Actions) | -| **Files** | Base + Batched Deltas (Low Count) | Many Op Files (High Count) | -| **Logic** | "Dumb" Merge (Trust the data) | Replay Reducer (Calculate state) | -| **Relations** | Source Client calculates cascades | Destination Client calculates cascades | -| **Conflict** | Field-level merge + Last Write Wins | User Intent Preservation | -| **Complexity** | **Medium** | **Very High** | - -## 8. Conclusion - -The **Batched Per-Entity Delta Sync** solves the bandwidth and data-loss problems while adhering to the "File Count" constraints. By shifting the responsibility of "Cascade Deletes" to the _source_ client (recording the effects, not the cause), we avoid the complexity of an Event Sourcing replay engine. diff --git a/docs/ai/sync/operation-log-execution-plan.md b/docs/ai/sync/operation-log-execution-plan.md new file mode 100644 index 000000000..76a3bdf45 --- /dev/null +++ b/docs/ai/sync/operation-log-execution-plan.md @@ -0,0 +1,691 @@ +# Operation Log Sync: Execution Plan + +**Created:** December 2, 2025 +**Branch:** `feat/operation-logs` +**Status:** Implementation in Progress + +--- + +## 1. Executive Summary + +The Operation Log sync system provides per-entity conflict detection with semantic merge capabilities, replacing the whole-file Last-Writer-Wins (LWW) approach. The current implementation (~800+ lines across 16 files) has a solid foundation: IndexedDB persistence, NgRx effect capture, vector clock conflict detection, multi-tab coordination, and genesis migration. Key gaps remain in **conflict resolution UX**, **dependency retry logic**, **effect guards during replay**, and **comprehensive testing**. This plan outlines 5 phases to bring the system to production-ready status. + +--- + +## 2. Assessment Findings + +### 2.1 What's Implemented (Working) + +| Component | File | Status | Notes | +| -------------------- | ------------------------------------------------- | ----------- | -------------------------------------------------------------------- | +| Operation Types | `operation.types.ts` | ✅ Complete | Well-defined types for `Operation`, `EntityConflict`, `VectorClock` | +| IndexedDB Store | `operation-log-store.service.ts` | ✅ Complete | `SUP_OPS` database with ops + state_cache stores, indexes | +| Effect Capture | `operation-log.effects.ts` | ✅ Complete | Captures persistent actions, increments vector clock, appends to log | +| Sync Upload/Download | `operation-log-sync.service.ts` | ✅ Complete | Manifest-based chunked file sync, deduplication | +| Conflict Detection | `operation-log-sync.service.ts:detectConflicts()` | ✅ Complete | Per-entity vector clock comparison | +| Hydrator | `operation-log-hydrator.service.ts` | ✅ Complete | Snapshot + tail replay on startup | +| Compaction | `operation-log-compaction.service.ts` | ✅ Complete | 7-day retention window, snapshot creation | +| Multi-Tab Sync | `multi-tab-coordinator.service.ts` | ✅ Complete | BroadcastChannel API coordination | +| Genesis Migration | `operation-log-migration.service.ts` | ✅ Complete | Legacy data → first operation | +| Lock Service | `lock.service.ts` | ✅ Complete | Web Locks API + localStorage fallback | +| Action Converter | `operation-converter.util.ts` | ✅ Complete | Op → Action with `isRemote` flag | +| Dependency Extractor | `dependency-resolver.service.ts` | 🚧 Partial | Extracts deps, but no retry queue | + +### 2.2 What's Stubbed or Incomplete + +| Component | File | Gap | Impact | +| -------------------- | ----------------------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------- | +| **Conflict UI** | `dialog-conflict-resolution.component.ts` | Basic local/remote choice only; no field-level diff, no merge preview | Users can't make informed decisions | +| **Conflict Service** | `conflict-resolution.service.ts:51-54` | TODO: Revert/remove local ops on "remote wins" | Potential state inconsistency | +| **Dependency Retry** | `operation-applier.service.ts:20-21` | TODO: Queue + retry for missing hard deps | Subtasks orphaned if parent arrives later | +| **Smart Resolution** | `operation-log-sync.service.ts:278` | TODO: `suggestResolution` always returns 'manual' | No auto-merge for trivial conflicts | +| **Action Whitelist** | `action-whitelist.ts` | Only 9 blacklisted actions | Need explicit whitelist per entity type | +| **Effect Guards** | `operation-log.effects.ts` | No replay guard flag | Side effects (notifications, analytics) may fire during replay | +| **Error Recovery** | `operation-log.effects.ts:77-80` | Commented out rollback | Optimistic updates not recoverable | +| **Testing** | `*.spec.ts` | Only 1 spec file (multi-tab) | No coverage for sync, compaction, hydration | + +### 2.3 Architectural Observations + +1. **Integration Point**: Operation log sync runs _before_ legacy full-file sync in `sync.service.ts:99-111`. This is correct for incremental adoption. + +2. **Replay Safety**: `convertOpToAction()` sets `isRemote: true`, which correctly prevents re-logging in effects. However, NgRx effects other than `OperationLogEffects` may still trigger (e.g., notification effects). + +3. **Compaction Risk**: Deleting synced ops older than 7 days is safe, but if a device is offline for >7 days, it may miss ops that were compacted away. The snapshot should contain the full state, but conflict detection loses granularity. + +4. **Manifest Consistency**: The manifest file is overwritten atomically (`forceOverwrite: true`), which is correct for eventual consistency but may cause issues if two devices upload simultaneously. + +--- + +## 3. Phased Implementation Plan + +### Phase 1: Core Stability & Safety Guards + +**Objective:** Ensure replay and sync operations don't trigger unintended side effects. + +**Duration Estimate:** 1 week + +#### 1.1 Add Replay Guard Flag + +**Files to modify:** + +- `src/app/core/persistence/operation-log/operation-log-hydrator.service.ts` +- New: `src/app/core/persistence/operation-log/replay-guard.service.ts` + +**Implementation:** + +```typescript +// replay-guard.service.ts +@Injectable({ providedIn: 'root' }) +export class ReplayGuardService { + private _isReplaying = signal(false); + readonly isReplaying = this._isReplaying.asReadonly(); + + enterReplayMode(): void { + this._isReplaying.set(true); + } + exitReplayMode(): void { + this._isReplaying.set(false); + } +} +``` + +**Changes to hydrator:** + +```typescript +async hydrateStore(): Promise { + this.replayGuard.enterReplayMode(); + try { + // ... existing hydration logic ... + } finally { + this.replayGuard.exitReplayMode(); + } +} +``` + +**Changes to effects that shouldn't fire during replay:** + +- Inject `ReplayGuardService` and add `filter(() => !this.replayGuard.isReplaying())` +- Identify effects: notification effects, analytics, external API calls + +**Acceptance Criteria:** + +- [ ] Replay flag is set during `hydrateStore()` and remote op application +- [ ] Notification effects don't fire during hydration +- [ ] Unit test verifies flag state transitions + +#### 1.2 Complete Action Whitelist + +**File to modify:** `src/app/core/persistence/operation-log/action-whitelist.ts` + +**Implementation:** + +- Audit all NgRx actions in `src/app/features/*/store/*.actions.ts` +- Create explicit whitelist map: `Map` +- Update effect to use whitelist instead of blacklist + +**Deliverable:** Complete mapping of ~50-100 action types to entity types + +**Acceptance Criteria:** + +- [ ] Every persistent action has explicit entity type mapping +- [ ] Unit test verifies all feature actions are categorized + +#### 1.3 Error Recovery for Optimistic Updates + +**File to modify:** `src/app/core/persistence/operation-log/operation-log.effects.ts` + +**Implementation:** + +```typescript +// On persist failure, dispatch a compensating action +private notifyUserAndTriggerRollback(action: PersistentAction): void { + // Show notification to user + this.snackbarService.showError('Failed to save change. Please retry.'); + // Optionally: Dispatch inverse action or reload state + // For now, just warn - full rollback requires inverse action generation +} +``` + +**Acceptance Criteria:** + +- [ ] User sees notification on persist failure +- [ ] System remains in consistent state (no partial writes) + +--- + +### Phase 2: Dependency Resolution & Retry + +**Objective:** Handle operations that arrive before their dependencies (e.g., subtask before parent task). + +**Duration Estimate:** 1 week + +#### 2.1 Implement Retry Queue + +**Files to modify:** + +- `src/app/core/persistence/operation-log/dependency-resolver.service.ts` +- `src/app/core/persistence/operation-log/operation-applier.service.ts` + +**Implementation:** + +```typescript +// dependency-resolver.service.ts - add retry queue +interface PendingOp { + op: Operation; + missingDeps: OperationDependency[]; + retryCount: number; + addedAt: number; +} + +private pendingQueue: PendingOp[] = []; +private readonly MAX_RETRIES = 5; +private readonly RETRY_DELAY_MS = 1000; + +queueForRetry(op: Operation, missingDeps: OperationDependency[]): void { + this.pendingQueue.push({ op, missingDeps, retryCount: 0, addedAt: Date.now() }); + this.scheduleRetry(); +} + +private async retryPending(): Promise { + const stillPending: PendingOp[] = []; + for (const item of this.pendingQueue) { + const { missing } = await this.checkDependencies(item.missingDeps); + if (missing.length === 0) { + await this.operationApplier.applyOperations([item.op]); + } else if (item.retryCount < this.MAX_RETRIES) { + item.retryCount++; + stillPending.push(item); + } else { + PFLog.error('Dropping op after max retries', { op: item.op, missingDeps: missing }); + // TODO: Surface to user or create orphan recovery UI + } + } + this.pendingQueue = stillPending; + if (stillPending.length > 0) this.scheduleRetry(); +} +``` + +**Changes to operation-applier.service.ts:** + +```typescript +if (missingHardDeps.length > 0) { + this.dependencyResolver.queueForRetry(op, missingHardDeps); + continue; +} +``` + +**Acceptance Criteria:** + +- [ ] Subtask op queued if parent doesn't exist yet +- [ ] Parent op arrival triggers retry, subtask applies successfully +- [ ] Ops dropped after MAX_RETRIES with error log +- [ ] Unit tests for queue/retry logic + +#### 2.2 Add Dependency Types for All Entities + +**File to modify:** `src/app/core/persistence/operation-log/dependency-resolver.service.ts` + +**Implementation:** + +- Add `checkEntityExists` cases for: `TAG`, `NOTE`, `SIMPLE_COUNTER`, `WORK_CONTEXT`, `TASK_REPEAT_CFG`, `ISSUE_PROVIDER` +- Add selectors for each entity type + +**Acceptance Criteria:** + +- [ ] All entity types have dependency checking +- [ ] Integration test: cross-entity references resolve correctly + +--- + +### Phase 3: Conflict Resolution UI & Logic + +**Objective:** Enable users to make informed conflict resolution decisions with field-level visibility. + +**Duration Estimate:** 2 weeks + +#### 3.1 Enhanced Conflict Data Structure + +**File to modify:** `src/app/core/persistence/operation-log/operation.types.ts` + +**Implementation:** + +```typescript +export interface FieldDiff { + field: string; + localValue: unknown; + remoteValue: unknown; + isMergeable: boolean; // true if different fields changed +} + +export interface EnhancedEntityConflict extends EntityConflict { + fieldDiffs: FieldDiff[]; + canAutoMerge: boolean; + autoMergedPayload?: unknown; + entitySnapshot?: { + local: unknown; + remote: unknown; + base?: unknown; // Common ancestor if available + }; +} +``` + +#### 3.2 Smart Resolution Suggestions + +**File to modify:** `src/app/core/persistence/operation-log/conflict-resolution.service.ts` + +**Implementation:** + +```typescript +suggestResolution(conflict: EntityConflict): EnhancedEntityConflict { + const localPayload = this.extractLatestPayload(conflict.localOps); + const remotePayload = this.extractLatestPayload(conflict.remoteOps); + const fieldDiffs = this.computeFieldDiffs(localPayload, remotePayload); + + // Auto-merge if different fields changed + const localFields = new Set(fieldDiffs.filter(d => d.localValue !== undefined).map(d => d.field)); + const remoteFields = new Set(fieldDiffs.filter(d => d.remoteValue !== undefined).map(d => d.field)); + const overlappingFields = [...localFields].filter(f => remoteFields.has(f)); + + const canAutoMerge = overlappingFields.length === 0; + const autoMergedPayload = canAutoMerge + ? { ...localPayload, ...remotePayload } // Non-overlapping merge + : undefined; + + return { + ...conflict, + fieldDiffs, + canAutoMerge, + autoMergedPayload, + suggestedResolution: canAutoMerge ? 'merge' : + (conflict.localOps[0].timestamp > conflict.remoteOps[0].timestamp ? 'local' : 'remote'), + }; +} +``` + +#### 3.3 Enhanced Conflict Resolution UI + +**Files to modify:** + +- `src/app/imex/sync/dialog-conflict-resolution/dialog-conflict-resolution.component.ts` +- `src/app/imex/sync/dialog-conflict-resolution/dialog-conflict-resolution.component.html` + +**Implementation:** + +- Show field-level diff for each conflict +- Display timestamps and device IDs +- Offer "Auto-merge" button when `canAutoMerge: true` +- Allow per-field resolution (advanced mode) +- Show preview of resulting state before confirmation + +**UI Mockup:** + +``` ++---------------------------------------------------------------+ +| Sync Conflict: Task "Meeting notes" | ++---------------------------------------------------------------+ +| Field | Local (Desktop, 2m ago) | Remote (Phone, 5m ago)| +|-------------|-------------------------|-----------------------| +| title | "Meeting notes v2" | "Meeting notes" | +| isDone | false | > true | +| timeSpent | 3600 | 3600 (same) | ++---------------------------------------------------------------+ +| Suggestion: Can auto-merge (different fields changed) | +| | +| [Auto-merge] [Use Local] [Use Remote] [Resolve per field] | ++---------------------------------------------------------------+ +``` + +**Acceptance Criteria:** + +- [ ] Field diffs computed and displayed +- [ ] Auto-merge works for non-overlapping changes +- [ ] User can choose local/remote/merge per conflict +- [ ] Preview shows final state before confirmation +- [ ] E2E test: simulate conflict, verify UI renders correctly + +#### 3.4 Implement "Remote Wins" State Cleanup + +**File to modify:** `src/app/core/persistence/operation-log/conflict-resolution.service.ts` + +**Implementation:** + +```typescript +// When user chooses "remote", we need to: +// 1. Apply remote ops to state +// 2. Mark local conflicting ops as "superseded" (not deleted, for audit trail) +// 3. Optionally: generate compensating ops to undo local changes + +if (resolution === 'remote') { + // Mark local ops as superseded + for (const localOp of conflict.localOps) { + await this.opLogStore.markSuperseded(localOp.id, conflict.remoteOps[0].id); + } + // Apply remote ops + await this.operationApplier.applyOperations(conflict.remoteOps); +} +``` + +**New method in operation-log-store.service.ts:** + +```typescript +async markSuperseded(opId: string, supersededById: string): Promise { + // Add supersededBy field to entry (for audit) + // This op is still in log but won't be re-synced +} +``` + +--- + +### Phase 4: Testing & Hardening + +**Objective:** Comprehensive test coverage for all sync scenarios. + +**Duration Estimate:** 2 weeks + +#### 4.1 Unit Tests + +**New test files:** + +- `operation-log-store.service.spec.ts` +- `operation-log-sync.service.spec.ts` +- `operation-log-hydrator.service.spec.ts` +- `operation-log-compaction.service.spec.ts` +- `conflict-resolution.service.spec.ts` +- `dependency-resolver.service.spec.ts` +- `operation-applier.service.spec.ts` + +**Test scenarios:** + +1. **Store Service:** + + - Append ops and verify sequence + - Get unsynced ops + - Mark synced + - Delete old ops (compaction) + - Vector clock accumulation + +2. **Sync Service:** + + - Upload pending ops → verify manifest updated + - Download remote ops → verify applied + - Conflict detection (CONCURRENT vector clocks) + - Non-conflicting ops (one HAPPENED_BEFORE other) + +3. **Hydrator:** + + - Cold start with snapshot + tail ops + - Cold start with no snapshot (full replay) + - Migration path (legacy data → genesis op) + +4. **Compaction:** + + - Snapshot creation + - Old op deletion (respecting retention window) + - Never delete unsynced ops + +5. **Conflict Resolution:** + - Field diff computation + - Auto-merge for non-overlapping + - Manual resolution paths + +#### 4.2 Integration Tests + +**New test file:** `operation-log-integration.spec.ts` + +**Scenarios:** + +1. Two clients edit same task → conflict detected +2. Two clients edit different tasks → no conflict, both apply +3. Client A creates subtask, Client B creates parent → dependency resolution +4. Client offline 8+ days → syncs via snapshot, no missing ops + +#### 4.3 E2E Tests (Playwright) + +**New test file:** `e2e/tests/sync/operation-log-sync.spec.ts` + +**Scenarios:** + +1. Create task on Device A → sync → verify on Device B (mocked) +2. Concurrent edits → conflict dialog appears → resolve → state consistent +3. Large dataset (1000 tasks) → sync completes in <5s +4. Offline mode → queue ops → reconnect → sync succeeds + +#### 4.4 Performance Benchmarks + +**Targets:** + +- Startup with 10k ops in log: <2s +- Incremental sync (100 new ops): <3s +- Compaction (50k ops → snapshot): <5s + +**Benchmark script:** `scripts/benchmark-oplog.ts` + +--- + +### Phase 5: Rollout & Migration + +**Objective:** Safe rollout with feature flag and rollback capability. + +**Duration Estimate:** 1 week + +#### 5.1 Feature Flag + +**File to modify:** `src/app/features/config/global-config.model.ts` + +**Implementation:** + +```typescript +export interface SyncConfig { + // ... existing ... + useOperationLogSync: boolean; // Default: false +} +``` + +**Conditional in sync.service.ts:** + +```typescript +if (this._globalConfigService.cfg.sync.useOperationLogSync) { + await this._operationLogSyncService.uploadPendingOps(currentSyncProvider); + await this._operationLogSyncService.downloadRemoteOps(currentSyncProvider); +} +// Legacy sync continues regardless (hybrid mode during rollout) +``` + +#### 5.2 Hybrid Sync Mode + +During rollout, both sync systems run: + +1. Operation log sync handles incremental changes +2. Legacy full-file sync provides safety net + +This ensures: + +- Users can disable oplog sync if issues arise +- Full state is still backed up to remote +- Gradual confidence building + +#### 5.3 Migration Path + +**For new users:** + +- Operation log starts empty +- Genesis op created on first sync if legacy data exists + +**For existing users:** + +- Enable feature flag +- Genesis migration runs on next startup +- Both sync systems active + +**Rollback:** + +- Disable feature flag +- Legacy sync continues working +- Operation log data preserved but unused + +#### 5.4 Monitoring & Alerts + +**Metrics to track:** + +- Operation log size (ops count, bytes) +- Sync duration (upload/download) +- Conflict rate (per day) +- Resolution choices (local/remote/merge distribution) +- Dependency retry count +- Compaction frequency + +**Log events:** + +```typescript +PFLog.metric('oplog_sync_complete', { + uploadedOps: n, + downloadedOps: m, + conflicts: c, + durationMs: t, +}); +``` + +--- + +## 4. Critical Decisions Needed + +### 4.1 Architecture Decisions + +| Decision | Options | Recommendation | Status | +| ----------------------- | --------------------------------------------------------------- | --------------------------- | ----------------- | +| Hybrid vs. Replace sync | A) Run oplog alongside legacy B) Replace legacy entirely | A) Hybrid during rollout | ❓ Needs approval | +| Conflict auto-merge | A) Always manual B) Auto for non-overlapping C) User preference | B) Auto for non-overlapping | ❓ Needs approval | +| Compaction retention | A) 7 days B) 14 days C) Configurable | A) 7 days (current) | ✅ Decided | +| Offline tolerance | How long offline before snapshot-only sync? | 7 days (matches compaction) | ❓ Needs approval | + +### 4.2 UX Decisions + +| Decision | Options | Recommendation | Status | +| ---------------------- | -------------------------------------------------------------- | ------------------------- | ----------------- | +| Conflict notification | A) Modal dialog B) Non-blocking notification C) Both | A) Modal (current) | ❓ Needs approval | +| Field-level resolution | A) Always available B) Advanced mode toggle C) Not implemented | B) Advanced mode toggle | ❓ Needs approval | +| Conflict defer | Can user dismiss conflict dialog and resolve later? | Yes, with badge indicator | ❓ Needs approval | + +### 4.3 Technical Decisions + +| Decision | Options | Recommendation | Status | +| -------------------- | ---------------------------------------------------------------------- | -------------------------------------------------------- | ---------- | +| Manifest locking | A) Optimistic (current) B) Pessimistic (file lock) C) CRDT-style merge | A) Optimistic is sufficient | ✅ Decided | +| Op file chunking | A) 100 ops/file (current) B) Time-based C) Size-based | A) 100 ops/file | ✅ Decided | +| Vector clock pruning | A) Never B) After 30 days C) After device removal | B) After 30 days (implemented in `limitVectorClockSize`) | ✅ Decided | + +--- + +## 5. Validation Checklist + +### Phase 1 Validation + +- [ ] Run `npm test` - all existing tests pass +- [ ] Manual: Create task during hydration → no duplicate notifications +- [ ] Manual: Persist failure → user sees error message + +### Phase 2 Validation + +- [ ] Unit test: Dependency retry queue works +- [ ] Manual: Create subtask, then parent on different device → both sync correctly +- [ ] Manual: After 5 retries, orphan op is logged with error + +### Phase 3 Validation + +- [ ] Unit test: Field diff computation matches expected output +- [ ] Manual: Edit same task on two devices → conflict UI shows field diff +- [ ] Manual: Auto-merge works for different fields +- [ ] E2E: Full conflict resolution flow + +### Phase 4 Validation + +- [ ] All unit tests pass with >80% coverage on oplog files +- [ ] Integration tests pass +- [ ] E2E tests pass +- [ ] Benchmark: Startup <2s with 10k ops +- [ ] Benchmark: Sync <3s with 100 new ops + +### Phase 5 Validation + +- [ ] Feature flag toggles oplog sync correctly +- [ ] Hybrid mode: both syncs complete without error +- [ ] Migration: Legacy data becomes genesis op +- [ ] Rollback: Disabling flag doesn't break sync + +--- + +## 6. Risk Register + +| Risk | Likelihood | Impact | Mitigation | +| ----------------------------------------------------- | ---------- | -------- | ----------------------------------------------------- | +| Compaction deletes ops still needed by offline device | Medium | High | Longer retention window; snapshot contains full state | +| Concurrent manifest updates cause data loss | Low | High | Atomic upload; manifest is append-only list | +| Replay fires side effects | High | Medium | Phase 1 replay guard implementation | +| Large op log slows startup | Medium | Medium | Compaction; snapshot hydration | +| Field-level merge produces invalid state | Low | High | Typia validation after merge; manual override option | +| User confusion with conflict UI | Medium | Medium | Clear UX; auto-merge for simple cases | +| Migration breaks existing data | Low | Critical | Backup before migration; rollback path | + +--- + +## 7. File Reference + +### Core Implementation + +``` +src/app/core/persistence/operation-log/ +- action-whitelist.ts # Action → Entity mapping +- conflict-resolution.service.ts # Conflict UI and resolution +- dependency-resolver.service.ts # Dep extraction and checking +- lock.service.ts # Cross-tab locking +- multi-tab-coordinator.service.ts # BroadcastChannel sync +- operation-applier.service.ts # Apply ops to store +- operation-converter.util.ts # Op → Action conversion +- operation-log-compaction.service.ts +- operation-log-hydrator.service.ts +- operation-log-migration.service.ts +- operation-log-store.service.ts # IndexedDB persistence +- operation-log-sync.service.ts # Remote sync +- operation-log-effects.ts # NgRx effect capture +- operation.types.ts # Type definitions +- persistent-action.interface.ts # Action metadata +``` + +### UI Components + +``` +src/app/imex/sync/dialog-conflict-resolution/ +- dialog-conflict-resolution.component.ts +- dialog-conflict-resolution.component.html +- dialog-conflict-resolution.component.scss +``` + +### Integration Points + +``` +src/app/pfapi/api/sync/sync.service.ts # Calls oplog sync +src/app/pfapi/api/pfapi.ts # Injects oplog service +``` + +--- + +## 8. Appendix: Vector Clock Primer + +For reference, the system uses vector clocks for conflict detection: + +```typescript +// Vector clock comparison results: +enum VectorClockComparison { + EQUAL, // Same state - no conflict + HAPPENED_BEFORE, // Local is ancestor of remote - apply remote + HAPPENED_AFTER, // Remote is ancestor of local - remote is stale + CONCURRENT, // Neither is ancestor - TRUE CONFLICT +} + +// Example: +// Local: { 'deviceA': 3, 'deviceB': 2 } +// Remote: { 'deviceA': 2, 'deviceB': 3 } +// Result: CONCURRENT (each has changes the other doesn't) +``` + +The operation log stores the vector clock state _after_ each operation, enabling precise conflict detection at the entity level. diff --git a/docs/ai/operation-log-sync.md b/docs/ai/sync/operation-log-sync.md similarity index 98% rename from docs/ai/operation-log-sync.md rename to docs/ai/sync/operation-log-sync.md index 8ed2bfcb1..f2f0ec591 100644 --- a/docs/ai/operation-log-sync.md +++ b/docs/ai/sync/operation-log-sync.md @@ -21,7 +21,7 @@ ### 1.3. Key Benefits Over Current Implementation -The current sync system (see [`sync.service.ts`](../../src/app/pfapi/api/sync/sync.service.ts)) already uses vector clocks for conflict detection at the whole-data level. This proposal extends that to per-operation granularity: +The current sync system (see [`sync.service.ts`](src/app/pfapi/api/sync/sync.service.ts)) already uses vector clocks for conflict detection at the whole-data level. This proposal extends that to per-operation granularity: ``` Current: [Device A State] vs [Device B State] → Conflict if concurrent @@ -34,7 +34,7 @@ Proposed: [Op1, Op2, Op3] vs [Op4, Op5] → Merge non-conflicting, flag c ### 2.1. The Vector Clock (Existing) -Already implemented in [`vector-clock.ts`](../../src/app/pfapi/api/util/vector-clock.ts). No changes needed. +Already implemented in [`vector-clock.ts`](src/app/pfapi/api/util/vector-clock.ts). No changes needed. ```typescript // Map of Client ID -> Counter @@ -942,12 +942,12 @@ Phase 4: Remove snapshot sync (operation log is primary) ### 6.2. pfapi Integration Points -| Existing Component | Integration Approach | -| ------------------------------------------------------------------------ | ----------------------------- | -| [`MetaModelCtrl`](../../src/app/pfapi/api/model-ctrl/meta-model-ctrl.ts) | Add op log sequence to meta | -| [`ModelSyncService`](../../src/app/pfapi/api/sync/model-sync.service.ts) | Keep for snapshot backup | -| [`SyncService.sync()`](../../src/app/pfapi/api/sync/sync.service.ts:91) | Add operation exchange phase | -| Vector Clock utilities | Reuse existing implementation | +| Existing Component | Integration Approach | +| ----------------------------------------------------------------------- | ----------------------------- | +| [`MetaModelCtrl`](src/app/pfapi/api/model-ctrl/meta-model-ctrl.ts) | Add op log sequence to meta | +| [`ModelSyncService`](src/app/pfapi/api/sync/model-sync.service.ts) | Keep for snapshot backup | +| [`SyncService.sync()`](../../src/app/pfapi/api/sync/sync.service.ts:91) | Add operation exchange phase | +| Vector Clock utilities | Reuse existing implementation | ### 6.3. Remote Storage Format @@ -1291,5 +1291,5 @@ describe('Operation Log E2E', () => { - [CRDT Primer](https://crdt.tech/) - [Vector Clocks Explained](https://en.wikipedia.org/wiki/Vector_clock) - [Web Locks API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API) -- [Existing Vector Clock Implementation](../../src/app/pfapi/api/util/vector-clock.ts) -- [Current Sync Service](../../src/app/pfapi/api/sync/sync.service.ts) +- [Existing Vector Clock Implementation](src/app/pfapi/api/util/vector-clock.ts) +- [Current Sync Service](src/app/pfapi/api/sync/sync.service.ts) diff --git a/docs/ai/operationlog-critique.md b/docs/ai/sync/operationlog-critique.md similarity index 100% rename from docs/ai/operationlog-critique.md rename to docs/ai/sync/operationlog-critique.md diff --git a/docs/ai/webdav-non-etag-implementation-plan.md b/docs/ai/webdav-non-etag-implementation-plan.md deleted file mode 100644 index 30a42dded..000000000 --- a/docs/ai/webdav-non-etag-implementation-plan.md +++ /dev/null @@ -1,220 +0,0 @@ -# WebDAV Implementation Plan: Support for Non-ETag Servers - -! Commit after each step ! - -## 🎯 Objective - -Extend the current WebDAV implementation to support servers that don't provide ETags, while maintaining full backward compatibility and ensuring all existing unit tests continue to pass. - -## 📋 Implementation Strategy - -### Phase 1: Analysis & Foundation (Safe Changes) - -**Goal**: Understand current implementation and prepare for changes without breaking existing functionality. - -#### 1.1 Analyze Current Implementation - -- **File**: `src/app/pfapi/api/sync/providers/webdav/webdav-api.ts` -- **Action**: Document current ETag usage patterns -- **Test Requirement**: ✅ All existing tests must pass -- **Verification**: `npm run test:file src/app/pfapi/api/sync/providers/webdav/webdav-api.spec.ts` - -#### 1.2 Review Test Coverage - -- **Files**: All `webdav-*.spec.ts` files -- **Action**: Identify ETag-dependent tests and ensure they remain valid -- **Test Requirement**: ✅ Document which tests validate ETag behavior - -#### 1.3 Create Fallback Strategy Design - -- **Action**: Design capability detection and fallback mechanisms -- **Test Requirement**: ✅ No code changes yet - design only -- when server does not support etags (`NoRevAPIError`), we should check support (see 2.2) and switch to fallback mode for all other requests in this session and retry current request with the other approach once - -### Phase 2: Server Capability Detection (Incremental) - -**Goal**: Add ability to detect server capabilities without breaking existing ETag functionality. - -#### 2.1 Add Server Capability Detection Interface - -```typescript -interface WebdavServerCapabilities { - supportsETags: boolean; - supportsIfHeader: boolean; - supportsLocking: boolean; - supportsLastModified: boolean; -} -``` - -- **Test Requirement**: ✅ Add new tests for capability detection -- **Verification**: Existing tests must continue to pass - -#### 2.2 Implement ETag Detection Method - -```typescript -private async _detectETagSupport(path: string): Promise -``` - -- **Action**: Test server response for ETag headers -- **Test Requirement**: ✅ Mock servers with/without ETag support -- **Verification**: `npm run test:file ` - -### Phase 3: Last-Modified Support (Parallel Implementation) - -**Goal**: Add Last-Modified support alongside existing ETag functionality. - -#### 3.1 Extend Response Header Processing - -```typescript -private _extractValidators(headers: Record): { - etag?: string; - lastModified?: string; - validator: string; // The chosen validator - validatorType: 'etag' | 'last-modified'; -} -``` - -- **Test Requirement**: ✅ Test both ETag and Last-Modified extraction -- **Verification**: Existing ETag tests must still pass - -#### 3.2 Add Last-Modified Conditional Headers - -```typescript -private _createConditionalHeaders( - isOverwrite?: boolean, - expectedEtag?: string | null, - expectedLastModified?: string | null, - preferLastModified?: boolean -): Record -``` - -- **Test Requirement**: ✅ Test all conditional header combinations -- **Verification**: Existing conditional header tests must pass - -#### 3.3 Update Core Methods with Fallback Logic - -- **Methods**: `upload()`, `download()`, `getFileMeta()`, `remove()` -- **Strategy**: Try ETag first, fallback to Last-Modified if needed -- **Test Requirement**: ✅ Test both code paths for each method - -### Phase 4: Alternative Creation Methods (Advanced) - -**Goal**: Implement safe resource creation for non-ETag servers. - -#### 4.1 WebDAV If Header Support - -```typescript -private _createWebDAVIfHeader(condition: 'not-exists' | 'match', value?: string): string -``` - -- **Test Requirement**: ✅ Mock WebDAV If header responses -- **Verification**: Existing creation tests must pass - -#### 4.2 HEAD-then-PUT Fallback - -```typescript -private async _createWithExistenceCheck(path: string, data: string): Promise -``` - -- **Test Requirement**: ✅ Test race condition handling -- **Verification**: Document race condition limitations - -### Phase 5: Testing & Validation (Critical) - -**Goal**: Comprehensive test coverage without breaking existing functionality. - -#### 5.1 Mock Server Scenarios - -- **ETag-only servers** (existing behavior) -- **Last-Modified-only servers** (new behavior) -- **No conditional header support** (fallback behavior) -- **Mixed capability servers** (detection behavior) - -#### 5.2 Integration Tests - -```typescript -describe('WebDAV Non-ETag Server Support', () => { - describe('Last-Modified fallback', () => { - it('should use Last-Modified when ETag unavailable'); - it('should prefer ETag when both available'); - it('should handle missing timestamps gracefully'); - }); - - describe('Safe resource creation', () => { - it('should use If-None-Match when ETag available'); - it('should fallback to WebDAV If header'); - it('should fallback to LOCK/UNLOCK mechanism'); - it('should fallback to HEAD-then-PUT with warnings'); - }); -}); -``` - -## 🧪 Test Strategy - -### Continuous Verification Protocol - -1. **Before Any Changes**: - - ```bash - npm run test:file src/app/pfapi/api/sync/providers/webdav/webdav-api.spec.ts - npm run test:file src/app/pfapi/api/sync/providers/webdav/webdav.spec.ts - ``` - -2. **After Each Phase**: - - ```bash - npm run test # Full test suite - npm run checkFile src/app/pfapi/api/sync/providers/webdav/webdav-api.ts - npm run lint - ``` - -3. **Integration Testing**: - ```bash - npm run e2e # WebDAV e2e tests - ``` - -### Test Data Requirements - -#### Mock Responses for Different Server Types - -```typescript -// ETag-capable server (existing) -const etagResponse = { - headers: { etag: '"abc123"' }, - status: 200, -}; - -// Last-Modified-only server (new) -const lastModifiedResponse = { - headers: { 'last-modified': 'Wed, 21 Oct 2015 07:28:00 GMT' }, - status: 200, -}; - -// Minimal server (fallback) -const minimalResponse = { - headers: {}, - status: 200, -}; -``` - -### Performance Considerations - -1. **Capability detection caching**: Avoid repeated server capability checks -2. **Minimal overhead for ETag servers**: No performance regression for existing servers -3. **Graceful degradation**: Fallback methods clearly documented as slower/less reliable - -## 🎯 Success Criteria - -1. ✅ **Zero test regressions**: All existing WebDAV tests continue to pass -2. ✅ **ETag server performance**: No performance degradation for ETag-capable servers -3. ✅ **Last-Modified support**: Functional sync with Last-Modified-only servers -4. ✅ **Graceful degradation**: Clear fallback behavior for limited servers -5. ✅ **Documentation**: Comprehensive server compatibility matrix -6. ✅ **Configuration**: Optional server capability overrides for testing - -## 📝 Implementation Notes - -- **Conservative approach**: Preserve all existing behavior as default -- **Feature flags**: Allow testing of new functionality without affecting production -- **Comprehensive logging**: Clear indicators of which compatibility mode is active -- **Error context**: Specific error messages for different server limitation scenarios diff --git a/docs/sync/SYNC-PLAN.md b/docs/sync/SYNC-PLAN.md deleted file mode 100644 index 4484f3413..000000000 --- a/docs/sync/SYNC-PLAN.md +++ /dev/null @@ -1,58 +0,0 @@ -# SuperSync - Custom Sync Server Plan - -## Overview - -SuperSync is a custom sync server solution built on top of WebDAV, designed to provide enhanced synchronization capabilities for Super Productivity. While using WebDAV as the underlying protocol, SuperSync adds custom features that address limitations of standard WebDAV sync. - -## Feature Phases - -### 1. Super Basic Sync ✓ - -Working WebDAV clone implemented and works both on server and client side with multi-file sync just as WebDAV would. - -### 2. Authentication and Registration ✓ - -- We need to implement user authentication and registration to secure access to the sync server. -- We should move to an auth token rather than basic auth for better security. -- We should provide Google and GitHub and Apple OAuth login options for easier access. - -### 3. Safer Sync with Sync Completion Detection - -Server detects incomplete syncs using the **existing meta file lock mechanism** (no extra requests needed). - -**How it works (already implemented in client):** - -1. Client writes meta file with `META_FILE_LOCK_CONTENT_PREFIX` before uploading -2. Client uploads all data files -3. Client writes final meta file (without prefix) to signal completion - -**Server-side enhancement:** - -- When meta file contains lock prefix → mark sync as "in progress" -- Stage uploaded files in temp location while lock is active -- When final meta file arrives (no prefix, valid JSON) → atomically move staged files to final location -- If lock remains for > timeout (e.g., 5 min) → discard staged files - -**Zero extra requests** - reuses existing client behavior. - -### 4. Automatic Backups - -The server automatically creates backups of complete datasets before they are overwritten or deleted, allowing clients to restore previous versions if needed. - -**Backup slots (3 total):** - -- Last valid sync before current -- Last valid state before conflicting sync -- Last valid state from yesterday -- Last valid from last week - -### 5. Improve UX - -- Login and register experience is improved and simplified -- Better error messages and sync status feedback -- Connection testing before sync - -### 6. Incremental File Updates - -- instead of uploading entire files we try to derive deltas and upload those - _– needs concept –_