docs: Document Hybrid Manifest & Snapshot Architecture

Introduces a new documentation file detailing the Hybrid Manifest and Snapshot architecture
designed to improve efficiency for file-based sync providers like WebDAV and Dropbox.
This approach reduces API requests and file proliferation by embedding small operations
directly into the manifest file and periodically consolidating changes into snapshots.

Also updates the architecture diagrams to include a visualization of this new hybrid approach.

Additionally, a "Conflict-Aware Migration Strategy" section was added to the existing
operation-log-architecture.md during initial codebase investigation to provide further
context on managing schema version differences during sync.
This commit is contained in:
Johannes Millan 2025-12-03 18:24:30 +01:00
parent fb590c7fd6
commit 386c297ade
2 changed files with 109 additions and 0 deletions

View file

@ -151,3 +151,81 @@ graph TD
API <--> ServerDB
```
## 3. Conflict-Aware Migration Strategy
This mindmap outlines the strategy for handling version conflicts during sync by migrating operations before conflict detection.
```mermaid
mindmap
root((Conflict-Aware<br/>Migration))
Strategies
Operation-Level Migration
Transform V1 Op to V2 Op
Extend SchemaMigration Interface
Inbound Path Receive
Intercept Remote Ops
Check Op Schema Version
Migrate Old Ops
Detect Conflicts on Migrated Ops
Outbound Path Send
Get Unsynced Ops
Migrate Pending Ops if Old
Ensure Upload matches Current Schema
Conflict Resolution
Unified Comparison
Local Current vs Remote Migrated
Prevent False Conflicts
```
## 4. Hybrid Manifest & Snapshot Architecture (WebDAV / Dropbox Fallback)
This diagram illustrates the efficient "Hybrid Manifest" approach for file-based sync, showing how small operations are buffered in the manifest to reduce request counts, how overflow creates new files, and how snapshotting consolidates history.
```mermaid
graph TD
%% Styles
classDef local fill:#fff,stroke:#333,stroke-width:2px,color:black;
classDef remote fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:black;
classDef decision fill:#fff9c4,stroke:#fbc02d,stroke-width:2px,color:black;
classDef storage fill:#f9f,stroke:#333,stroke-width:2px,color:black;
subgraph "Client: Write Path (Hybrid)"
StartWrite((Sync Trigger)) --> LoadMan[Download manifest.json]
LoadMan --> CheckBuff{Buffer Full?<br/>> 50 Ops}:::decision
%% Path A: Buffer Open
CheckBuff -- No --> AppendBuff[Append Ops to<br/>manifest.embeddedOperations]
AppendBuff --> WriteMan[Upload manifest.json]:::remote
%% Path B: Buffer Full (Overflow)
CheckBuff -- Yes --> CreateFile[Flush embeddedOperations<br/>to ops/overflow_TIMESTAMP.json]:::storage
CreateFile --> UploadFile[Upload Op File]:::remote
UploadFile --> ClearBuff[Clear Buffer &<br/>Add Filename to<br/>manifest.operationFiles]
ClearBuff --> AppendBuff
end
subgraph "Client: Read Path"
StartRead((Sync Start)) --> DownMan[Download manifest.json]:::remote
DownMan --> CheckSnap{Newer<br/>Snapshot?}:::decision
CheckSnap -- Yes --> DownSnap[Download & Apply<br/>Snapshot File]:::remote
CheckSnap -- No --> CheckFiles
DownSnap --> CheckFiles
CheckFiles[Download & Apply<br/>New Op Files]:::remote
CheckFiles --> ApplyEmb[Apply<br/>embeddedOperations]
ApplyEmb --> Done((Sync Done))
end
subgraph "Client: Snapshotting (Compaction)"
Trigger{Trigger?<br/>> 50 Files}:::decision
Trigger -- Yes --> GenSnap[Generate Full Snapshot]:::storage
GenSnap --> UpSnap[Upload Snapshot File]:::remote
UpSnap --> UpManSnap[Update manifest.json:<br/>1. Set lastSnapshot<br/>2. Clear operationFiles]:::remote
UpManSnap --> Cleanup[Delete Old Files<br/>(Async)]
end
class LoadMan,WriteMan,CreateFile,UploadFile,DownMan,DownSnap,CheckFiles,UpSnap,UpManSnap remote;
class CheckBuff,CheckSnap,Trigger decision;
```

View file

@ -727,6 +727,37 @@ No, not yet. It provides the bridge from older versions of the app to the Operat
2. All future schema changes should use the **Schema Migration** system (A.7) described above.
3. Once the Operation Log is fully established and legacy data is considered obsolete (e.g., after several major versions), the legacy migration code can be removed.
### A.7.11 Conflict-Aware Migration Strategy
**Status:** Design Ready (Not Implemented)
To handle synchronization between clients on different schema versions, the system must ensure that operations are comparable ("apples-to-apples") before conflict detection occurs.
#### Strategy
1. **Operation-Level Migration Pipeline**
- Extend `SchemaMigration` interface to include `migrateOperation?: (op: Operation) => Operation`.
- This allows transforming a V1 `UPDATE` (e.g., `{ changes: { oldField: 'val' } }`) into a V2 `UPDATE` (e.g., `{ changes: { newField: 'val' } }`).
2. **Inbound Migration (Receive Path)**
- **Location:** `OperationLogSyncService.processRemoteOps`
- **Logic:**
1. Receive `remoteOps`.
2. Check `op.schemaVersion` for each op.
3. If `op.schemaVersion < CURRENT_SCHEMA_VERSION`, run `SchemaMigrationService.migrateOperation(op)`.
4. Pass _migrated_ ops to `detectConflicts()`.
- **Benefit:** Conflict detection works on the _current_ schema structure, preventing false negatives (missing a conflict because field names differ) and confusing diffs.
3. **Outbound Migration (Send Path)**
- **Location:** `OperationLogStore.getUnsynced()`
- **Logic:** Ensure all pending operations sent to the server match `CURRENT_SCHEMA_VERSION`. If an op was created before a local migration (e.g., pending from last session), migrate it on-the-fly before upload.
4. **Conflict Resolution**
- The `ConflictResolutionService` will display the _migrated_ remote operation against the current local state, ensuring the user sees a consistent view of the data (e.g., "Time Estimate" on both sides, rather than "Estimate" vs "Time Estimate").
---
# Part B: Legacy Sync Bridge