mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-01-23 02:36:05 +00:00
docs: Document Hybrid Manifest & Snapshot Architecture
Introduces a new documentation file detailing the Hybrid Manifest and Snapshot architecture designed to improve efficiency for file-based sync providers like WebDAV and Dropbox. This approach reduces API requests and file proliferation by embedding small operations directly into the manifest file and periodically consolidating changes into snapshots. Also updates the architecture diagrams to include a visualization of this new hybrid approach. Additionally, a "Conflict-Aware Migration Strategy" section was added to the existing operation-log-architecture.md during initial codebase investigation to provide further context on managing schema version differences during sync.
This commit is contained in:
parent
fb590c7fd6
commit
386c297ade
2 changed files with 109 additions and 0 deletions
|
|
@ -151,3 +151,81 @@ graph TD
|
|||
|
||||
API <--> ServerDB
|
||||
```
|
||||
|
||||
## 3. Conflict-Aware Migration Strategy
|
||||
|
||||
This mindmap outlines the strategy for handling version conflicts during sync by migrating operations before conflict detection.
|
||||
|
||||
```mermaid
|
||||
mindmap
|
||||
root((Conflict-Aware<br/>Migration))
|
||||
Strategies
|
||||
Operation-Level Migration
|
||||
Transform V1 Op to V2 Op
|
||||
Extend SchemaMigration Interface
|
||||
Inbound Path Receive
|
||||
Intercept Remote Ops
|
||||
Check Op Schema Version
|
||||
Migrate Old Ops
|
||||
Detect Conflicts on Migrated Ops
|
||||
Outbound Path Send
|
||||
Get Unsynced Ops
|
||||
Migrate Pending Ops if Old
|
||||
Ensure Upload matches Current Schema
|
||||
Conflict Resolution
|
||||
Unified Comparison
|
||||
Local Current vs Remote Migrated
|
||||
Prevent False Conflicts
|
||||
```
|
||||
|
||||
## 4. Hybrid Manifest & Snapshot Architecture (WebDAV / Dropbox Fallback)
|
||||
|
||||
This diagram illustrates the efficient "Hybrid Manifest" approach for file-based sync, showing how small operations are buffered in the manifest to reduce request counts, how overflow creates new files, and how snapshotting consolidates history.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
%% Styles
|
||||
classDef local fill:#fff,stroke:#333,stroke-width:2px,color:black;
|
||||
classDef remote fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:black;
|
||||
classDef decision fill:#fff9c4,stroke:#fbc02d,stroke-width:2px,color:black;
|
||||
classDef storage fill:#f9f,stroke:#333,stroke-width:2px,color:black;
|
||||
|
||||
subgraph "Client: Write Path (Hybrid)"
|
||||
StartWrite((Sync Trigger)) --> LoadMan[Download manifest.json]
|
||||
LoadMan --> CheckBuff{Buffer Full?<br/>> 50 Ops}:::decision
|
||||
|
||||
%% Path A: Buffer Open
|
||||
CheckBuff -- No --> AppendBuff[Append Ops to<br/>manifest.embeddedOperations]
|
||||
AppendBuff --> WriteMan[Upload manifest.json]:::remote
|
||||
|
||||
%% Path B: Buffer Full (Overflow)
|
||||
CheckBuff -- Yes --> CreateFile[Flush embeddedOperations<br/>to ops/overflow_TIMESTAMP.json]:::storage
|
||||
CreateFile --> UploadFile[Upload Op File]:::remote
|
||||
UploadFile --> ClearBuff[Clear Buffer &<br/>Add Filename to<br/>manifest.operationFiles]
|
||||
ClearBuff --> AppendBuff
|
||||
end
|
||||
|
||||
subgraph "Client: Read Path"
|
||||
StartRead((Sync Start)) --> DownMan[Download manifest.json]:::remote
|
||||
DownMan --> CheckSnap{Newer<br/>Snapshot?}:::decision
|
||||
|
||||
CheckSnap -- Yes --> DownSnap[Download & Apply<br/>Snapshot File]:::remote
|
||||
CheckSnap -- No --> CheckFiles
|
||||
|
||||
DownSnap --> CheckFiles
|
||||
CheckFiles[Download & Apply<br/>New Op Files]:::remote
|
||||
CheckFiles --> ApplyEmb[Apply<br/>embeddedOperations]
|
||||
ApplyEmb --> Done((Sync Done))
|
||||
end
|
||||
|
||||
subgraph "Client: Snapshotting (Compaction)"
|
||||
Trigger{Trigger?<br/>> 50 Files}:::decision
|
||||
Trigger -- Yes --> GenSnap[Generate Full Snapshot]:::storage
|
||||
GenSnap --> UpSnap[Upload Snapshot File]:::remote
|
||||
UpSnap --> UpManSnap[Update manifest.json:<br/>1. Set lastSnapshot<br/>2. Clear operationFiles]:::remote
|
||||
UpManSnap --> Cleanup[Delete Old Files<br/>(Async)]
|
||||
end
|
||||
|
||||
class LoadMan,WriteMan,CreateFile,UploadFile,DownMan,DownSnap,CheckFiles,UpSnap,UpManSnap remote;
|
||||
class CheckBuff,CheckSnap,Trigger decision;
|
||||
```
|
||||
|
|
|
|||
|
|
@ -727,6 +727,37 @@ No, not yet. It provides the bridge from older versions of the app to the Operat
|
|||
2. All future schema changes should use the **Schema Migration** system (A.7) described above.
|
||||
3. Once the Operation Log is fully established and legacy data is considered obsolete (e.g., after several major versions), the legacy migration code can be removed.
|
||||
|
||||
### A.7.11 Conflict-Aware Migration Strategy
|
||||
|
||||
**Status:** Design Ready (Not Implemented)
|
||||
|
||||
To handle synchronization between clients on different schema versions, the system must ensure that operations are comparable ("apples-to-apples") before conflict detection occurs.
|
||||
|
||||
#### Strategy
|
||||
|
||||
1. **Operation-Level Migration Pipeline**
|
||||
|
||||
- Extend `SchemaMigration` interface to include `migrateOperation?: (op: Operation) => Operation`.
|
||||
- This allows transforming a V1 `UPDATE` (e.g., `{ changes: { oldField: 'val' } }`) into a V2 `UPDATE` (e.g., `{ changes: { newField: 'val' } }`).
|
||||
|
||||
2. **Inbound Migration (Receive Path)**
|
||||
|
||||
- **Location:** `OperationLogSyncService.processRemoteOps`
|
||||
- **Logic:**
|
||||
1. Receive `remoteOps`.
|
||||
2. Check `op.schemaVersion` for each op.
|
||||
3. If `op.schemaVersion < CURRENT_SCHEMA_VERSION`, run `SchemaMigrationService.migrateOperation(op)`.
|
||||
4. Pass _migrated_ ops to `detectConflicts()`.
|
||||
- **Benefit:** Conflict detection works on the _current_ schema structure, preventing false negatives (missing a conflict because field names differ) and confusing diffs.
|
||||
|
||||
3. **Outbound Migration (Send Path)**
|
||||
|
||||
- **Location:** `OperationLogStore.getUnsynced()`
|
||||
- **Logic:** Ensure all pending operations sent to the server match `CURRENT_SCHEMA_VERSION`. If an op was created before a local migration (e.g., pending from last session), migrate it on-the-fly before upload.
|
||||
|
||||
4. **Conflict Resolution**
|
||||
- The `ConflictResolutionService` will display the _migrated_ remote operation against the current local state, ensuring the user sees a consistent view of the data (e.g., "Time Estimate" on both sides, rather than "Estimate" vs "Time Estimate").
|
||||
|
||||
---
|
||||
|
||||
# Part B: Legacy Sync Bridge
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue