super-productivity/docs/long-term-plans/multi-client-file-sync-reliability.md

Making File-Based Sync Reliable with Multiple Concurrent Clients

Status: Planned

Current Vulnerabilities

The single-file approach (sync-data.json) has these specific weaknesses when multiple clients sync simultaneously:

1. Bounded retries on upload conflict

_uploadWithMismatchFallback() (file-based-sync-adapter.service.ts) makes up to 1 + _MAX_UPLOAD_RETRIES conditional attempts (_MAX_UPLOAD_RETRIES is currently 2, so 3 total) and never force-overwrites. On a rev mismatch it re-downloads: if the remote rev actually changed it treats that as a genuine concurrent write and throws a retryable error immediately (the extra attempts exist only for the transient case where the re-downloaded rev is unchanged). The next sync cycle then downloads the concurrent ops and rebuilds a consistent snapshot. This handles a concurrent write that is visible at check time; it does not close the check-then-write race described in §5.

2. Wide race window

The upload cycle is: download → read state snapshot → merge ops → encrypt → compress → upload. This can take seconds (especially with large state + archives). Any other client uploading during that window causes a conflict.

3. Full state on every upload

Every upload includes the complete application state (line 452: getStateSnapshot()), both archives, and 500 recent ops. This makes the file large and the upload slow, widening the race window.

4. WebDAV revision tracking is coarse

WebDAV uses lastmod (seconds resolution) as the revision. Two uploads within the same second can't be distinguished. The syncVersion counter inside the file compensates, but only if the file is actually re-downloaded between attempts.

5. No atomic CAS for LocalFile (accepted limitation — #8898)

For local file sync there is no server-side compare-and-swap. uploadFile() (local-file-sync-base.ts) does the rev check (downloadFile + hash compare) and the writeFile as two separate, non-atomic steps, so a concurrent writer that lands between check and write is not detected and can be overwritten (a classic TOCTOU race). This is an accepted limitation, LOW severity in practice because several layers narrow the window or soften the outcome:

• Within one client, concurrent uploads share an upload lock (LockService, LOCK_NAMES.UPLOAD — Web Locks cross-tab, in-process mutex fallback on Electron/Android), so a client's own upload cycles don't race on the file. This does not extend across machines. (Downloads use a separate lock; only uploads write the file.)
• Cross-machine contention needs multiple writers on the same file — an external folder-sync tool (Syncthing/Dropbox) or a directly shared/network-mounted sync folder. An OS-level lock wouldn't help across machines anyway.
• A concurrent write that is visible at check time is caught, not clobbered: _uploadWithMismatchFallback never force-overwrites; on a rev mismatch it re-downloads and throws retryably, and the next cycle re-applies the concurrent ops. Only a write landing inside the check→write window escapes this.
• Backup-before-overwrite (.bak, #8786, best-effort): the current remote content is copied to a .bak before overwrite, letting the next download recover a corrupt/interrupted primary. It does not recover a valid concurrent overwrite, nor a primary that went fully missing (e.g. an Android delete-then-crash), and the .bak write is non-fatal if it fails.

So the residual risk is narrow but real: a writer whose write falls inside another client's check→write window can have its update lost — recoverable only if that client's local op-log still holds the ops and re-uploads them on a later cycle. Two distinct problems live here; keep them separate:

• Torn writes (crash mid-write → partial/corrupt file) are already prevented on Electron/desktop: FILE_SYNC_SAVE (electron/local-file-sync.ts) writes to a temp file (flag: 'wx') then renameSync (atomic on ext4/APFS/NTFS), with temp cleanup on failure. Android SAF still writes in place (SafBridgePlugin.writeFile → openOutputStream), so a torn write is possible there — only partly mitigated by the best-effort .bak recovery above (and not at all if the primary goes missing rather than corrupt). A native temp-DocumentFile + rename would close it, but it's low value (mobile is effectively single-writer).
• The check-then-write CAS race itself is NOT closed by atomic rename — rename only makes the write atomic, not the read-compare-write sequence. Portably closing it needs OS-level CAS (O_EXCL / advisory locks) that isn't uniformly available across the LocalFile backends. Left as accepted.

How Bad Is It in Practice?

It works reasonably well for 2 clients because:

• The piggybacking mechanism merges concurrent uploads on the retry
• Vector clocks + LWW correctly resolve entity-level conflicts
• The 500-op buffer is generous enough to catch concurrent changes
• Sync intervals (e.g., 5 minutes) usually provide enough separation

It gets fragile with 3+ clients or short sync intervals because the single retry isn't enough, and the large file size makes uploads slow.

---

Three Levels of Improvement

Level 1: Harden the Single-File Approach (Small Change)

What: Fix the most obvious weaknesses without changing the storage model.

Changes to file-based-sync-adapter.service.ts:

1. Retry loop with exponential backoff instead of single retry

   • Replace _uploadWithRetry() with a loop: attempt up to 3-5 times
   • Add randomized backoff (200ms, 400ms, 800ms + jitter) between retries
   • Each retry re-downloads, re-merges, re-uploads
   • ~30 lines changed

2. Lock file before upload (optional, for providers that support it)

   • Write a sync.lock file with client ID + timestamp before uploading
   • Other clients check the lock and skip/wait if it's recent (< 30s)
   • Delete lock after upload
   • Already have precedent: migration.lock in the codebase
   • ~50 lines added

3. WebDAV: use ETag headers instead of lastmod for revision

   • More precise conflict detection
   • Requires checking WebDAV provider implementation

Pros: Minimal code change, backward compatible, no migration needed Cons: Still fundamentally limited — single file remains the bottleneck Reliability improvement: Good enough for 3-4 clients with reasonable sync intervals (2+ minutes)

---

Level 2: Separate Operations from State (Medium Change)

What: Split into two files — a state snapshot (updated infrequently) and an operations log (updated every sync). This reduces contention because most sync cycles only touch the ops file.

Storage structure:

sync-data.json          → state snapshot (updated every Nth sync or on demand)
sync-ops.jsonl          → append-only operation log (updated every sync)
sync-meta.json          → vector clock + syncVersion + metadata

How it works:

• Upload ops: Append new operations to sync-ops.jsonl. This is smaller and faster than rewriting the full state.
• Download ops: Read sync-ops.jsonl, filter to new ops. Fast because it's just the ops, not the full state.
• Snapshot update: Periodically (every 10th sync, or when ops file gets large), rewrite sync-data.json with current state and reset sync-ops.jsonl.
• Conflict: sync-meta.json has the syncVersion counter. Only contested during uploads, and the file is tiny (fast upload → small race window).

The key insight: Most sync cycles don't need to touch the large state file at all. Ops are small. Conflicts on a small file are rare and fast to resolve.

Pros: Significantly less contention, smaller uploads, backward-compatible migration path Cons: Three files to manage instead of one; append-only JSONL needs periodic compaction; providers that don't support append (Dropbox) would need to re-upload the ops file Reliability improvement: Handles 4-5+ concurrent clients well

Files to modify:

• file-based-sync-adapter.service.ts — split upload/download into ops-only and snapshot paths
• file-based-sync.types.ts — add new file type constants, ops file format
• Provider interfaces — possibly add appendFile() method (or just re-upload the ops file for providers that don't support append)

---

Level 3: Per-Client Files (Large Change, Most Robust)

What: Each client writes only to its own files. Other clients only read. Zero write conflicts by design.

Storage structure:

sp-sync/
  clients/
    <client-id-A>/
      manifest.json                 # Batch list + vector clock (unencrypted)
      ops/
        <timestamp>-<seq>.jsonl     # Immutable operation batch files
      snapshot.json                  # This client's state snapshot (encrypted)
      snapshot-archive-young.json
      snapshot-archive-old.json
    <client-id-B>/
      manifest.json
      ops/
        ...

How it works:

• Upload: Write a new batch file to clients/<myId>/ops/, update manifest.json. Never modify another client's files.
• Download: For each known peer, read manifest.json → download new batch files by exact path.
• Bootstrap: New client reads any peer's snapshot.json for initial state, then catches up with batch files.
• GC: Client deletes its own old batch files once all peers' vector clocks show they've advanced past them.

Why it eliminates conflicts:

• No two clients ever write the same file
• Batch files are immutable once written (append-only model)
• manifest.json is the only mutable file per client, and only the owning client writes it
• Works with ANY file storage: WebDAV, Dropbox, LocalFile, and Syncthing/Resilio

Implementation: This would be a new provider (not modifying existing file-based sync), implementing OperationSyncCapable directly. The existing FileBasedSyncAdapterService stays unchanged for users who don't need multi-client reliability.

Pros: Zero contention, scales to any number of clients, works with folder sync tools Cons: More files to manage, needs directory listing support, biggest implementation effort, needs migration path Reliability improvement: Handles unlimited concurrent clients reliably

New files:

• src/app/op-log/sync-providers/file-based/multi-client/multi-client-sync-adapter.service.ts
• src/app/op-log/sync-providers/file-based/multi-client/multi-client-sync.types.ts
• src/app/op-log/sync-providers/file-based/multi-client/multi-client-gc.service.ts

Modified files:

• provider.const.ts — new provider ID (or config flag on existing providers)
• provider-manager.service.ts — register new provider
• global-config.model.ts — config for multi-client mode
• sync-form.const.ts — UI toggle or separate provider option

---

Recommendation

Level 1 (retry + backoff) is a quick win worth doing regardless — it's a small change that makes the current system more robust.

Level 3 (per-client files) is the correct long-term solution if multi-client reliability is a priority. It also naturally enables Syncthing compatibility as a side effect. Level 2 is a half-measure that adds complexity without fully solving the problem.

The question is whether to go 1 → 3 (quick fix now, proper solution later) or straight to 3.

---

Level 3 Coordination Design

Do we need listFiles()?

Yes, but only for peer discovery — and it can be minimized with a manifest approach.

Level 3 needs listFiles() for two things:

1. Discover peers: List clients/ directory to find other client IDs
2. Find batch files: List clients/<peerId>/ops/ to find new operation batches

We can eliminate need #2 entirely with per-client manifest files. Each client updates its own manifest.json with the list of its batch files. Other clients read the manifest by exact path (clients/<peerId>/manifest.json) — no directory listing needed.

This reduces listFiles() to just peer discovery (listing clients/ once to find new peers). Known peers are cached locally.

Coordination flow (minimal listFiles())

First sync / peer discovery (needs listFiles() once):

1. listFiles('clients/') → discover peer directories
2. Store known peer IDs locally (localStorage)
3. Read each peer's manifest.json → get their batch files + vector clock
4. Download batch files by exact path → apply operations
5. If bootstrapping: read any peer's snapshot.json for initial state

Normal sync cycle (no listFiles() needed):

1. Upload: Write new batch file → update own manifest.json
2. Download: For each known peer, read manifest.json → download new batch files
3. Periodic discovery: listFiles('clients/') occasionally (every Nth cycle) to find new peers

Can we avoid listFiles() entirely?

Alternatives considered:

1. User-configured peers: User manually enters device IDs. Works for 2-3 devices but bad UX.
2. Registration file per client: Each client writes register/<myId>.json. Still needs listing register/ to find peers.
3. Shared registry file: One peers.json listing all peers. Creates the shared-mutable-file problem we're trying to avoid.

Verdict: listFiles() is the cleanest solution. The missing implementations are trivial:

• Electron: Add ipcMain.handle(IPC_FILE_SYNC_LIST_FILES, ...) with fs.readdirSync() — ~10 lines
• Android SAF: Call DocumentFile.listFiles() in Capacitor plugin — natural SAF capability

Implementing listFiles() is much simpler than designing a discovery mechanism that avoids it.

Directory creation requirements

Level 3 needs clients/<id>/ops/ directories to exist:

• WebDAV: Auto-creates parent directories via MKCOL on upload (already implemented)
• Dropbox: create_folder_v2 API (already available in the Dropbox API)
• Electron: fs.mkdirSync(path, { recursive: true }) — add to IPC handler
• Android SAF: DocumentFile.createDirectory() — add to Capacitor plugin

Level 3 prerequisites by provider

Prerequisite	WebDAV	Dropbox	Electron	Android
listFiles()	exists	exists	needs IPC handler (~10 lines)	needs implementation
Directory creation	auto (MKCOL)	needs createDir() call	needs mkdirSync() call	needs createDirectory() call
uploadFile() to subdirs	works	works	works	works
downloadFile() from subdirs	works	works	works	works

---

Additional Findings

Resolved: Piggybacking removed (commit 6ec885cce2)

Piggybacking was removed from the file-based sync adapter. Remote ops are now discovered exclusively via downloadOps() on the next sync cycle, eliminating the stale piggyback bug and simplifying the upload path.

Unused checksum field

FileBasedSyncData already has an unused checksum?: string field (line 83 in file-based-sync.types.ts). Could be leveraged for integrity verification in any level of improvement.

Confirmed in the wild

Recent commit 87d884ed17 ("fix(sync): prevent recurring task duplication across clients") confirms multi-client sync issues are a real problem users hit, not just theoretical.

Electron LocalFile also missing listFiles()

The IPC event FILE_SYNC_LIST_FILES is defined in ipc-events.const.ts:46 and exposed in preload.ts:47-48, but there is no ipcMain.handle() implementation in the Electron main process. So listFiles() is missing on both Android SAF and Electron LocalFile.

Directory creation varies by provider

• WebDAV: Auto-creates parent directories via MKCOL on upload (lines 314-345 in webdav-api.ts)
• Dropbox & LocalFile: Do NOT auto-create directories — uploads fail if parent doesn't exist