mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-07-17 16:37:43 +00:00
087b9dd43f
* refactor(sync): tighten extracted package surfaces
Combined polish from the post-extraction review:
- sync-core: strip NgRx-shaped types from EntityConfig/EntityRegistry;
expose host extensions via generic param. Move StateSelector,
PropsStateSelector, SelectByIdFactory, SelectById, EntityUpdateLike,
EntityAdapterLike to a new app-side entity-registry-host.types.ts.
- sync-core: mark OpType.SyncImport/BackupImport/Repair as @deprecated;
hosts should use createFullStateOpTypeHelpers().
- sync-providers: resolve provider.types.ts vs provider-types.ts
duplication; inline implementation into the dashed canonical name.
- sync-providers: drop unused root barrel and "." export; consumers
already use focused subpath barrels (/dropbox, /webdav, etc.).
- sync-providers: replace wildcard "@sp/sync-providers/*" tsconfig path
alias with 11 explicit subpath entries matching package.json exports;
deep-internal imports now fail at typecheck.
- sync-providers: move @sp/sync-core from dependencies to
peerDependencies (kept in devDependencies for tests).
- both packages: add composite: true to enable project references;
introduce tsconfig.build.json overlay so tsup DTS bundler still works.
- gitignore: ignore **/*.tsbuildinfo composite outputs.
* refactor(sync-core): prune 47 unused barrel exports
Removes exports with zero consumers outside the package. Source files
are unchanged; only the public barrel is trimmed. Covers compression
helper classes, sync-file-prefix error/config types, replay coordinator
internals, remote-apply result types, upload/download planning option
and plan types, ports misc, conflict-resolution helper types, and
sync-import-filter decision types.
* refactor(sync-core): drop unused encryption migration path
decryptWithMigration and DecryptResult had no host consumer; they
exposed a structural-migration entry point ("here is your ciphertext
re-encrypted under Argon2id") that nothing in the codebase reads. The
side-channel setLegacyKdfWarningHandler — which IS used — stays.
encryptWithDerivedKey/decryptWithDerivedKey lose their export keyword
and remain as module-internal helpers; encrypt/decrypt/encryptBatch/
decryptBatch still call them. Wire format and legacy-fallback semantics
are unchanged, so existing ciphertext continues to decrypt.
Test imports for compression and sync-file-prefix specs now go via
their source files instead of the trimmed barrel.
* fix(sync-providers): bound dropbox token refresh to single retry; share md5 rev helper
The five hand-rolled token-refresh blocks in Dropbox.{getFileRev,
downloadFile, uploadFile, removeFile, listFiles} recursed on themselves
after refresh. If the post-refresh call still saw a token error (real
case: the refresh token itself was revoked), the recursion would not
terminate. Consolidated into a single _withTokenRefresh helper that
attempts the call, refreshes once on a token error, retries once, then
lets the outer 401 classifier surface AuthFailSPError.
Same log message, same _isTokenError discriminator, same refresh call.
Same five sites still apply their post-call non-token error mapping
(NoRev, InvalidData, RemoteFileNotFound, path-not-found swallow, etc.).
Also extracts md5 content-rev computation duplicated between
LocalFileSyncBase._getLocalRev and WebdavApi._computeContentHash into a
shared file-based/content-rev.ts; both call sites preserve their own
error wrapping at the boundary.
* refactor(sync): split oversized super-sync and conflict-resolution
sync-providers: extract request-ID hashing from super-sync.ts (1017 ->
918 lines) into a new request-id.ts. The helpers were free functions
already in disguise (none referenced this), so the move is mechanical.
HTTP plumbing (_doWebFetch/_doNativeFetch/_fetchApi*) stays as private
methods — it transitively touches 12 instance members and would need
either a wide context object or a separate http-client collaborator
class to extract cleanly. Left as a follow-up.
sync-core: split conflict-resolution.ts into three cohesive files:
- entity-frontier.ts now owns buildEntityFrontier and
adjustForClockCorruption (per-entity vector-clock domain).
- extractEntityFromPayload and extractUpdateChanges move to
operation.types.ts next to the existing extractActionPayload.
- conflict-resolution.ts keeps deep-equality, LWW planning,
partitioning, and identical-conflict detection.
Public barrel exports unchanged; tests now import the moved symbols
from their new homes.
* refactor(sync-core): drop redundant OperationStorePort
OperationStorePort overlapped with RemoteOperationApplyStorePort on the
two state-transition methods (markSynced/markApplied,
markRejected/markFailed) and had zero non-structural consumers — the
only implementer was OperationLogStoreService, which already exposes
the three methods as its own public surface. Removing the port leaves
the service contract intact and removes the verb-pair confusion noted
in the post-extraction review.
Spec contract test still drives the same state transitions; only the
local typing of the test fixture changes from the deleted interface to
Pick<OperationLogStoreService, ...>.
* refactor(sync-providers): decouple SuperSync provider from SP-specific host
Two coupling leaks the package shouldn't carry:
1. SUPER_SYNC_DEFAULT_BASE_URL was an implicit fallback inside
SuperSyncProvider — an SP-specific URL baked into a "framework-
agnostic" package. Make defaultBaseUrl a required SuperSyncDeps
field; the host factory supplies the SP default. The constant stays
exported as a suggested default for hosts targeting the SP-hosted
server.
2. Consumers that wanted the WebSocket path had to do
`provider as unknown as SuperSyncProvider` to call
getWebSocketParams. Introduce SuperSyncWebSocketAccess interface +
isSuperSyncWebSocketAccess structural guard; SuperSyncProvider
implements it. sync-wrapper.service drops its cast in favor of the
guard.
super-sync-restore.service still casts to SuperSyncProvider for the
restore path — same pattern would solve it, but out of scope here.
* test(sync-providers): extract shared test helpers and prefer barrels
Adds tests/helpers/sync-logger.ts and tests/helpers/credential-store.ts
to centralize the noopLogger and CredentialStore mocks that were copy-
pasted across 8 spec files. createStatefulCredentialStore covers the
"load/upsert/clear with state" cases; createMockCredentialStore covers
bare vi.fn() ports. Spec sites that needed a unique mockResolvedValue
chain it after the helper, preserving behavior 1:1.
Also migrates 5 spec files from deep ../src/<file> paths to the
matching sub-barrel (../src/webdav, /http, /super-sync, /platform) for
symbols already exported there. No new barrel exports added — internal
types (WebDavHttpAdapter, WebdavApi, DropboxApi, etc.) stay on deep
paths because they are intentionally not part of the public surface.
super-sync.spec.ts keeps its own credential/logger mocks (special
__asPort wrapper and vi.spyOn against the live NOOP_SYNC_LOGGER) that
the generic helpers cannot reproduce without bloat.
* test(sync): pin vector-clock pruning, error-meta privacy, and sync-import edges
Fills three test gaps surfaced by the post-extraction review:
- vector-clock pruning correctness across clocks: 4 cases pinning that
pruning legitimately flips GREATER_THAN to CONCURRENT/LESS_THAN when
the dropped keys are still present in the comparison clock. This is
the documented behavior (compareVectorClocks is intentionally not
pruning-aware); the protocol handles flips server-side via the
rejected-ops retry loop. preserveClientIds case also covered.
- error-meta privacy boundary: 22 new cases covering urlPathOnly (strip
query/fragment/userinfo, preserve host+path+port, leave non-URLs
intact) and errorMeta (no leakage of headers, response bodies, OAuth
tokens, signed-URL params, user emails, or attached error fields).
Real negative assertions (.not.toContain), not shape checks.
- sync-import-filter edge cases: 8 cases covering empty clocks on
either side, op clock listing the import client at 0, same-client
with equal counter (pinning the strict-greater-than boundary), and
different-client knowledge above the import counter.
sync-core 195 -> 207 tests, sync-providers 319 -> 341 tests; no
production code changed.
* style(sync-core): format sync-file-prefix.spec import line
* fix(sync): address package review feedback
|
||
|---|---|---|
| .. | ||
| background-info | ||
| diagrams | ||
| long-term-plans | ||
| file-based-sync-flowchart.md | ||
| operation-log-architecture-diagrams.md | ||
| operation-log-architecture.md | ||
| operation-payload-optimization-discussion.md | ||
| operation-rules.md | ||
| package-boundaries.md | ||
| quick-reference.md | ||
| README.md | ||
| supersync-encryption-architecture.md | ||
| supersync-scenarios-flowchart.md | ||
| supersync-scenarios.md | ||
| vector-clocks.md | ||
Operation Log Documentation
Last Updated: January 2026
This directory contains the architectural documentation for Super Productivity's Operation Log system - an event-sourced persistence and synchronization layer that handles ALL sync providers (SuperSync, WebDAV, Dropbox, LocalFile).
Quick Start
| If you want to... | Read this |
|---|---|
| Understand the overall architecture | operation-log-architecture.md |
| See visual diagrams | diagrams/ (split by topic) |
| Learn the design rules | operation-rules.md |
| Understand package boundaries | package-boundaries.md |
| Understand file-based sync | diagrams/04-file-based-sync.md |
| Understand SuperSync encryption | supersync-encryption-architecture.md |
Documentation Overview
Core Documentation
| Document | Description | Status |
|---|---|---|
| operation-log-architecture.md | Comprehensive architecture reference covering Parts A-F: Local Persistence, File-Based Sync, Server Sync, Validation & Repair, Smart Archive Handling, and Atomic State Consistency | Active |
| diagrams/ | Mermaid diagrams split by topic (local persistence, server sync, file-based sync, etc.) | Active |
| operation-rules.md | Design rules and guidelines for the operation log store and operations | Active |
| package-boundaries.md | Dependency direction and ownership boundaries for @sp/sync-core, @sp/sync-providers, and app sync wiring |
Active |
Sync Architecture
| Document | Description | Status |
|---|---|---|
| diagrams/04-file-based-sync.md | File-based sync with single sync-data.json (WebDAV/Dropbox/LocalFile) | Implemented |
| diagrams/02-server-sync.md | SuperSync server sync architecture | Implemented |
| supersync-encryption-architecture.md | End-to-end encryption for SuperSync (AES-256-GCM + Argon2id) | Implemented |
Historical / Completed Plans
| Document | Description | Status |
|---|---|---|
| replace-pfapi-with-oplog-plan.md | Plan to unify sync by replacing PFAPI with operation log | Completed (Jan 2026) |
| e2e-encryption-plan.md | Original E2EE design (see supersync-encryption for impl) | Implemented (Dec 2025) |
Architecture at a Glance
The Operation Log system is the single sync system for all providers:
User Action
│
▼
NgRx Store
(Runtime Source of Truth)
│
┌───────────────────┼───────────────────┐
▼ │ ▼
OpLogEffects │ Other Effects
│ │
├──► SUP_OPS ◄──────┘
│ (Local Persistence - IndexedDB)
│
└──► Sync Providers
├── SuperSync (operation-based, real-time)
├── WebDAV (file-based, single-file snapshot)
├── Dropbox (file-based, single-file snapshot)
└── LocalFile (file-based, single-file snapshot)
Sync Provider Types
| Provider Type | Providers | How It Works |
|---|---|---|
| Server-based | SuperSync | Individual operations uploaded/downloaded via HTTP API |
| File-based | WebDAV, Dropbox, LocalFile | Single sync-data.json file with state snapshot + recent ops |
The Core Parts
| Part | Purpose | Description |
|---|---|---|
| A. Local Persistence | Fast writes, crash recovery | Operations stored in IndexedDB (SUP_OPS), with snapshots for fast hydration |
| B. File-Based Sync | WebDAV/Dropbox/LocalFile | Single-file sync with state snapshot and embedded operations buffer |
| C. Server Sync | Operation-based sync | Upload/download individual operations via SuperSync server |
| D. Validation & Repair | Data integrity | Checkpoint validation with automatic repair and REPAIR operations |
Additional architectural patterns:
| Pattern | Purpose |
|---|---|
| E. Smart Archive Handling | Deterministic archive operations synced via instructions, not data |
| F. Atomic State Consistency | Meta-reducers ensure multi-entity changes are atomic |
Key Concepts
Event Sourcing
The Operation Log treats the database as a timeline of events rather than mutable state:
- Source of Truth: The log is truth; current state is derived by replaying the log
- Immutability: Operations are never modified, only appended
- Snapshots: Periodic snapshots speed up hydration (replay from snapshot + tail ops)
Vector Clocks
Vector clocks track causality for conflict detection:
- Each client has its own counter in the vector clock
- Comparison reveals:
EQUAL,LESS_THAN,GREATER_THAN, orCONCURRENT CONCURRENTindicates a true conflict requiring resolution
LOCAL_ACTIONS Token
Effects that perform side effects (snacks, external APIs, UI) must use LOCAL_ACTIONS instead of Actions:
private _actions$ = inject(LOCAL_ACTIONS); // Excludes remote operations
This prevents duplicate side effects when syncing operations from other clients.
Key Files
Sync Packages
packages/sync-core/src/
├── operation.types.ts # Generic operation primitives
├── vector-clock.ts # Compare/merge/prune algorithms
├── conflict-resolution.ts # Pure conflict helpers
├── replay-coordinator.ts # Generic remote replay ordering
└── ports.ts # App-side orchestration contracts
packages/sync-providers/src/
├── super-sync/ # SuperSync provider implementation
├── file-based/dropbox/ # Dropbox provider implementation
├── file-based/webdav/ # WebDAV + Nextcloud providers
├── file-based/local-file/ # LocalFile provider classes
└── provider-types.ts # Provider-neutral contracts
App Sync Wiring
src/app/op-log/sync-providers/
├── sync-providers.factory.ts # Composes app deps into package providers
├── credential-store.service.ts # OAuth/credential storage implementation
├── wrapped-provider.service.ts # Provider wrapper with encryption
├── file-based/ # File-based adapter + app shims
└── super-sync/ # SuperSync app shims + validators
Core Operation Log
src/app/op-log/
├── core/ # Core types and operations
├── persistence/ # IndexedDB storage
├── sync/ # Sync orchestration
└── validation/ # Data validation and repair
Related Documentation
| Location | Content |
|---|---|
| vector-clocks.md | Vector clock implementation details |
| packages/super-sync-server/ | SuperSync server implementation |
| background-info/ | Research and best practices documents |
Implementation Status
| Component | Status |
|---|---|
| Local Persistence (Part A) | Complete |
| File-Based Sync (Part B) | Complete (WebDAV, Dropbox, LocalFile) |
| Server Sync (Part C) | Complete (SuperSync) |
| Validation & Repair (Part D) | Complete |
| End-to-End Encryption | Complete (AES-256-GCM + Argon2id) |
| PFAPI Elimination | Complete (Jan 2026) |
| Cross-version Sync (A.7.11) | Documented (not yet implemented) |
| Schema Migrations | Infrastructure ready (no migrations defined yet) |
See operation-log-architecture.md#implementation-status for detailed status.