super-productivity/packages/sync-core
Johannes Millan 087b9dd43f
refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595)
* refactor(sync): tighten extracted package surfaces

Combined polish from the post-extraction review:

- sync-core: strip NgRx-shaped types from EntityConfig/EntityRegistry;
  expose host extensions via generic param. Move StateSelector,
  PropsStateSelector, SelectByIdFactory, SelectById, EntityUpdateLike,
  EntityAdapterLike to a new app-side entity-registry-host.types.ts.
- sync-core: mark OpType.SyncImport/BackupImport/Repair as @deprecated;
  hosts should use createFullStateOpTypeHelpers().
- sync-providers: resolve provider.types.ts vs provider-types.ts
  duplication; inline implementation into the dashed canonical name.
- sync-providers: drop unused root barrel and "." export; consumers
  already use focused subpath barrels (/dropbox, /webdav, etc.).
- sync-providers: replace wildcard "@sp/sync-providers/*" tsconfig path
  alias with 11 explicit subpath entries matching package.json exports;
  deep-internal imports now fail at typecheck.
- sync-providers: move @sp/sync-core from dependencies to
  peerDependencies (kept in devDependencies for tests).
- both packages: add composite: true to enable project references;
  introduce tsconfig.build.json overlay so tsup DTS bundler still works.
- gitignore: ignore **/*.tsbuildinfo composite outputs.

* refactor(sync-core): prune 47 unused barrel exports

Removes exports with zero consumers outside the package. Source files
are unchanged; only the public barrel is trimmed. Covers compression
helper classes, sync-file-prefix error/config types, replay coordinator
internals, remote-apply result types, upload/download planning option
and plan types, ports misc, conflict-resolution helper types, and
sync-import-filter decision types.

* refactor(sync-core): drop unused encryption migration path

decryptWithMigration and DecryptResult had no host consumer; they
exposed a structural-migration entry point ("here is your ciphertext
re-encrypted under Argon2id") that nothing in the codebase reads. The
side-channel setLegacyKdfWarningHandler — which IS used — stays.

encryptWithDerivedKey/decryptWithDerivedKey lose their export keyword
and remain as module-internal helpers; encrypt/decrypt/encryptBatch/
decryptBatch still call them. Wire format and legacy-fallback semantics
are unchanged, so existing ciphertext continues to decrypt.

Test imports for compression and sync-file-prefix specs now go via
their source files instead of the trimmed barrel.

* fix(sync-providers): bound dropbox token refresh to single retry; share md5 rev helper

The five hand-rolled token-refresh blocks in Dropbox.{getFileRev,
downloadFile, uploadFile, removeFile, listFiles} recursed on themselves
after refresh. If the post-refresh call still saw a token error (real
case: the refresh token itself was revoked), the recursion would not
terminate. Consolidated into a single _withTokenRefresh helper that
attempts the call, refreshes once on a token error, retries once, then
lets the outer 401 classifier surface AuthFailSPError.

Same log message, same _isTokenError discriminator, same refresh call.
Same five sites still apply their post-call non-token error mapping
(NoRev, InvalidData, RemoteFileNotFound, path-not-found swallow, etc.).

Also extracts md5 content-rev computation duplicated between
LocalFileSyncBase._getLocalRev and WebdavApi._computeContentHash into a
shared file-based/content-rev.ts; both call sites preserve their own
error wrapping at the boundary.

* refactor(sync): split oversized super-sync and conflict-resolution

sync-providers: extract request-ID hashing from super-sync.ts (1017 ->
918 lines) into a new request-id.ts. The helpers were free functions
already in disguise (none referenced this), so the move is mechanical.
HTTP plumbing (_doWebFetch/_doNativeFetch/_fetchApi*) stays as private
methods — it transitively touches 12 instance members and would need
either a wide context object or a separate http-client collaborator
class to extract cleanly. Left as a follow-up.

sync-core: split conflict-resolution.ts into three cohesive files:
- entity-frontier.ts now owns buildEntityFrontier and
  adjustForClockCorruption (per-entity vector-clock domain).
- extractEntityFromPayload and extractUpdateChanges move to
  operation.types.ts next to the existing extractActionPayload.
- conflict-resolution.ts keeps deep-equality, LWW planning,
  partitioning, and identical-conflict detection.

Public barrel exports unchanged; tests now import the moved symbols
from their new homes.

* refactor(sync-core): drop redundant OperationStorePort

OperationStorePort overlapped with RemoteOperationApplyStorePort on the
two state-transition methods (markSynced/markApplied,
markRejected/markFailed) and had zero non-structural consumers — the
only implementer was OperationLogStoreService, which already exposes
the three methods as its own public surface. Removing the port leaves
the service contract intact and removes the verb-pair confusion noted
in the post-extraction review.

Spec contract test still drives the same state transitions; only the
local typing of the test fixture changes from the deleted interface to
Pick<OperationLogStoreService, ...>.

* refactor(sync-providers): decouple SuperSync provider from SP-specific host

Two coupling leaks the package shouldn't carry:

1. SUPER_SYNC_DEFAULT_BASE_URL was an implicit fallback inside
   SuperSyncProvider — an SP-specific URL baked into a "framework-
   agnostic" package. Make defaultBaseUrl a required SuperSyncDeps
   field; the host factory supplies the SP default. The constant stays
   exported as a suggested default for hosts targeting the SP-hosted
   server.

2. Consumers that wanted the WebSocket path had to do
   `provider as unknown as SuperSyncProvider` to call
   getWebSocketParams. Introduce SuperSyncWebSocketAccess interface +
   isSuperSyncWebSocketAccess structural guard; SuperSyncProvider
   implements it. sync-wrapper.service drops its cast in favor of the
   guard.

super-sync-restore.service still casts to SuperSyncProvider for the
restore path — same pattern would solve it, but out of scope here.

* test(sync-providers): extract shared test helpers and prefer barrels

Adds tests/helpers/sync-logger.ts and tests/helpers/credential-store.ts
to centralize the noopLogger and CredentialStore mocks that were copy-
pasted across 8 spec files. createStatefulCredentialStore covers the
"load/upsert/clear with state" cases; createMockCredentialStore covers
bare vi.fn() ports. Spec sites that needed a unique mockResolvedValue
chain it after the helper, preserving behavior 1:1.

Also migrates 5 spec files from deep ../src/<file> paths to the
matching sub-barrel (../src/webdav, /http, /super-sync, /platform) for
symbols already exported there. No new barrel exports added — internal
types (WebDavHttpAdapter, WebdavApi, DropboxApi, etc.) stay on deep
paths because they are intentionally not part of the public surface.

super-sync.spec.ts keeps its own credential/logger mocks (special
__asPort wrapper and vi.spyOn against the live NOOP_SYNC_LOGGER) that
the generic helpers cannot reproduce without bloat.

* test(sync): pin vector-clock pruning, error-meta privacy, and sync-import edges

Fills three test gaps surfaced by the post-extraction review:

- vector-clock pruning correctness across clocks: 4 cases pinning that
  pruning legitimately flips GREATER_THAN to CONCURRENT/LESS_THAN when
  the dropped keys are still present in the comparison clock. This is
  the documented behavior (compareVectorClocks is intentionally not
  pruning-aware); the protocol handles flips server-side via the
  rejected-ops retry loop. preserveClientIds case also covered.

- error-meta privacy boundary: 22 new cases covering urlPathOnly (strip
  query/fragment/userinfo, preserve host+path+port, leave non-URLs
  intact) and errorMeta (no leakage of headers, response bodies, OAuth
  tokens, signed-URL params, user emails, or attached error fields).
  Real negative assertions (.not.toContain), not shape checks.

- sync-import-filter edge cases: 8 cases covering empty clocks on
  either side, op clock listing the import client at 0, same-client
  with equal counter (pinning the strict-greater-than boundary), and
  different-client knowledge above the import counter.

sync-core 195 -> 207 tests, sync-providers 319 -> 341 tests; no
production code changed.

* style(sync-core): format sync-file-prefix.spec import line

* fix(sync): address package review feedback
2026-05-14 13:06:08 +02:00
..
src refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
tests refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
.gitignore build: ignore files 2026-05-13 19:59:02 +02:00
package.json refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
README.md refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
tsconfig.build.json refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
tsconfig.json refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
tsconfig.spec.json refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
tsup.config.ts refactor(sync): post-extraction review cleanup of @sp/sync-core and @sp/sync-providers (#7595) 2026-05-14 13:06:08 +02:00
vitest.config.ts refactor(sync): move vector clocks to sync-core 2026-05-11 15:21:08 +02:00

@sp/sync-core

Framework-agnostic primitives for the Super Productivity sync engine: operation-log types, vector clocks, conflict resolution, gzip compression, and end-to-end encryption. Consumed by the main app and the SuperSync server; no Angular/Electron/Capacitor dependencies.

Encryption

The encryption layer provides Argon2id key derivation and AES-256-GCM authenticated encryption, with a WebCrypto path and an @noble/ciphers fallback for environments where crypto.subtle is unavailable (notably Android Capacitor on http://localhost).

import {
  encrypt,
  decrypt,
  encryptBatch,
  decryptBatch,
  clearSessionKeyCache,
  setLegacyKdfWarningHandler,
} from '@sp/sync-core';

const cipher = await encrypt('hello', password);
const plain = await decrypt(cipher, password);

Wire format (public contract)

Format Bytes
Argon2id [SALT (16)] [IV (12)] [AES-GCM ciphertext + auth tag (>= 16)]
Legacy [IV (12)] [AES-GCM ciphertext + auth tag (>= 16)]

All ciphertexts are base64-encoded for transport. The format is discriminated by length: < 28 bytes is invalid, < 44 bytes is unambiguously legacy, >= 44 bytes is treated as Argon2id with a legacy fallback on auth failure. Do not change this without a versioning migration.

Salt and IV semantics

  • The IV (12 bytes) is freshly random per call. AES-GCM security under a fixed key reduces to IV uniqueness, which this guarantees.
  • The salt (16 bytes) is derived once per (process session, password) pair and reused across every encrypt/encryptBatch call in that session. This is intentional — it lets the session cache amortize the ~500 ms2 s Argon2id derivation. Two encryptions of the same plaintext within a session therefore share the salt prefix and differ only in IV and ciphertext. Do not assert per-call salt uniqueness in tests.

Session key caching

encrypt/decrypt/encryptBatch/decryptBatch all share three in-memory caches (encrypt key, decrypt key by salt, legacy PBKDF2 key) that survive across sync cycles. Argon2id derivation is expensive (~5002000 ms on mobile with the default 64 MiB / 3 iterations); the cache turns repeated syncs from minutes into seconds.

Call clearSessionKeyCache() whenever the user changes their password or logs out. Keys live in memory only and are never persisted.

Legacy-KDF migration

Old data was encrypted with PBKDF2 using the password as its own salt — cryptographically weak. decrypt() and decryptBatch() still read legacy ciphertexts so existing sync data remains accessible.

setLegacyKdfWarningHandler(fn) registers a callback fired on every successful legacy decrypt, regardless of which entry point was used. The host throttles user-facing messages (e.g. show a deprecation banner once per session).

Argon2id parameters

Defaults are OWASP 2023 mobile guidance (parallelism: 1, iterations: 3, memorySize: 64 MiB). Tests can weaken them via setArgon2ParamsForTesting({ ... }) — this throws when called with NODE_ENV === 'production' in Node bundles. Restore defaults by calling with no argument.

Other exports

  • OpType, Operation, VectorClock and friends — op-log primitive types
  • compareVectorClocks, mergeVectorClocks, limitVectorClockSize — clock algebra
  • classifyOpAgainstSyncImport — full-state-import op disposition
  • createSyncFilePrefixHelpers — host-configured file prefix codec
  • compressWithGzip, decompressGzipFromString — gzip helpers
  • replayOperationBatch, applyRemoteOperations — replay and apply coordinators
  • planRegularOpsAfterFullStateUpload, planSnapshotHydration, etc. — sync planning

See src/index.ts for the full barrel and the JSDoc on individual symbols for usage.

Tests

npm test           # typecheck specs + vitest run, Node WebCrypto + @noble fallback
npm run test:watch # watch mode
npm run build      # tsup -> ESM + CJS + .d.ts

Browser-context smoke coverage lives in the consuming app at src/app/op-log/encryption/encryption.browser.spec.ts (Karma + real Chrome).