Commit graph

23 commits

Author SHA1 Message Date
Thorsten Klein
7fa8f12132
feat(sync): add Helm chart and WebSocket push for SuperSync (#6971)
* feat(sync): add Helm chart for SuperSync Kubernetes deployment

Includes Deployment, StatefulSet (PostgreSQL), Service, Ingress,
ConfigMap, Secret, HPA, PDB, NetworkPolicy, and test templates.
Supports both bundled PostgreSQL and external database configurations.

* feat(sync): add WebSocket push notifications for near-realtime sync

Server: Fastify WebSocket plugin with connection manager, app-level
heartbeat (30s), debounced notifications, and per-user routing.
Client: WebSocket service with exponential backoff reconnection,
WS-triggered download service, and reduced polling when connected.

* fix(sync): improve WebSocket error handling and reactivity

Make syncInterval$ reactive to WS connection state, fix Set/Map
mutation during heartbeat iteration, add error handling to WS route
handler, separate JSON parsing from message handling, detect auth
failures in WS-triggered downloads, and add logging to all empty
catch blocks.

* fix(sync): address PR review findings for WebSocket and Helm

Fix race condition in WS-triggered download pipeline by moving
isSyncInProgress filter after debounce and adding guard in
_downloadOps. Add logging to remaining empty catch blocks, fix
missing $NODE_IP in Helm NOTES.txt, and correct inaccurate
comments in values.yaml and ws-triggered-download.service.ts.

* fix(sync): add rate limiting to WebSocket upgrade endpoint

Limit WS connection attempts to 10 per minute per IP to prevent
connection flooding, matching the rate-limit pattern used by other
sync endpoints.

* fix(sync): extract shared constants, fix debounce correctness, replace deprecated toPromise

Extract CLIENT_ID_REGEX and MAX_CLIENT_ID_LENGTH into sync.const.ts
to prevent drift between sync.routes.ts and websocket.routes.ts.

Fix debounce in notifyNewOps to accumulate excluded client IDs across
rapid calls from different clients, preventing self-notifications.

Replace deprecated toPromise() with firstValueFrom in connectWebSocket
and _sync methods. Add .catch() to reconnect path and logging to
silent early returns in _downloadOps.

* fix(build): restore correct package-lock.json and fix upstream lint errors

* revert: restore upstream HTML formatting to match CI prettier config

* fix(sync): use DownloadOutcome discriminated union in WsTriggeredDownloadService

* fix(sync): narrow TokenVerificationResult before accessing userId

* fix(sync): await async getProviderById in connectWebSocket

Missing await caused the Promise object to be cast to
SuperSyncProvider, making getWebSocketParams undefined.

* feat(sync): add SEED_USERS env var to create verified users on startup

For self-hosted single-user setups: set SEED_USERS=email1,email2 to
create verified users on boot and log their access tokens. Skips
existing users. Removes need for SMTP/magic link registration.

Also fix Dockerfile to use npm install instead of npm ci for lockfile
compatibility.

* fix(sync): address PR review feedback for Helm chart and server

Security:
- Remove seed-users.ts (logged full JWT tokens to stdout)
- Add fail assertions for missing jwtSecret and postgresql.password
- Add smtp.user/smtp.password values fields
- Add from: selector to NetworkPolicy ingress
- Add egress rule for external database when postgresql.enabled=false

Correctness:
- Fix PostgreSQL StatefulSet indentation for non-persistent mode
- Fix NOTES.txt panic on empty tls list (use len instead of index)
- Restore npm ci in Dockerfile by using node:24-alpine (ships npm 11)
- Add Recreate deployment strategy when using RWO PVC

Operational:
- Add fail guard preventing HPA maxReplicas > 1 (in-memory WS state)
- Fix PDB to use maxUnavailable instead of minAvailable
- Add WebSocket ingress timeout annotation examples
- Add Prisma db push init container for schema migrations
- Default serviceAccount.automount to false
- Add Chart.yaml maintainers, home, sources metadata

* feat(sync): add ALLOWED_EMAILS env var to restrict registration

Supports fully qualified emails (user@example.com) and domain
wildcards (*@example.com). When unset, all emails are allowed.
Applied to all three registration endpoints (passkey options,
passkey verify, magic link).

* fix(sync): exempt health endpoint from rate limiting

Kubernetes liveness/readiness probes hit /health every 5-15s,
exhausting the global rate limit (100 req/15min) and causing
429 responses that trigger container restarts.

* fix(sync): harden WebSocket, Helm chart, and sync services from multi-agent review

- Pin prisma@5.22.0 in init container and use migrate deploy instead of db push
- Add per-user WebSocket connection limit (max 10) to prevent resource exhaustion
- Add replicaCount > 1 fail guard in deployment template (in-memory WS state)
- Set maxPayload: 1024 on WebSocket plugin (only pong messages expected)
- Remove premature IN_SYNC status from download-only WS-triggered sync
- Fix double removeConnection on error+close events (let close handle cleanup)
- DRY debounce logic in notifyNewOps and store latestSeq on pending entry
- Remove dead fromClientId from NewOpsNotification and WsMessage interfaces
- Add takeUntilDestroyed to _wsProviderCleanup subscription
- Simplify email-allowlist.ts to eager init (eliminate mutable state)
- Guard connectWebSocket at call site for non-SuperSync providers
- Add distinctUntilChanged to syncInterval$ to prevent unnecessary resets
- Add container-level securityContext to PostgreSQL StatefulSet
- Fix HPA maxReplicas default to 1 (matches single-replica constraint)

* fix(sync): restore migration in Dockerfile CMD for non-Helm Docker users

Helm users get migration via init container (runs first, CMD becomes no-op).
Docker-compose/plain Docker users get automatic migration back on startup.

* fix(boards): add missing drag delay for touch and extract shared signal

Add cdkDragStartDelay to board-panel drag items, which was missing
entirely. Extract the repeated `isTouchActive() ? DRAG_DELAY_FOR_TOUCH : 0`
expression into a shared `dragDelayForTouch` computed signal and refactor
all 8 components to use it.

* test(sync): add comprehensive WebSocket test coverage

Add unit tests for the new WebSocket real-time sync notification pipeline:

- WebSocketConnectionService (server): connection lifecycle, max-per-user
  limits, notification debouncing, heartbeat, graceful shutdown (13 tests)
- WebSocket routes validation: token/clientId validation, close codes,
  error handling, validation ordering (18 tests)
- SuperSyncWebSocketService (frontend): reconnection with exponential
  backoff, heartbeat, disconnect cleanup, URL conversion (6 tests)
- WsTriggeredDownloadService: auth error handling, pipeline resilience,
  start() idempotency (3 tests)
- SyncWrapperService: connectWebSocket guards for non-SuperSync, null
  params, and already-connected cases (3 tests)
- E2E realtime push: verifies WS-triggered download between two clients

Quality fixes:
- Replace waitForTimeout with syncAndWait in E2E
- Add explanatory comment for microtask flushing in sync-wrapper spec
- Use getter for isSyncInProgress mock in ws-triggered-download spec

---------

Co-authored-by: Johannes Millan <johannes.millan@gmail.com>
2026-03-30 21:34:30 +02:00
Johannes Millan
f26732755f fix(sync): prevent transient server errors from wiping auth credentials
Server: verifyToken() catch-all was returning 401 "Invalid token" for ALL
errors including database failures. Now only JWT-specific errors
(JsonWebTokenError, TokenExpiredError) return 401. Database/infrastructure
errors propagate as 500s with sanitized logging.

Client: SuperSync AuthFailSPError no longer clears stored credentials —
the server rejection may be transient. The snackbar with "Configure"
button still shows so users can re-auth on genuine failures. Dropbox and
WebDAV still clear immediately so their OAuth/re-auth flows work.
2026-03-20 21:36:30 +01:00
Johannes Millan
7d27400d03 perf(sync): lazy-load sync providers and repair utilities to reduce bundle size
Convert sync providers from eagerly instantiated array to lazy-loaded
factories using dynamic imports. Make dataRepair and migrateLegacyBackup
dynamically imported at their call sites (error/migration paths only).

- getProviderById is now async with promise-cached instantiation
- _setActiveProvider guards against redundant re-activation
- Staleness counter prevents race conditions on rapid provider switches
2026-03-10 10:13:09 +01:00
Johannes Millan
efaffe97a5 refactor(sync): use discriminated unions for sync orchestrator results
Replace flag-based result objects with DownloadOutcome and UploadOutcome
discriminated unions at the orchestrator level. This eliminates null
returns, optional boolean checks, and ambiguous flag combinations.

- Add DownloadOutcome (5 variants) and UploadOutcome (3 variants) types
- Update OperationLogSyncService to return discriminated unions
- Update SyncWrapperService and ImmediateUploadService consumers
- Use exhaustive switch in downloadCallback adapter
- Replace piggybackedOps array pass-through with piggybackedOpsCount
2026-03-06 12:38:09 +01:00
Johannes Millan
5b47358842 fix(sync): detect provider switch and force fresh download to trigger conflict dialog
When switching sync providers (e.g., SuperSync → Dropbox), the stale
lastServerSeq causes sinceSeq > 0, skipping snapshotState and the
conflict dialog. Both clients silently report IN_SYNC with no data
transfer.

Track last synced provider in localStorage. On provider switch, pass
forceFromSeq0 to downloadRemoteOps so sinceSeq=0 returns snapshotState,
entering the conflict dialog path. Also check NgRx store for meaningful
data when pending ops are config-only (provider-switch scenario).
2026-03-01 21:27:09 +01:00
Johannes Millan
baa9a46935 test(sync): add encryption and supersync e2e test coverage
Add comprehensive test coverage for the encryption and SuperSync features:

- Add e2e tests for encryption prompt loop, error scenarios, no-op sync
- Add e2e tests for provider switch, re-enable, account reset, server migration
- Add e2e tests for import clean-slate scenario
- Add unit tests for encryption dialogs, toggle, enable/disable services
- Add unit tests for sync-config, sync-wrapper, file-based-encryption
- Add unit tests for operation-log-sync, sync-import-filter, clean-slate
- Update supersync page object for mandatory encryption flows
2026-03-01 21:27:08 +01:00
Johannes Millan
90900e3aa1 fix(sync): set error state on SuperSync auth failure
Auth error branch (AuthFailSPError, MissingCredentialsSPError,
MissingRefreshTokenAPIError) was the only error handler not calling
setSyncStatus('ERROR') or clearing stale sync status signals. This
caused the sync icon to never show sync_problem and stale checkmarks
to linger after auth failure.
2026-03-01 21:27:08 +01:00
Johannes Millan
08e8329f97 refactor(sync): consolidate LegacySyncProvider into SyncProviderId
Remove duplicate LegacySyncProvider enum and use SyncProviderId everywhere.
The two enums had identical values but were separate types for historical
reasons, making the "Legacy" name misleading since providers are actively used.
2026-02-15 11:19:22 +01:00
Johannes Millan
2e2dad8eb8 fix(sync): clear stale auth credentials on auth errors to enable re-auth
When Dropbox tokens become invalid (e.g., after enabling 2FA), isReady()
still returns true because tokens are non-empty, preventing the re-auth
dialog from opening. Add clearAuthCredentials() to each sync provider
that clears auth fields while preserving non-auth config (encryptKey),
and call it in the auth error handler so the Configure button works.

Closes #6397
2026-02-07 13:40:06 +01:00
Johannes Millan
3c0aa03822 fix(sync): handle MissingCredentialsSPError with re-auth prompt
Previously, MissingCredentialsSPError (thrown when sync credentials are
missing or cleared) fell through to the generic error handler, showing
a raw error string like "Dropbox no token" with no actionable recovery.
Now it shows the "Incomplete configuration" snack with a "Configure"
button, matching the existing AuthFailSPError and MissingRefreshTokenAPIError
behavior. Also consolidates the three identical handlers into one branch.
2026-02-07 13:04:49 +01:00
Johannes Millan
c5409bbd25 fix(sync): add server pruning log, harden test coverage for vector clocks
- Add Logger.warn() in ValidationService when oversized vector clocks
  are pruned server-side, improving observability for buggy clients
- Add tests for hasVectorClockChanges (previously zero coverage)
- Add symmetric LESS_THAN test for pruning-aware comparison mode
- Export MAX_LWW_REUPLOAD_RETRIES constant to eliminate magic number
  coupling between sync-wrapper service and its test
- Add test verifying uploading client ID is preserved during server-side
  clock pruning
- Add test for limitVectorClockSize when preserveClientIds exceeds MAX
2026-01-30 19:50:30 +01:00
Johannes Millan
495f0fe0d3 fix(sync): improve vector clock pruning awareness and harden tests
Make hasVectorClockChanges pruning-aware by checking clock size before
logging missing keys — downgrade to verbose when pruning is likely,
warn when corruption is likely. Add return value assertion to LWW retry
exhaustion test, asymmetric pruning test cases, server-side pruning
tradeoff documentation, and fix stale "max 50" in docs.
2026-01-30 19:35:51 +01:00
Johannes Millan
fd3d06c5ff test(sync): add tests for vector clock pruning and LWW retry loop fix
Cover all 4 changes from cb36c09538: shared-schema vector clock tests
(22 Vitest), sync wrapper LWW retry loop limit tests (4 Jasmine),
client vector clock pruning-aware comparison tests (3 Jasmine), and
E2E tests for heavy LWW conflict, TAG:TODAY convergence, and
multi-client vector clock consistency (3 Playwright).
2026-01-30 19:04:44 +01:00
Johannes Millan
2a83a0d627 test(sync): add tests for isEncryptionOperationInProgress and runWithSyncBlocked
- Test getter returns correct values before, during, and after operations
- Test runWithSyncBlocked executes operations and propagates results/errors
- Test sync is blocked during encryption operations
- Test sync is allowed after encryption operations complete
- Test runWithSyncBlocked waits for ongoing sync before starting
2026-01-27 16:16:27 +01:00
Johannes Millan
c29194c17c test(sync): add comprehensive timeout handling tests
Add unit tests for timeout handling functionality:

- SuperSync provider: AbortController cleanup, error conversion, logging
- Sync wrapper: timeout error detection with various error patterns
- Restore service: exponential backoff retry logic (2s, 4s delays)

Tests validate:
- Memory leak prevention (timeout cleanup)
- AbortError → timeout message conversion
- Network error classification for retries
- Retry attempt counting (max 3 attempts)
- Case-insensitive error detection

All tests passing (143 total across 3 test suites)
2026-01-20 17:07:24 +01:00
Johannes Millan
c9075bbe4d fix(test): update test expectations to match implementation changes
- Add opId parameter to uploadSnapshot expectation
- Update setSyncStatus assertion to check for IN_SYNC specifically
- Add icon property to CreateTagData expectation
- Increase performance test threshold for CI variability
2026-01-15 17:15:40 +01:00
Johannes Millan
6501950760 fix: resolve build issues and update E2E tests for sync import conflict
- Fix TypeScript error in passkey.ts (Buffer to Uint8Array conversion)
- Update Dockerfile.test to use npm install instead of npm ci for workspace sync
- Update E2E test to not setup sync before import
- Add additional unit tests for sync-wrapper.service

Note: E2E tests for sync import conflict dialog need further work due to
automatic upload of SYNC_IMPORT when sync is enabled. The core dialog
functionality is implemented and unit tested.
2026-01-10 18:48:47 +01:00
Johannes Millan
4780f08333 fix(sync): show error instead of falsely claiming IN_SYNC when upload fails
When SYNC_IMPORT upload is rejected (e.g., payload too complex/large),
the sync status was incorrectly set to IN_SYNC, misleading users into
thinking their data was synced when it wasn't.

Changes:
- Check uploadResult.rejectedCount before setting IN_SYNC
- Set status to ERROR when upload has rejected operations
- Show alert dialog for payload size/complexity errors
- Add ERROR_PAYLOAD_TOO_LARGE translation key
- Add tests for rejected ops handling
2026-01-10 18:38:49 +01:00
Johannes Millan
40def4a576 fix(sync): enable checkmark indicators for all sync providers
The checkmark UI was only working for SuperSync. Now all providers
(WebDAV, Dropbox, LocalFile) show the correct state:
- No checkmark: pending local changes not yet uploaded
- Single checkmark: all local changes uploaded
- Double checkmark: uploaded AND remote checked within 1 minute

Changes:
- Add hasNoPendingOps signal to SuperSyncStatusService
- Mark pending ops when local operations are created in effects
- Replace syncState-based UI logic with pending ops signal
- Add 1-minute timer expiry for remote check freshness
2026-01-08 12:31:14 +01:00
Johannes Millan
77d09796d8 test(sync): add edge case tests for conflict resolution error handling
- Add tests for forceUploadLocalState/forceDownloadRemoteState failures
- Add test for provider becoming unavailable during resolution
- Add tests for UserInputWaitState lifecycle (startWaiting/stopWaiting)
- Fix bug: add catch block to _handleLocalDataConflict to handle
  resolution errors (previously caused unhandled promise rejections)
2026-01-08 11:54:54 +01:00
Johannes Millan
2b5fafccae feat(sync): add local data conflict handling for file-based sync
Add conflict resolution when file-based sync (WebDAV/Dropbox) detects
local unsynced changes that would be lost if remote snapshot is applied.

Changes:
- Add LocalDataConflictError to signal first-sync conflicts
- Add _handleLocalDataConflict in SyncWrapperService to show dialog
- User can choose USE_LOCAL (upload) or USE_REMOTE (download)
- Add forceUploadLocalState and forceDownloadRemoteState methods
- Fix test mocks and update test expectations for new behavior

The conflict dialog allows users to explicitly choose whether to keep
their local changes or accept the remote state during first sync.
2026-01-08 10:10:48 +01:00
Johannes Millan
d50dc73674 fix(e2e): fix stale-clock regression tests timeout after import
Replace waitForAppReady() with waitForLoadState('networkidle') after
page reloads in stale-clock regression tests. waitForAppReady expects
route matching /#/(tag|project)/.+/tasks, but after reload with
imported data the app may navigate to a different default route,
causing test timeout.

Also includes sync-wrapper.service.spec.ts test improvements.
2026-01-07 20:58:18 +01:00
Johannes Millan
6470725eaf fix(sync): improve robustness and safety of sync operations
- Fix race condition in sync-wrapper.service.ts with early return check
- Convert async subscribe anti-patterns to firstValueFrom for proper error handling
- Add atomic flag to conflict dialog timeout to prevent race conditions
- Create safety backup before conflict resolution via SyncSafetyBackupService
- Add SyncAlreadyInProgressError class to replace fragile string-based detection
- Redact sensitive fields (passwords, encryption keys) in sync config logging
- Add type-safe toSyncProviderId helper to replace unsafe double assertions
- Add defensive null check for vector clocks in sync status comparison
- Extract magic numbers to documented constants (SYNC_WAIT_TIMEOUT_MS, etc.)
- Add placeholder test file for sync-wrapper.service.ts documenting expected tests
2025-12-15 10:47:35 +01:00