Commit graph

14 commits

Author SHA1 Message Date
Thorsten Klein
7fa8f12132
feat(sync): add Helm chart and WebSocket push for SuperSync (#6971)
* feat(sync): add Helm chart for SuperSync Kubernetes deployment

Includes Deployment, StatefulSet (PostgreSQL), Service, Ingress,
ConfigMap, Secret, HPA, PDB, NetworkPolicy, and test templates.
Supports both bundled PostgreSQL and external database configurations.

* feat(sync): add WebSocket push notifications for near-realtime sync

Server: Fastify WebSocket plugin with connection manager, app-level
heartbeat (30s), debounced notifications, and per-user routing.
Client: WebSocket service with exponential backoff reconnection,
WS-triggered download service, and reduced polling when connected.

* fix(sync): improve WebSocket error handling and reactivity

Make syncInterval$ reactive to WS connection state, fix Set/Map
mutation during heartbeat iteration, add error handling to WS route
handler, separate JSON parsing from message handling, detect auth
failures in WS-triggered downloads, and add logging to all empty
catch blocks.

* fix(sync): address PR review findings for WebSocket and Helm

Fix race condition in WS-triggered download pipeline by moving
isSyncInProgress filter after debounce and adding guard in
_downloadOps. Add logging to remaining empty catch blocks, fix
missing $NODE_IP in Helm NOTES.txt, and correct inaccurate
comments in values.yaml and ws-triggered-download.service.ts.

* fix(sync): add rate limiting to WebSocket upgrade endpoint

Limit WS connection attempts to 10 per minute per IP to prevent
connection flooding, matching the rate-limit pattern used by other
sync endpoints.

* fix(sync): extract shared constants, fix debounce correctness, replace deprecated toPromise

Extract CLIENT_ID_REGEX and MAX_CLIENT_ID_LENGTH into sync.const.ts
to prevent drift between sync.routes.ts and websocket.routes.ts.

Fix debounce in notifyNewOps to accumulate excluded client IDs across
rapid calls from different clients, preventing self-notifications.

Replace deprecated toPromise() with firstValueFrom in connectWebSocket
and _sync methods. Add .catch() to reconnect path and logging to
silent early returns in _downloadOps.

* fix(build): restore correct package-lock.json and fix upstream lint errors

* revert: restore upstream HTML formatting to match CI prettier config

* fix(sync): use DownloadOutcome discriminated union in WsTriggeredDownloadService

* fix(sync): narrow TokenVerificationResult before accessing userId

* fix(sync): await async getProviderById in connectWebSocket

Missing await caused the Promise object to be cast to
SuperSyncProvider, making getWebSocketParams undefined.

* feat(sync): add SEED_USERS env var to create verified users on startup

For self-hosted single-user setups: set SEED_USERS=email1,email2 to
create verified users on boot and log their access tokens. Skips
existing users. Removes need for SMTP/magic link registration.

Also fix Dockerfile to use npm install instead of npm ci for lockfile
compatibility.

* fix(sync): address PR review feedback for Helm chart and server

Security:
- Remove seed-users.ts (logged full JWT tokens to stdout)
- Add fail assertions for missing jwtSecret and postgresql.password
- Add smtp.user/smtp.password values fields
- Add from: selector to NetworkPolicy ingress
- Add egress rule for external database when postgresql.enabled=false

Correctness:
- Fix PostgreSQL StatefulSet indentation for non-persistent mode
- Fix NOTES.txt panic on empty tls list (use len instead of index)
- Restore npm ci in Dockerfile by using node:24-alpine (ships npm 11)
- Add Recreate deployment strategy when using RWO PVC

Operational:
- Add fail guard preventing HPA maxReplicas > 1 (in-memory WS state)
- Fix PDB to use maxUnavailable instead of minAvailable
- Add WebSocket ingress timeout annotation examples
- Add Prisma db push init container for schema migrations
- Default serviceAccount.automount to false
- Add Chart.yaml maintainers, home, sources metadata

* feat(sync): add ALLOWED_EMAILS env var to restrict registration

Supports fully qualified emails (user@example.com) and domain
wildcards (*@example.com). When unset, all emails are allowed.
Applied to all three registration endpoints (passkey options,
passkey verify, magic link).

* fix(sync): exempt health endpoint from rate limiting

Kubernetes liveness/readiness probes hit /health every 5-15s,
exhausting the global rate limit (100 req/15min) and causing
429 responses that trigger container restarts.

* fix(sync): harden WebSocket, Helm chart, and sync services from multi-agent review

- Pin prisma@5.22.0 in init container and use migrate deploy instead of db push
- Add per-user WebSocket connection limit (max 10) to prevent resource exhaustion
- Add replicaCount > 1 fail guard in deployment template (in-memory WS state)
- Set maxPayload: 1024 on WebSocket plugin (only pong messages expected)
- Remove premature IN_SYNC status from download-only WS-triggered sync
- Fix double removeConnection on error+close events (let close handle cleanup)
- DRY debounce logic in notifyNewOps and store latestSeq on pending entry
- Remove dead fromClientId from NewOpsNotification and WsMessage interfaces
- Add takeUntilDestroyed to _wsProviderCleanup subscription
- Simplify email-allowlist.ts to eager init (eliminate mutable state)
- Guard connectWebSocket at call site for non-SuperSync providers
- Add distinctUntilChanged to syncInterval$ to prevent unnecessary resets
- Add container-level securityContext to PostgreSQL StatefulSet
- Fix HPA maxReplicas default to 1 (matches single-replica constraint)

* fix(sync): restore migration in Dockerfile CMD for non-Helm Docker users

Helm users get migration via init container (runs first, CMD becomes no-op).
Docker-compose/plain Docker users get automatic migration back on startup.

* fix(boards): add missing drag delay for touch and extract shared signal

Add cdkDragStartDelay to board-panel drag items, which was missing
entirely. Extract the repeated `isTouchActive() ? DRAG_DELAY_FOR_TOUCH : 0`
expression into a shared `dragDelayForTouch` computed signal and refactor
all 8 components to use it.

* test(sync): add comprehensive WebSocket test coverage

Add unit tests for the new WebSocket real-time sync notification pipeline:

- WebSocketConnectionService (server): connection lifecycle, max-per-user
  limits, notification debouncing, heartbeat, graceful shutdown (13 tests)
- WebSocket routes validation: token/clientId validation, close codes,
  error handling, validation ordering (18 tests)
- SuperSyncWebSocketService (frontend): reconnection with exponential
  backoff, heartbeat, disconnect cleanup, URL conversion (6 tests)
- WsTriggeredDownloadService: auth error handling, pipeline resilience,
  start() idempotency (3 tests)
- SyncWrapperService: connectWebSocket guards for non-SuperSync, null
  params, and already-connected cases (3 tests)
- E2E realtime push: verifies WS-triggered download between two clients

Quality fixes:
- Replace waitForTimeout with syncAndWait in E2E
- Add explanatory comment for microtask flushing in sync-wrapper spec
- Use getter for isSyncInProgress mock in ws-triggered-download spec

---------

Co-authored-by: Johannes Millan <johannes.millan@gmail.com>
2026-03-30 21:34:30 +02:00
Johannes Millan
554cc09608 feat(supersync): add production Docker monitoring wrapper and tooling
Add docker-monitor.sh wrapper script for running monitoring tools in
production Docker containers, with automatic tsx installation.

Changes:
- docker-monitor.sh: Wrapper script for all monitoring commands
  - Supports basic monitoring (stats, usage, ops, logs)
  - Supports analysis commands (user-deep-dive, large-ops, etc)
  - Auto-installs tsx in container when needed
  - Handles report extraction from container
- Dockerfile: Include TypeScript source files for tsx execution
- package.json: Add docker:monitor npm scripts
- DOCKER-MONITORING.md: Production Docker monitoring guide

New npm scripts:
- npm run docker:monitor -- <command>: Run any monitoring command
- npm run docker:monitor:usage: Quick storage check
- npm run docker:monitor:all: Full monitoring suite
- npm run docker🐚 Interactive shell

Usage:
  ./scripts/docker-monitor.sh usage
  ./scripts/docker-monitor.sh analyze user-deep-dive --user 29
  ./scripts/docker-monitor.sh monitor-all --save
  ./scripts/docker-monitor.sh get-reports ./reports
2026-01-23 14:31:24 +01:00
Johannes Millan
187bfaf24c fix(docker): upgrade npm to v11 to match lockfile version
npm 10.9.4 bundled with node:22-alpine handles optional peer
dependencies differently than npm 11, causing build failures
when resolving vitest peer dep from @angular-devkit/build-angular.
2026-01-10 19:08:38 +01:00
Johannes Millan
8a449fb944 fix(sync-server): compile scripts for production Docker image
ts-node is a devDependency and not available in production containers.
Updated tsconfig to compile scripts/ alongside src/, and changed npm
scripts to use compiled JS. Added monitor:dev for local development.
2025-12-19 15:58:21 +01:00
Johannes Millan
c83e66bf88 fix(super-sync-server): Copy public folder as sibling to dist
Server expects public at ../public relative to dist, not inside dist.
2025-12-12 20:48:40 +01:00
Johannes Millan
cc8fb01848 perf(super-sync-server): Optimize Dockerfile build time
- Create user before copying files, use --chown on COPY to avoid
  expensive chown -R on node_modules (was ~155s)
- Consolidate npm install commands into single layer
- Use npx prisma@version directly instead of install/remove cycle
- Clean npm cache to reduce image size
2025-12-12 20:48:40 +01:00
Johannes Millan
0b1e8646b8 fix(super-sync-server): Generate Prisma client in production stage
Instead of copying pre-generated .prisma directory from builder,
generate the client fresh in production stage to ensure correct paths.
2025-12-12 20:48:40 +01:00
Johannes Millan
4f21998a12 fix(super-sync-server): Copy generated prisma client to production image 2025-12-12 20:48:40 +01:00
Johannes Millan
e4da7a0ccb fix(super-sync-server): Fix permissions for prisma runtime 2025-12-12 20:48:40 +01:00
Johannes Millan
8110e6a5fb fix(super-sync-server): Robust Dockerfile with explicit runtime install 2025-12-12 20:48:40 +01:00
Johannes Millan
0c050044ed fix(super-sync-server): Simplify Dockerfile to single stage build 2025-12-12 20:48:40 +01:00
Johannes Millan
4bb3d53cf1 fix(super-sync-server): fix docker build missing hoisted dependencies 2025-12-12 20:48:40 +01:00
Johannes Millan
6cd93e57b3 fix(super-sync-server): Correct Dockerfile for npm workspace hoisted deps
This commit fixes an issue where the super-sync-server Docker image failed to start due to missing node_modules. When using npm workspaces, many dependencies are hoisted to the root node_modules directory. The previous Dockerfile only copied the workspace-specific node_modules, omitting these hoisted dependencies.

The fix changes the COPY command in the production stage of the Dockerfile to copy the root /app/node_modules, ensuring all necessary dependencies, including 'fastify', are present in the final image.
2025-12-12 20:48:40 +01:00
Johannes Millan
ff41dcaae0 build(superSync): add docker config 2025-12-12 20:48:40 +01:00