super-productivity/packages/super-sync-server
Thorsten Klein 7fa8f12132
feat(sync): add Helm chart and WebSocket push for SuperSync (#6971)
* feat(sync): add Helm chart for SuperSync Kubernetes deployment

Includes Deployment, StatefulSet (PostgreSQL), Service, Ingress,
ConfigMap, Secret, HPA, PDB, NetworkPolicy, and test templates.
Supports both bundled PostgreSQL and external database configurations.

* feat(sync): add WebSocket push notifications for near-realtime sync

Server: Fastify WebSocket plugin with connection manager, app-level
heartbeat (30s), debounced notifications, and per-user routing.
Client: WebSocket service with exponential backoff reconnection,
WS-triggered download service, and reduced polling when connected.

* fix(sync): improve WebSocket error handling and reactivity

Make syncInterval$ reactive to WS connection state, fix Set/Map
mutation during heartbeat iteration, add error handling to WS route
handler, separate JSON parsing from message handling, detect auth
failures in WS-triggered downloads, and add logging to all empty
catch blocks.

* fix(sync): address PR review findings for WebSocket and Helm

Fix race condition in WS-triggered download pipeline by moving
isSyncInProgress filter after debounce and adding guard in
_downloadOps. Add logging to remaining empty catch blocks, fix
missing $NODE_IP in Helm NOTES.txt, and correct inaccurate
comments in values.yaml and ws-triggered-download.service.ts.

* fix(sync): add rate limiting to WebSocket upgrade endpoint

Limit WS connection attempts to 10 per minute per IP to prevent
connection flooding, matching the rate-limit pattern used by other
sync endpoints.

* fix(sync): extract shared constants, fix debounce correctness, replace deprecated toPromise

Extract CLIENT_ID_REGEX and MAX_CLIENT_ID_LENGTH into sync.const.ts
to prevent drift between sync.routes.ts and websocket.routes.ts.

Fix debounce in notifyNewOps to accumulate excluded client IDs across
rapid calls from different clients, preventing self-notifications.

Replace deprecated toPromise() with firstValueFrom in connectWebSocket
and _sync methods. Add .catch() to reconnect path and logging to
silent early returns in _downloadOps.

* fix(build): restore correct package-lock.json and fix upstream lint errors

* revert: restore upstream HTML formatting to match CI prettier config

* fix(sync): use DownloadOutcome discriminated union in WsTriggeredDownloadService

* fix(sync): narrow TokenVerificationResult before accessing userId

* fix(sync): await async getProviderById in connectWebSocket

Missing await caused the Promise object to be cast to
SuperSyncProvider, making getWebSocketParams undefined.

* feat(sync): add SEED_USERS env var to create verified users on startup

For self-hosted single-user setups: set SEED_USERS=email1,email2 to
create verified users on boot and log their access tokens. Skips
existing users. Removes need for SMTP/magic link registration.

Also fix Dockerfile to use npm install instead of npm ci for lockfile
compatibility.

* fix(sync): address PR review feedback for Helm chart and server

Security:
- Remove seed-users.ts (logged full JWT tokens to stdout)
- Add fail assertions for missing jwtSecret and postgresql.password
- Add smtp.user/smtp.password values fields
- Add from: selector to NetworkPolicy ingress
- Add egress rule for external database when postgresql.enabled=false

Correctness:
- Fix PostgreSQL StatefulSet indentation for non-persistent mode
- Fix NOTES.txt panic on empty tls list (use len instead of index)
- Restore npm ci in Dockerfile by using node:24-alpine (ships npm 11)
- Add Recreate deployment strategy when using RWO PVC

Operational:
- Add fail guard preventing HPA maxReplicas > 1 (in-memory WS state)
- Fix PDB to use maxUnavailable instead of minAvailable
- Add WebSocket ingress timeout annotation examples
- Add Prisma db push init container for schema migrations
- Default serviceAccount.automount to false
- Add Chart.yaml maintainers, home, sources metadata

* feat(sync): add ALLOWED_EMAILS env var to restrict registration

Supports fully qualified emails (user@example.com) and domain
wildcards (*@example.com). When unset, all emails are allowed.
Applied to all three registration endpoints (passkey options,
passkey verify, magic link).

* fix(sync): exempt health endpoint from rate limiting

Kubernetes liveness/readiness probes hit /health every 5-15s,
exhausting the global rate limit (100 req/15min) and causing
429 responses that trigger container restarts.

* fix(sync): harden WebSocket, Helm chart, and sync services from multi-agent review

- Pin prisma@5.22.0 in init container and use migrate deploy instead of db push
- Add per-user WebSocket connection limit (max 10) to prevent resource exhaustion
- Add replicaCount > 1 fail guard in deployment template (in-memory WS state)
- Set maxPayload: 1024 on WebSocket plugin (only pong messages expected)
- Remove premature IN_SYNC status from download-only WS-triggered sync
- Fix double removeConnection on error+close events (let close handle cleanup)
- DRY debounce logic in notifyNewOps and store latestSeq on pending entry
- Remove dead fromClientId from NewOpsNotification and WsMessage interfaces
- Add takeUntilDestroyed to _wsProviderCleanup subscription
- Simplify email-allowlist.ts to eager init (eliminate mutable state)
- Guard connectWebSocket at call site for non-SuperSync providers
- Add distinctUntilChanged to syncInterval$ to prevent unnecessary resets
- Add container-level securityContext to PostgreSQL StatefulSet
- Fix HPA maxReplicas default to 1 (matches single-replica constraint)

* fix(sync): restore migration in Dockerfile CMD for non-Helm Docker users

Helm users get migration via init container (runs first, CMD becomes no-op).
Docker-compose/plain Docker users get automatic migration back on startup.

* fix(boards): add missing drag delay for touch and extract shared signal

Add cdkDragStartDelay to board-panel drag items, which was missing
entirely. Extract the repeated `isTouchActive() ? DRAG_DELAY_FOR_TOUCH : 0`
expression into a shared `dragDelayForTouch` computed signal and refactor
all 8 components to use it.

* test(sync): add comprehensive WebSocket test coverage

Add unit tests for the new WebSocket real-time sync notification pipeline:

- WebSocketConnectionService (server): connection lifecycle, max-per-user
  limits, notification debouncing, heartbeat, graceful shutdown (13 tests)
- WebSocket routes validation: token/clientId validation, close codes,
  error handling, validation ordering (18 tests)
- SuperSyncWebSocketService (frontend): reconnection with exponential
  backoff, heartbeat, disconnect cleanup, URL conversion (6 tests)
- WsTriggeredDownloadService: auth error handling, pipeline resilience,
  start() idempotency (3 tests)
- SyncWrapperService: connectWebSocket guards for non-SuperSync, null
  params, and already-connected cases (3 tests)
- E2E realtime push: verifies WS-triggered download between two clients

Quality fixes:
- Replace waitForTimeout with syncAndWait in E2E
- Add explanatory comment for microtask flushing in sync-wrapper spec
- Use getter for isSyncInProgress mock in ws-triggered-download spec

---------

Co-authored-by: Johannes Millan <johannes.millan@gmail.com>
2026-03-30 21:34:30 +02:00
..
archive/encryption-attempts-openvz-incompatible chore(sync-server): archive non-working encryption implementations 2026-01-23 17:36:01 +01:00
docs feat(sync): add backup strategy with accounts-only dump and disaster recovery docs 2026-03-18 20:15:45 +01:00
helm/supersync feat(sync): add Helm chart and WebSocket push for SuperSync (#6971) 2026-03-30 21:34:30 +02:00
prisma fix(sync): transmit syncImportReason through sync pipeline 2026-03-29 20:06:41 +02:00
public feat(sync): add magic link registration to SuperSync server 2026-03-12 13:36:05 +01:00
scripts fix(sync-server): remove jq dependency from deploy script 2026-03-27 17:55:49 +01:00
src feat(sync): add Helm chart and WebSocket push for SuperSync (#6971) 2026-03-30 21:34:30 +02:00
tests feat(sync): add Helm chart and WebSocket push for SuperSync (#6971) 2026-03-30 21:34:30 +02:00
tools chore(sync-server): archive non-working encryption implementations 2026-01-23 17:36:01 +01:00
.env.example docs(supersync): document CORS wildcard syntax in .env.example 2026-01-24 21:14:57 +01:00
.gitignore fix(sync-server): security hardening and GDPR log compliance 2026-03-18 20:48:51 +01:00
Caddyfile fix(sync-server): remove invalid timeout subdirective from Caddyfile 2026-03-23 13:05:40 +01:00
docker-compose.build.yml feat(sync-server): add GHCR-based deployment workflow 2025-12-19 15:24:20 +01:00
docker-compose.monitoring.yml feat(sync-server): add deployment and monitoring scripts 2025-12-19 14:26:12 +01:00
docker-compose.yml fix(sync-server): harden deploy with Caddyfile validation and container checks 2026-03-23 13:34:05 +01:00
DOCKER-MONITORING.md Add active users monitoring command with engagement metrics (#6921) 2026-03-22 23:15:14 +01:00
Dockerfile feat(sync): add Helm chart and WebSocket push for SuperSync (#6971) 2026-03-30 21:34:30 +02:00
Dockerfile.test fix: resolve build issues and update E2E tests for sync import conflict 2026-01-10 18:48:47 +01:00
env.example feat(sync-server): add GHCR-based deployment workflow 2025-12-19 15:24:20 +01:00
package.json feat(sync): add Helm chart and WebSocket push for SuperSync (#6971) 2026-03-30 21:34:30 +02:00
privacy-policy-en.md docs(compliance): document encryption risk and update privacy policy 2026-01-22 13:34:54 +01:00
privacy-policy.md docs(compliance): document encryption risk and update privacy policy 2026-01-22 13:34:54 +01:00
README.md feat(sync): add backup strategy with accounts-only dump and disaster recovery docs 2026-03-18 20:15:45 +01:00
sync-server-architecture-diagrams.md refactor(sync): rename "stale" to "superseded" across sync/operation domain 2026-01-30 16:59:40 +01:00
terms-of-service-en.md docs: add draft for privacy policy and terms of service 2025-12-12 20:48:13 +01:00
terms-of-service.md docs: add draft for privacy policy and terms of service 2025-12-12 20:48:13 +01:00
tsconfig.json fix(sync-server): compile scripts for production Docker image 2025-12-19 15:58:21 +01:00
vitest.config.ts chore(sync): exclude integration test from default run, scope integration config 2026-03-17 13:59:40 +01:00
vitest.integration.config.ts chore(sync): exclude integration test from default run, scope integration config 2026-03-17 13:59:40 +01:00

SuperSync Server

A custom, high-performance synchronization server for Super Productivity.

Note: This server implements a custom operation-based synchronization protocol (Event Sourcing), not WebDAV. It is designed specifically for the Super Productivity client's efficient sync requirements.

Related Documentation:

Architecture

The server uses an Append-Only Log architecture backed by PostgreSQL (via Prisma):

  1. Operations: Clients upload atomic operations (Create, Update, Delete, Move).
  2. Sequence Numbers: The server assigns a strictly increasing server_seq to each operation.
  3. Synchronization: Clients request "all operations since sequence X".
  4. Snapshots: The server can regenerate the full state by replaying operations, optimizing initial syncs.

Key Design Principles

Principle Description
Server-Authoritative Server assigns monotonic sequence numbers for total ordering
Client-Side Conflict Resolution Server stores operations as-is; clients detect and resolve conflicts
E2E Encryption Support Payloads can be encrypted client-side; server treats them as opaque blobs
Idempotent Uploads Request ID deduplication prevents duplicate operations

Quick Start

The easiest way to run the server is using the provided Docker Compose configuration.

# 1. Copy environment example
cp .env.example .env

# 2. Configure .env (Set JWT_SECRET, DOMAIN, POSTGRES_PASSWORD)
nano .env

# 3. Start the stack (Server + Postgres + Caddy)
docker-compose up -d

Manual Setup (Development)

# Install dependencies
npm install

# Generate Prisma Client
npx prisma generate

# Set up .env
cp .env.example .env
# Edit .env to point to your PostgreSQL instance (DATABASE_URL)

# Push schema to DB
npx prisma db push

# Start the server
npm run dev

# Or build and run
npm run build
npm start

Configuration

All configuration is done via environment variables.

Variable Default Description
PORT 1900 Server port
DATABASE_URL - PostgreSQL connection string (e.g. postgresql://user:pass@localhost:5432/db)
JWT_SECRET - Required. Secret for signing JWTs (min 32 chars)
PUBLIC_URL - Required. Public URL used for email links (e.g. https://sync.example.com)
CORS_ORIGINS https://app.super-productivity.com Allowed CORS origins
SMTP_HOST - SMTP Server for emails

API Endpoints

Authentication

Register a new user

POST /api/register
Content-Type: application/json

{
  "email": "user@example.com",
  "password": "yourpassword"
}

Response:

{
  "message": "User registered. Please verify your email.",
  "id": 1,
  "email": "user@example.com"
}

Login

POST /api/login
Content-Type: application/json

{
  "email": "user@example.com",
  "password": "yourpassword"
}

Response:

{
  "token": "jwt-token",
  "user": { "id": 1, "email": "user@example.com" }
}

Synchronization

All sync endpoints require Bearer authentication: Authorization: Bearer <jwt-token>

1. Upload Operations

Send new changes to the server.

POST /api/sync/ops

2. Download Operations

Get changes from other devices.

GET /api/sync/ops?sinceSeq=123

3. Get Snapshot

Get the full current state (optimized).

GET /api/sync/snapshot

4. Sync Status

Check pending operations and device status.

GET /api/sync/status

Client Configuration

In Super Productivity, configure the Custom Sync provider with:

  • Base URL: https://sync.your-domain.com (or your deployed URL)
  • Auth Token: JWT token from login

Maintenance

Scripts

The server includes scripts for administrative tasks. These use the configured database.

# Delete a user account
npm run delete-user -- user@example.com

# Clear sync data (preserves account)
npm run clear-data -- user@example.com

# Clear ALL sync data (dangerous)
npm run clear-data -- --all

API Details

Upload Operations (POST /api/sync/ops)

Request body:

{
  "ops": [
    {
      "id": "uuid-v7",
      "opType": "UPD",
      "entityType": "TASK",
      "entityId": "task-123",
      "payload": { "changes": { "title": "New title" } },
      "vectorClock": { "clientA": 5 },
      "timestamp": 1701234567890,
      "schemaVersion": 1
    }
  ],
  "clientId": "clientA",
  "lastKnownSeq": 100
}

Response:

{
  "results": [{ "opId": "uuid-v7", "accepted": true, "serverSeq": 101 }],
  "newOps": [],
  "latestSeq": 101
}

Download Operations (GET /api/sync/ops)

Query parameters:

  • sinceSeq (required): Server sequence number to start from
  • limit (optional): Max operations to return (default: 500)

Upload Snapshot (POST /api/sync/snapshot)

Used for full-state operations (BackupImport, SyncImport, Repair):

{
  "state": {
    /* Full AppDataComplete */
  },
  "clientId": "clientA",
  "reason": "initial",
  "vectorClock": { "clientA": 10 },
  "schemaVersion": 1
}

Security Features

Feature Implementation
Authentication JWT Bearer tokens in Authorization header
Timing Attack Mitigation Dummy hash comparison on invalid users
Input Validation Operation ID, entity ID, schema version validated
Rate Limiting Configurable per-user limits
Vector Clock Sanitization Limited to 50 entries, 255 char keys
Entity Type Allowlist Prevents injection of invalid entity types
Request Deduplication Prevents duplicate operations on retry

Multi-Instance Deployment Considerations

When deploying multiple server instances behind a load balancer, be aware of these limitations:

Passkey Challenge Storage

Issue: WebAuthn challenges are stored in an in-memory Map, which doesn't work across instances.

Symptom: Passkey registration/login fails if the challenge generation request hits instance A but verification hits instance B.

Solution for multi-instance:

  • Implement Redis-backed challenge storage
  • Or use sticky sessions (less ideal)

Current status: A warning is logged at startup in production if in-memory storage is used.

Snapshot Generation Locks

Issue: Concurrent snapshot generation prevention uses an in-memory Map.

Symptom: Same user may trigger duplicate snapshot computations across different instances.

Impact: Performance only (no data corruption) - snapshots are deterministic.

Solution for multi-instance:

  • Implement Redis distributed lock (optional, only for performance)

Single-Instance Deployment

For single-instance deployments, these limitations do not apply. The current implementation is fully functional and well-tested for single-instance use.

Security Notes

  • Set JWT_SECRET to a secure random value in production (min 32 characters).
  • Use HTTPS in production. The Docker setup includes Caddy to handle this automatically.
  • Restrict CORS origins in production.
  • Database backups are recommended for production deployments.