Prisma operation.aggregate({ _min: { serverSeq } }) compiles to MIN()
over a `SELECT ... OFFSET 0` subquery. The OFFSET is a Postgres planner
optimization fence, so the (user_id, server_seq) index could not serve a
first-row seek and the query degraded to a per-user O(N) scan. Under a
client reconnect stampede this ran minutes inside the 60s interactive
download transaction, blowing the tx timeout (500s) and exhausting the
connection pool (cascading upload+download failures).
Replace it with findFirst ordered by serverSeq asc, which compiles to
ORDER BY server_seq ASC LIMIT 1 — a guaranteed index seek on the existing
unique (user_id, server_seq) index. Behaviour-preserving: same minSeq
value and same null-when-empty semantics.
Update both consuming specs (operation-download.service, gap-detection)
to the two-findFirst-call shape and add a regression test asserting the
indexed query is used and the aggregate path is not reintroduced.
The op-log validation system (src/app/op-log/validation/) handles all
data repair at sync checkpoints. The legacy DataRepairService had no
callers anywhere in the app.
The outer attemptRecovery catch already logs the thrown Error, but the
console timeline doesn't make it obvious that the failure was a typia
validation rejection (vs. an IDB error or anything else). Log a
distinguishable line with the typia error count and any cross-model
error before throwing so support can grep for it.
The startup hydration path called validateAndRepair(), which opens a
native confirm()/alert() dialog when typia flags state. On Windows this
steals renderer focus and leaves shortcuts/clicks dead until the window
is minimized and restored — exactly the symptom in #7631.
Switch hydration to binary validateState() (no dialogs) and treat
failures as non-fatal: dispatch the data so the UI is live, but skip
the snapshot save at Checkpoint C so a corrupted state is not cached
for next boot. Also gate two destructive paths with the same validator:
refuse to import legacy data that fails validation, and refuse to
clear the migration backup unless the migrated snapshot validates.
Drop the dead _repairMutex field and its guards.
Merge an internal wake-up subject into GlobalTrackingIntervalService.tick$
so a tick can be force-emitted with a cap that prevents over-crediting
when the WebView was suspended past session end. Wire the capped tick
into focus-mode pause/complete bridge callbacks, plumb a wall-clock-correct
completedDuration onto completeFocusSession, and have the pause reducer
freeze elapsed via updateTimer.
INFO_OUTSIDE_WORK_HOURS and INFO_OVERLAP were appended after the QA_*
group when added in #7559. Move them before QA_* to match the
alphabetic convention used in surrounding blocks.
Two defensive fixes from the multi-agent review of #7582:
- Push side filtered tagIds via tagService.tags() which includes the
virtual TODAY_TAG. A stray TODAY id in task.tagIds (rule 5 violation,
but defensive) would push the literal "TODAY" string as a provider
label. Filter it before the title lookup.
- _mapLabelsToTagIds snapshots tagsNoMyDayAndNoList() once at entry,
so a provider returning ['bug','bug'] would create two tags (the
second iteration's find() doesn't see the addTag dispatched in the
first). Dedupe inputs with a Set and track tags created within the
same call via a local createdByTitle map.
Two improvements surfaced by the multi-agent review of yesterday's
plugin-sync work:
- Replace the free-form decision.reason string match in
hasProviderOwnedSkip with a stable PushDecisionSkipReason enum
('direction-skip' | 'no-baseline' | 'provider-changed'). A typo in
either side previously broke baseline preservation silently.
The human-readable reason is kept for log output only.
- Move deletedTagTitles off the deleteTag/deleteTags action payload
and into a new DeletedTagTitlesSidecarService (same pattern as
DeletedTaskIssueSidecarService). The action is persisted into the
exportable op-log; tag titles are user content (rule 9). The effect
uses LOCAL_ACTIONS so the sidecar is consumed only on the originating
client; remote replays see an empty sidecar and short-circuit.
The 4GB-VPS compose hardening dropped supersync's mem_limit from 1g to
768m, but the Dockerfile still set NODE_OPTIONS=--max-old-space-size=896.
V8 only GCs near the old-space cap, and RSS (old-space + new-space +
code cache + native heap + ArrayBuffers) can exceed that on snapshot or
import bursts — the cgroup would OOM-kill before V8 reclaimed memory.
Drop to 576 (≈75% of 768m) so V8 reclaims before the cgroup acts.
npm run wrapping leaves stdin in non-blocking mode while still reporting
isTTY=true, so fs.readSync(0, ...) threw EAGAIN and crashed `npm run version`
at the "Generate release notes via AI?" prompt. Read from /dev/tty (always
blocking) when available, and fall back to stdin with an Atomics.wait retry
on EAGAIN for platforms without /dev/tty.
Right-size container limits to fit a 4GB / 2-vCore VPS host without
OOM risk, and apply minimal best-practice hardening informed by
multi-agent review.
Container memory budget (sum 2816 MB, leaves ~1.2 GB host headroom):
- postgres: 1g → 1.5g (was at 74% under 199 WSS connections)
- supersync: 1g → 768m (uses 143 MiB steady state)
- dozzle: add 64m cap (was unbounded)
- uptime-kuma: add 192m cap (was unbounded)
- caddy: unchanged at 256m
Postgres tuning (1.5g cap, shm_size 256m to allow shared_buffers > 64m):
- shared_buffers=384MB, effective_cache_size=1GB
- work_mem=4MB, maintenance_work_mem=128MB
- max_connections=40 (Prisma pool is single-digit; 40 leaves headroom
for migrations + psql; 100 was an OOM landmine vs 1.5g cap)
- random_page_cost=1.1 (SSD)
- idle_in_transaction_session_timeout=300000 (5 min — catches leaked
Prisma idle-in-tx but stays well above migrate-deploy.sh
STEP_TIMEOUT=1800 for CONCURRENTLY index recovery)
- wal_compression=on, huge_pages=off
Other hardening:
- Standardize restart policy to unless-stopped across all services
- Add caddy healthcheck via admin API at localhost:2019; Caddyfile
comment warns future edits not to disable admin
- Add ulimits.nofile=65535 to supersync (Alpine soft default 1024 is
tight for 200+ WSS + reconnect bursts)
- Pin networks.internal.name=super-sync-server_internal so the
monitoring overlay can attach via external: true regardless of
COMPOSE_PROJECT_NAME
- Attach dozzle + uptime-kuma to internal network so Kuma can probe
http://supersync:1900/health by service name
Notes for future maintenance:
- oom_score_adj was tried and removed: Compose v2 silently ignores it
(verified on host via /proc/<pid>/oom_score_adj returning 0).
Would need a systemd post-up hack to apply — not worth it.
- The reconnect storm that motivated this work is fixed separately
on master in b404bf8a3.
Two browser tabs of the same origin share one clientId (IndexedDB-scoped).
The server kicks the older socket with close code 4009 ("Replaced by
newer connection"), but the client only short-circuited reconnect on
4003 and 4008 — every other code (including 4009) scheduled a 1s
reconnect. Two tabs would kick each other in a 1Hz ping-pong loop,
producing ~9700 "Replacing stale connection" events per 5 min against
~150 live sockets in production.
Handle 4009 the same way as 4008: skip auto-reconnect and let the next
sync trigger re-establish the WS. SuperSync has no wall-clock periodic
timer, so an idle tab stays disconnected until user activity — that's
correct for a backgrounded tab.
- Tighten recovery guard to the idempotent drop-then-create shape (require
both DROP and CREATE INDEX CONCURRENTLY). A bare CREATE (20260511000000) is
now refused by gate, deterministically — matching its documented fail-loud
intent instead of relying on the CREATE erroring by accident. Reconciles
README with the committed migration + migration-sql.spec.ts.
- Escape single quotes in print_manual_recovery so the printed escape-hatch
commands are copy-pasteable for SQL containing quotes (e.g. the full-state
WHERE op_type IN ('SYNC_IMPORT', ...) migration).
- Fix MIGRATE_LOG temp-file leak across retry attempts.
- Add an in-script per-step timeout (with_timeout) so the Dockerfile CMD /
helm initContainer paths (no outer timeout) can't hang forever on a blocked
concurrent build; 124 fails loudly.
- Harden parse_failing_migration: sentence-anchored P3009 parse + reject
names outside the migration charset (path-traversal defence).
- Gate also accepts SQLSTATE 25001 (stable Postgres contract) not just the
localizable English message.
- MAX_ATTEMPTS -> documented tight bound (6); fail_loudly wording fixed
(was contradictory for the non-CONCURRENTLY case).
- Tests: bare-CREATE refusal, P3009 decoy-token parser hardening; trim
migration-sql.spec.ts to the architectural-invariant subset (no hardcoded
names) per 'test behavior not implementation'.
Full server suite: 36 files, 727 passed / 5 skipped.
prisma migrate deploy wraps each migration in a transaction; CONCURRENTLY
index migrations fail it (P3018/25001) and later deploys then stick (P3009).
Recovery was duplicated and migration-name-hardcoded in the host deploy.sh
and the in-image migrate-deploy.sh. The host script self-updates only via a
best-effort git pull, so a stale host deploy.sh had no recovery branch for a
new CONCURRENTLY migration and failed the deploy (the reported incident).
- migrate-deploy.sh: single, name-agnostic recovery. Parses the failing
migration from Prisma's own output, gates on the txn-block/P3009 signature
AND the migration's own SQL containing INDEX CONCURRENTLY, runs that SQL
out-of-band statement-by-statement, and only marks it applied if every
statement succeeded; otherwise fails loudly with manual steps. Bounded
retry loop; aborts instead of looping on re-failure.
- deploy.sh: ~290 lines of hardcoded host-side recovery removed; now invokes
the in-image scripts/migrate-deploy.sh (always version-locked to
prisma/migrations in the pulled image) and keeps only timeout/exit policy.
- tests: drive the script end-to-end via a fake npx (P3018, stuck P3009,
non-CONCURRENTLY refusal, statement-failure, re-failure abort, genuine
error passthrough, multi-migration chain); migration-sql.spec.ts updated
to the new contract.
- prisma/migrations/README.md: authoring rules the recovery relies on.
Design: docs/plans/2026-05-15-generic-concurrently-migration-recovery-design.md
* docs(sync): add super sync server perf plan
* perf(sync): implement supersync server perf phases
* fix(sync): bracket auth cache invalidation
* fix(sync): avoid empty replay state stringify
* fix(sync): harden supersync batch uploads
* fix super sync review findings
* fix(sync): guard payload bytes backfill rollout
* perf(sync): speed up payload_bytes backfill and index its scan
Raise the backfill batch size (DEFAULT 5->500, MAX 25->1000) so a
100M-row operations table backfills in minutes rather than tens of
hours. Add a CONCURRENTLY partial index on (user_id, id) WHERE
payload_bytes = 0: it drains to empty post-backfill so the boot-time
backfill self-check and the BOOL_OR quota probe stop doing a full
sequential scan to prove absence, and it makes the backfill's per-user
keyset paging a true index seek. Wire the new concurrent-index
migration into both deploy scripts' P3018 recovery path. Add
migration-SQL guard tests for the ADD COLUMN (metadata-only fast path)
and the new partial index.
* fix(sync): bound auth cache invalidation map and bracket every delete
The auth verification cache's invalidationVersions map grew one entry
per lifetime-invalidated user with no eviction (unbounded heap on a
long-lived single replica). Cap it at the same 10k LRU bound as the
entries map, re-inserting the just-invalidated user at the MRU tail so
the CAS race protection still holds for the only window that matters
(one DB round trip). Bracket the passkey/magic-link registration
cleanup deletes with pre+post invalidate to match the documented
convention, and invalidate on verifyEmail so a freshly-verified user
isn't denied for up to the cache TTL.
* perf(sync): skip the redundant exact replay-state measurement
The delta accounting is a proven over-estimate of the serialized state
size, so when the running bound stays within the cap the true size is
too and the final exact JSON.stringify is provably redundant. Skip it
in that case (still measure-and-throw whenever the bound does not prove
safety). This collapses the common small/incremental replay back to
zero expensive full stringifications, matching the old per-op loop
instead of regressing it. Name the entity-key JSON overhead constant
and document that assertReplayStateSize's return value is load-bearing.
* refactor(sync): split processOperationBatch into pipeline stages
Extract the 297-line batch upload method into a thin orchestrator plus
six named single-responsibility stage helpers (validate+clamp, intra-
batch dedupe, classify existing duplicates, conflict-detect, reserve
seq + insert, full-state clock). Behavior-preserving: every stage
writes terminal rejections into the shared results array by index and
the two empty-set guards short-circuit exactly as before. Also share
the timestamp clamp, the duplicate-op SELECT, and the merged
full-state clock persistence between the batch and legacy paths so
they cannot silently diverge.
* test(sync): pin batch error-code divergence and aggregate-once
Strengthen the intra-batch duplicate test to assert same-id /
different-content yields DUPLICATE_OPERATION (deliberate divergence
from the legacy INVALID_OP_ID), and document the divergence in the
plan. Replace the single-full-state aggregate test with two
full-state ops + a spy asserting _aggregatePriorVectorClock runs
exactly once and last-write-wins — the old test could not catch a
per-op-aggregate regression. Add a makeOp fixture factory. Correct
the plan's overstated replay-stringification numbers.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: johannesjo <1456265+johannesjo@users.noreply.github.com>
Google Calendar time-block sync failed with 403 rateLimitExceeded. Root cause: every upsert did POST insert -> 409 duplicate -> PATCH (two writes plus a guaranteed 409 per sync), and a single user edit dispatched several actions that each issued a write to the same event within ~1s, bursting past Google's per-event write limit.
- google-calendar-provider: patch-first idempotent upsert (POST only on 404; recover a concurrent-create 409 via PATCH) + bounded exponential backoff on 403 rateLimitExceeded / 429, retried per network call. Follows Google's documented 409 remedy.
- caldav-calendar-provider: PUT is already an idempotent upsert; add the same bounded backoff on transient 429/503 from self-hosted servers. 404-swallow on delete unchanged.
- time-block-sync.effects: merge createOrUpdateOnSchedule$ + updateOnFieldChange$ into one upsertOnTaskChange$ that groups by task id and debounces (COALESCE_MS=1000) then switchMaps, re-reading the latest task post-debounce. One settled edit = one write. Still LOCAL_ACTIONS + action-driven (sync-correctness rules 1-2 preserved).
Adds vitest coverage for both plugins' timeBlock paths and the first effects spec for time-block sync.
The user-exportable log (Settings → Provide Log file) recorded user
content — task titles, notes, project/tag names — and was advertised as
safe to attach to public bug reports. Sources:
- actionLoggerReducer logged every action's full payload at VERBOSE,
with production LOG_LEVEL also VERBOSE → now logs action.type only;
LOG_LEVEL is a flat VERBOSE constant (drop the no-op ternary)
- PluginBridgeService._dispatchAction logged { payload: action } at
NORMAL (the real leak; PluginAPI.dispatchAction delegates here) and
PluginAPI register*/notify/openDialog/persistDataSynced/batchUpdate
+ the bridge notify path logged full cfg/data objects → type/id only
- removed a dormant commented root-reducer stub that would dump entire
NgRx state into the exportable log if uncommented
Safe id/count logs ({ taskId }, { projectId }, counts) kept per rule #9.
Regression specs: action-logger reducer (props + legacy payload), real
PluginBridge dispatch (not mocked), and PluginAPI content sites.
WebdavBaseProvider.clearAuthCredentials() blanked the user-typed
userName/password on a single recoverable AuthFailSPError /
MissingCredentialsSPError. The transient-retry tolerance in
sync-wrapper is SuperSync-only, so a one-off 401 on WebDAV/Nextcloud
irreversibly erased the stored connection settings with no undo.
A WebDAV password is a user-typed, often-irrecoverable secret, not a
refreshable OAuth token. Remove the WebDAV/Nextcloud override so
ProviderManager's existing guard makes the auth-error clear path a
no-op for them; Dropbox/SuperSync keep their token clearing. The
actionable re-configure snackbar still fires unchanged. Document the
optional-hook contract on SyncProviderBase.
Add regression tests for the provider hook (incl. asserting WebDAV
exposes no clearAuthCredentials) and a legacy WebDAV credential
migration repro harness (the migration itself was sound).
Fixes#7616. Related: #7619 (exported logs leak task content).
Collapse the sprawling, partly-stale docs/sync-and-op-log/ tree into a
small authoritative set and make the sync-correctness invariant
partly lint-enforced instead of convention-only.
Docs:
- Delete superseded/duplicate/provably-stale design, plan, and
background-research docs (quick-reference, the architecture-diagrams
monolith, the "Hybrid Manifest" docs describing code that does not
exist, completed long-term plans, LLM-synthesis analyses).
- Salvage load-bearing decision history into the surviving docs before
deletion: rejected-alternatives rationale -> operation-log-architecture
("Why this architecture"); vector-clock pruning incident history ->
vector-clocks.md; archive-payload optimization -> architecture E.7.
- Add contributor-sync-model.md as the single-invariant entry point
(one user intent = one op; replayed/remote ops must not re-trigger
effects), with a decision table mapping to the enforcing linters.
- Repoint external/internal cross-refs; add CONTRIBUTING.md + CLAUDE.md
pointers; record the migration in a dated docs/plans/ design doc.
Enforcement (new eslint-local-rules):
- no-actions-in-effects (error): effects must inject LOCAL_ACTIONS /
ALL_ACTIONS, never the raw @ngrx/effects Actions stream.
- no-multi-entity-effect (warn, heuristic): flags a literal returned
array of >=2 action-creator calls; docstring + valid-case specs pin
exactly which shapes are and are not detected.
- run-specs.js runner wired into `npm run lint` via test:lint-rules;
refuses to run under test-framework globals and counts RuleTester.run
invocations so a spec that asserts nothing fails instead of passing.
- Correct the ALL_ACTIONS JSDoc in local-actions.token.ts to match
reality (archive-operation-handler uses LOCAL_ACTIONS).
Reviewed via parallel multi-agent review; findings W1/W2/W4 and a
dangling doc anchor addressed.
