docs(supersync): stop overclaiming GIN is forced; fix the batch-lookup doc

Three misleading safety claims, all in text a future contributor would rely on:

- vector-clocks.md still documented the batch lookup as the mutually exclusive
  'CASE WHEN cardinality(entity_ids) > 0 THEN entity_ids ELSE ARRAY[entity_id]'.
  Production unions the two columns instead, and the exclusive form is the
  #8334 bug: it drops a scalar that is not a member of its own entity_ids, so
  a later concurrent write to that entity is wrongly accepted - silent data
  loss. The section above it was rewritten last commit; this one was missed.

- 'The CTE wins STRUCTURALLY, not on cost' overstated. Removing the competing
  btree is structural; GIN being chosen is not. A sequential scan is always
  available and wins for an unselective id - reproducible on PG16, and exactly
  what a globally shared entity id produces. Reframed as a measured outcome,
  since a confident comment asserting what the planner will do is what
  preceded the outage.

- The plan spec still said production receives 'a CUSTOM plan nobody
  receives'. It receives custom plans for roughly the first 5 executions
  before settling generic, and this file covers only the generic mode.
This commit is contained in:
Johannes Millan 2026-07-20 17:12:01 +02:00
parent f483445d9f
commit 27d1baa9b7
3 changed files with 41 additions and 14 deletions

View file

@ -261,7 +261,17 @@ The lookups in `conflict.ts` match a requested entity as the scalar `entity_id`
> `packages/super-sync-server/tests/conflict-entity-lookup-plan.pglite.spec.ts` and the note
> at `detectConflictForEntity` in `packages/super-sync-server/src/sync/conflict.ts`.
- `detectConflictForEntities` / `prefetchLatestEntityOpsForBatch` (batch) — raw SQL unnesting `CASE WHEN cardinality(entity_ids) > 0 THEN entity_ids ELSE ARRAY[entity_id] END`, with a `entity_ids && ... OR entity_id = ANY(...)` prefilter so the `GIN(entity_ids)` index (migration `20260613000001`) and the existing `entity_id` btree stay usable.
- `detectConflictForEntities` / `prefetchLatestEntityOpsForBatch` (batch) — raw SQL unnesting the **union** of both columns, `entity_ids || CASE WHEN entity_id IS NULL THEN '{}' ELSE ARRAY[entity_id] END`, deduped by `DISTINCT ON`, with an `entity_ids && ... OR entity_id = ANY(...)` prefilter so the `GIN(entity_ids)` index (migration `20260613000001`) and the existing `entity_id` btree stay usable.
> ⚠️ It must be a **union**, not the mutually exclusive
> `CASE WHEN cardinality(entity_ids) > 0 THEN entity_ids ELSE ARRAY[entity_id] END`
> this section used to document. The server does **not** enforce
> `entity_id === entityIds[0]`, so a multi-entity op can carry a scalar that is not a
> member of its own `entity_ids` (see `getStoredEntityIds`). The exclusive form drops
> that scalar whenever the array is non-empty, making the entity invisible to conflict
> lookups — a later concurrent write to it is wrongly accepted, which is **silent data
> loss**. That was the #8334 bug; the divergent-scalar case is the decisive test in
> `tests/integration/conflict-detection-sql.integration.spec.ts`.
**Forward-only by design:** rows written before migration `20260613000000` have an empty `entity_ids` array and fall back to the scalar `entity_id` (= first entity) in the `CASE` expression / scalar branch above — there is no `UPDATE` backfill. Entities 2..n of already-stored multi-entity ops were never persisted and are unrecoverable, so they remain invisible to conflict detection until that entity gets a fresh write. This residual is bounded: client-side LWW is unaffected (the client persists the full op and `VectorClockService.getEntityFrontier()` fans each op out to **every** entity), and the server only builds an authoritative snapshot from non-encrypted ops (`replayOpsToState()` throws on encrypted ops), so the pre-fix gap could only surface a stale value to a fresh client on non-encrypted self-hosted servers.

View file

@ -243,10 +243,19 @@ export const detectConflictForEntity = async (
// Array branch — raw SQL, and the MATERIALIZED CTE is load-bearing.
//
// The CTE wins STRUCTURALLY, not on cost: inside it the only predicate is
// `entity_ids @> ...`, so the composite btree has no usable leading column and GIN is
// the only index available at ANY cost estimate. MATERIALIZED is what stops the outer
// user_id / entity_type predicates being pushed down, which would hand the btree back.
// What the CTE removes structurally is the COMPETING BTREE: inside it the only
// predicate is `entity_ids @> ...`, so the composite btree has no usable leading
// column and GIN is the only INDEX available at any cost estimate. MATERIALIZED is
// what stops the outer user_id / entity_type predicates being pushed down, which
// would hand the btree back.
//
// That is NOT a guarantee that GIN is chosen. A sequential scan is always still
// available, and wins when the probed id is unselective — both plans have been
// reproduced on PG16 depending on row shape (see the cross-tenant note below, where a
// shared id produces a Seq Scan). GIN winning on production-shaped data is a MEASURED
// outcome, not a structural one. Re-measure after any change rather than trusting
// this paragraph; a confidently-worded comment asserting what the planner "will" do
// is what preceded the outage.
// Every simpler form reads the whole (user_id, entity_type) slice instead: the
// array-only `findFirst` + `orderBy` (the outage), Prisma's `aggregate({ _max })`, and
// the flat `SELECT MAX(server_seq) ... AND @>`.

View file

@ -19,9 +19,13 @@ import type { Operation } from '../src/sync/sync.types';
* fail here instead of passing against a stale copy.
*
* MEASURE WITH `force_generic_plan`, NEVER WITH LITERALS. Prisma sends parameterized
* prepared statements, so production runs a GENERIC plan that cannot see parameter
* values; EXPLAIN with literal constants yields a CUSTOM plan nobody receives. This
* file once tested with literals and that blind spot passed two designs that were
* prepared statements, and under the default `auto` Postgres builds CUSTOM plans for the
* first ~5 executions before settling on a GENERIC one that cannot see parameter values.
* A hot path lives in that steady state, so generic is the mode that matters but note
* production does receive custom plans too, and this file does not cover them (a
* custom-plan-only regression is possible; see the rejected `server_seq >` narrowing in
* conflict.ts). `EXPLAIN` with literal constants is a third thing again, and is the trap:
* this file once tested that way and the blind spot passed two designs that were
* catastrophic in production. EVERYTHING here including the shim goes through
* explainGeneric. If you add a shape, use explainGeneric.
*
@ -420,12 +424,16 @@ describe('detectConflictForEntity does not scan the history (PGlite)', () => {
expect(result.hasConflict).toBe(false);
expectWithinBudget(stats);
// Structural, and on the REAL template rather than a copy of it: inside the CTE the
// only predicate is `entity_ids @> ...`, so the composite btree has no usable
// leading column and GIN is the only index available AT ANY COST ESTIMATE. That is
// why the shape survives generic planning. Every regression form — inlining the
// CTE, flattening it, reinstating the OR — reaches the btree instead, so this
// catches them structurally even if a future seed stops blowing the budget.
// Asserted on the REAL template rather than a copy of it. Inside the CTE the only
// predicate is `entity_ids @> ...`, so the composite btree has no usable leading
// column and GIN is the only INDEX available at any cost estimate — that much is
// structural, and it is why every regression form (inlining the CTE, flattening it,
// reinstating the OR) reaches the btree instead and is caught here even if a future
// seed stops blowing the budget.
//
// It does NOT prove GIN is forced: a sequential scan remains available at any time
// and wins for an unselective id (a globally shared entity id does exactly that).
// This pins the MEASURED plan for this seed, not a guarantee.
expect(stats.rawSql).toHaveLength(1);
const arrayBranchPlan = stats.nodes[stats.sql.indexOf(stats.rawSql[0])];
expect(arrayBranchPlan).toContain('operations_entity_ids_gin');