mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-07-29 18:50:07 +00:00
docs(supersync): stop overclaiming GIN is forced; fix the batch-lookup doc
Three misleading safety claims, all in text a future contributor would rely on: - vector-clocks.md still documented the batch lookup as the mutually exclusive 'CASE WHEN cardinality(entity_ids) > 0 THEN entity_ids ELSE ARRAY[entity_id]'. Production unions the two columns instead, and the exclusive form is the #8334 bug: it drops a scalar that is not a member of its own entity_ids, so a later concurrent write to that entity is wrongly accepted - silent data loss. The section above it was rewritten last commit; this one was missed. - 'The CTE wins STRUCTURALLY, not on cost' overstated. Removing the competing btree is structural; GIN being chosen is not. A sequential scan is always available and wins for an unselective id - reproducible on PG16, and exactly what a globally shared entity id produces. Reframed as a measured outcome, since a confident comment asserting what the planner will do is what preceded the outage. - The plan spec still said production receives 'a CUSTOM plan nobody receives'. It receives custom plans for roughly the first 5 executions before settling generic, and this file covers only the generic mode.
This commit is contained in:
parent
f483445d9f
commit
27d1baa9b7
3 changed files with 41 additions and 14 deletions
|
|
@ -261,7 +261,17 @@ The lookups in `conflict.ts` match a requested entity as the scalar `entity_id`
|
|||
> `packages/super-sync-server/tests/conflict-entity-lookup-plan.pglite.spec.ts` and the note
|
||||
> at `detectConflictForEntity` in `packages/super-sync-server/src/sync/conflict.ts`.
|
||||
|
||||
- `detectConflictForEntities` / `prefetchLatestEntityOpsForBatch` (batch) — raw SQL unnesting `CASE WHEN cardinality(entity_ids) > 0 THEN entity_ids ELSE ARRAY[entity_id] END`, with a `entity_ids && ... OR entity_id = ANY(...)` prefilter so the `GIN(entity_ids)` index (migration `20260613000001`) and the existing `entity_id` btree stay usable.
|
||||
- `detectConflictForEntities` / `prefetchLatestEntityOpsForBatch` (batch) — raw SQL unnesting the **union** of both columns, `entity_ids || CASE WHEN entity_id IS NULL THEN '{}' ELSE ARRAY[entity_id] END`, deduped by `DISTINCT ON`, with an `entity_ids && ... OR entity_id = ANY(...)` prefilter so the `GIN(entity_ids)` index (migration `20260613000001`) and the existing `entity_id` btree stay usable.
|
||||
|
||||
> ⚠️ It must be a **union**, not the mutually exclusive
|
||||
> `CASE WHEN cardinality(entity_ids) > 0 THEN entity_ids ELSE ARRAY[entity_id] END`
|
||||
> this section used to document. The server does **not** enforce
|
||||
> `entity_id === entityIds[0]`, so a multi-entity op can carry a scalar that is not a
|
||||
> member of its own `entity_ids` (see `getStoredEntityIds`). The exclusive form drops
|
||||
> that scalar whenever the array is non-empty, making the entity invisible to conflict
|
||||
> lookups — a later concurrent write to it is wrongly accepted, which is **silent data
|
||||
> loss**. That was the #8334 bug; the divergent-scalar case is the decisive test in
|
||||
> `tests/integration/conflict-detection-sql.integration.spec.ts`.
|
||||
|
||||
**Forward-only by design:** rows written before migration `20260613000000` have an empty `entity_ids` array and fall back to the scalar `entity_id` (= first entity) in the `CASE` expression / scalar branch above — there is no `UPDATE` backfill. Entities 2..n of already-stored multi-entity ops were never persisted and are unrecoverable, so they remain invisible to conflict detection until that entity gets a fresh write. This residual is bounded: client-side LWW is unaffected (the client persists the full op and `VectorClockService.getEntityFrontier()` fans each op out to **every** entity), and the server only builds an authoritative snapshot from non-encrypted ops (`replayOpsToState()` throws on encrypted ops), so the pre-fix gap could only surface a stale value to a fresh client on non-encrypted self-hosted servers.
|
||||
|
||||
|
|
|
|||
|
|
@ -243,10 +243,19 @@ export const detectConflictForEntity = async (
|
|||
|
||||
// Array branch — raw SQL, and the MATERIALIZED CTE is load-bearing.
|
||||
//
|
||||
// The CTE wins STRUCTURALLY, not on cost: inside it the only predicate is
|
||||
// `entity_ids @> ...`, so the composite btree has no usable leading column and GIN is
|
||||
// the only index available at ANY cost estimate. MATERIALIZED is what stops the outer
|
||||
// user_id / entity_type predicates being pushed down, which would hand the btree back.
|
||||
// What the CTE removes structurally is the COMPETING BTREE: inside it the only
|
||||
// predicate is `entity_ids @> ...`, so the composite btree has no usable leading
|
||||
// column and GIN is the only INDEX available at any cost estimate. MATERIALIZED is
|
||||
// what stops the outer user_id / entity_type predicates being pushed down, which
|
||||
// would hand the btree back.
|
||||
//
|
||||
// That is NOT a guarantee that GIN is chosen. A sequential scan is always still
|
||||
// available, and wins when the probed id is unselective — both plans have been
|
||||
// reproduced on PG16 depending on row shape (see the cross-tenant note below, where a
|
||||
// shared id produces a Seq Scan). GIN winning on production-shaped data is a MEASURED
|
||||
// outcome, not a structural one. Re-measure after any change rather than trusting
|
||||
// this paragraph; a confidently-worded comment asserting what the planner "will" do
|
||||
// is what preceded the outage.
|
||||
// Every simpler form reads the whole (user_id, entity_type) slice instead: the
|
||||
// array-only `findFirst` + `orderBy` (the outage), Prisma's `aggregate({ _max })`, and
|
||||
// the flat `SELECT MAX(server_seq) ... AND @>`.
|
||||
|
|
|
|||
|
|
@ -19,9 +19,13 @@ import type { Operation } from '../src/sync/sync.types';
|
|||
* fail here instead of passing against a stale copy.
|
||||
*
|
||||
* MEASURE WITH `force_generic_plan`, NEVER WITH LITERALS. Prisma sends parameterized
|
||||
* prepared statements, so production runs a GENERIC plan that cannot see parameter
|
||||
* values; EXPLAIN with literal constants yields a CUSTOM plan nobody receives. This
|
||||
* file once tested with literals and that blind spot passed two designs that were
|
||||
* prepared statements, and under the default `auto` Postgres builds CUSTOM plans for the
|
||||
* first ~5 executions before settling on a GENERIC one that cannot see parameter values.
|
||||
* A hot path lives in that steady state, so generic is the mode that matters — but note
|
||||
* production does receive custom plans too, and this file does not cover them (a
|
||||
* custom-plan-only regression is possible; see the rejected `server_seq >` narrowing in
|
||||
* conflict.ts). `EXPLAIN` with literal constants is a third thing again, and is the trap:
|
||||
* this file once tested that way and the blind spot passed two designs that were
|
||||
* catastrophic in production. EVERYTHING here — including the shim — goes through
|
||||
* explainGeneric. If you add a shape, use explainGeneric.
|
||||
*
|
||||
|
|
@ -420,12 +424,16 @@ describe('detectConflictForEntity does not scan the history (PGlite)', () => {
|
|||
expect(result.hasConflict).toBe(false);
|
||||
expectWithinBudget(stats);
|
||||
|
||||
// Structural, and on the REAL template rather than a copy of it: inside the CTE the
|
||||
// only predicate is `entity_ids @> ...`, so the composite btree has no usable
|
||||
// leading column and GIN is the only index available AT ANY COST ESTIMATE. That is
|
||||
// why the shape survives generic planning. Every regression form — inlining the
|
||||
// CTE, flattening it, reinstating the OR — reaches the btree instead, so this
|
||||
// catches them structurally even if a future seed stops blowing the budget.
|
||||
// Asserted on the REAL template rather than a copy of it. Inside the CTE the only
|
||||
// predicate is `entity_ids @> ...`, so the composite btree has no usable leading
|
||||
// column and GIN is the only INDEX available at any cost estimate — that much is
|
||||
// structural, and it is why every regression form (inlining the CTE, flattening it,
|
||||
// reinstating the OR) reaches the btree instead and is caught here even if a future
|
||||
// seed stops blowing the budget.
|
||||
//
|
||||
// It does NOT prove GIN is forced: a sequential scan remains available at any time
|
||||
// and wins for an unselective id (a globally shared entity id does exactly that).
|
||||
// This pins the MEASURED plan for this seed, not a guarantee.
|
||||
expect(stats.rawSql).toHaveLength(1);
|
||||
const arrayBranchPlan = stats.nodes[stats.sql.indexOf(stats.rawSql[0])];
|
||||
expect(arrayBranchPlan).toContain('operations_entity_ids_gin');
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue