Two related improvements to snapshot/full-state handling:
- Persist the incoming op's vector clock as `user_sync_state.latest_full_state_vector_clock`
so downloads can skip the expensive aggregate scan when the snapshot clock is
still valid; falls back to the legacy aggregate when missing or invalid.
- Treat retried snapshot uploads (same opId) as idempotent successes instead of
409/DUPLICATE_OPERATION so a dropped response doesn't push clients into the
download-and-merge path. Per-namespace request dedup keeps ops and snapshot
caches isolated.
Add tests for previously untested vector clock pruning scenarios:
- Multi-preserve-ID pruning (two low-counter IDs both preserved)
- Overlapping/duplicate preserve IDs (Set deduplication)
- Preserve IDs missing from clock (silent skip, no crash)
- Snapshot vector clock aggregation + pruning integration
- Post-snapshot comparison correctness after pruning
- Server sanitizeVectorClock counter cap at 100M
- DoS cap at 2.5x MAX (reject, not prune)
- Invalid inputs (null, arrays, empty keys, long keys, negative/float values)
https://claude.ai/code/session_01GdsbKoo8eax394j2UyWUu1
Co-authored-by: Claude <noreply@anthropic.com>
Remove references to deleted docs/ai/ files, update stale comments about
protectedClientIds and pruning-aware comparison to reflect current REPLACE
semantics, fix MAX=30→20 heading, client_0..29→19 comment, use ?? over ||,
preserve lastSeq in server clean-slate path to prevent sequence reuse, and
add clarifying comments for mock limitations and validation guards.
- Add deterministic tie-breaking test for vector clock pruning with
equal counters (shared-schema)
- Remove last isLikelyPruningArtifact reference from entity versioning doc
At MAX=10, pruning triggered frequently enough (11+ unique client IDs from
reinstalls/new browsers) to require 4 defense layers compensating for
information loss: pruning-aware comparison, protected client IDs with
migration, isLikelyPruningArtifact heuristic, and same-client check.
At MAX=30, pruning almost never triggers (needs 31+ unique client IDs).
A 30-entry clock is ~500 bytes — negligible bandwidth. This allows removing
most defense layers while keeping two cheap backward-compat checks for old
10-entry pruned data still on servers.
Removed:
- Pruning-aware mode in compareVectorClocks (standard comparison now)
- Protected client IDs mechanism (storage, migration, preservation)
- selectProtectedClientIds function
- Clock normalization in SyncImportFilterService
Kept temporarily (backward compat with old 10-entry data):
- isLikelyPruningArtifact with LEGACY_MAX=10
- Same-client check (always mathematically correct)
Cover the full round-trip of limitVectorClockSize → compareVectorClocks →
isLikelyPruningArtifact with realistic MAX-entry clocks to guard against
regressions in the layered pruning heuristics.
Client-side pruning in OperationLogEffects dropped client IDs from
vector clocks (12→10 entries). Combined with the >= threshold in
compareVectorClocks' bothPossiblyPruned check, this caused false
CONCURRENT results and an infinite sync rejection loop.
- Remove limitVectorClockSize call from OperationLogEffects (server
already prunes AFTER comparison but BEFORE storage per arch doc #13)
- Change bothPossiblyPruned from >= to === MAX_VECTOR_CLOCK_SIZE
(a clock exceeding MAX was never pruned by limitVectorClockSize)
- Update test expectations and comments in both shared-schema and
client-side specs to reflect the corrected behavior
- Replace fragile VectorClockComparison[result] enum lookup with safe cast
- Return CONCURRENT instead of EQUAL when only one side has non-shared
keys in pruning-aware mode (safe direction: triggers LWW instead of
silent skip)
- Fix docs claiming clocks "reset" at MAX_SAFE_INTEGER (they throw)
- Add Logger.warn() in ValidationService when oversized vector clocks
are pruned server-side, improving observability for buggy clients
- Add tests for hasVectorClockChanges (previously zero coverage)
- Add symmetric LESS_THAN test for pruning-aware comparison mode
- Export MAX_LWW_REUPLOAD_RETRIES constant to eliminate magic number
coupling between sync-wrapper service and its test
- Add test verifying uploading client ID is preserved during server-side
clock pruning
- Add test for limitVectorClockSize when preserveClientIds exceeds MAX
Make hasVectorClockChanges pruning-aware by checking clock size before
logging missing keys — downgrade to verbose when pruning is likely,
warn when corruption is likely. Add return value assertion to LWW retry
exhaustion test, asymmetric pruning test cases, server-side pruning
tradeoff documentation, and fix stale "max 50" in docs.
When both vector clocks are at MAX size (pruning-aware mode) and shared
keys compare as equal, non-shared keys on both sides indicate genuinely
different causal histories. Returning EQUAL caused silent data loss by
skipping operations as duplicates. Returning CONCURRENT safely triggers
LWW conflict resolution instead.
Also fixes server snapshot pruning to preserve the requesting client's
ID and the snapshot author's client ID when limiting vector clock size.
Return CONCURRENT instead of EQUAL when two max-size clocks share no
keys (independent client populations). Fix misleading SyncStatus.InSync
return when LWW retry exhaustion leaves pending ops. Document known
limitation of size-based pruning heuristic. Add missing edge case tests.
When splitting one operation into misc and tasks operations:
- Both operations now get unique ID suffixes (_misc and _tasks)
- Skip logic verifies both conditions before skipping:
1. tasks config has migrated field (isConfirmBeforeDelete)
2. misc config has NO migrated fields remaining
This prevents:
- Potential operation ID conflicts in the sync log
- Data loss in partial migration scenarios
Adds test cases for partial migration and correct skip scenarios.
Create @sp/shared-schema package with pure TypeScript migration functions
that work in both Angular frontend and Node.js backend environments.
Package contents:
- schema-version.ts: Version constants (CURRENT=1, MAX_SKIP=3)
- migration.types.ts: OperationLike, SchemaMigration interfaces
- migrate.ts: Pure functions (migrateState, migrateOperation, etc.)
- migrations/index.ts: Empty migrations array (ready for first migration)
- 22 unit tests covering all migration scenarios
Backend changes:
- Add snapshot_schema_version column to user_sync_state table
- Migrate snapshots during generateSnapshot if outdated
- Migrate operations during replayOpsToState if outdated
- Drop operations that return null from migration (removed features)
Frontend changes:
- Refactor SchemaMigrationService to use @sp/shared-schema
- Re-export constants for backwards compatibility
- All 20 existing tests pass
This enables coordinated schema migrations across client and server,
ensuring old snapshots and operations can be upgraded when the state
structure changes.
Rollout strategy: Deploy backend first, then frontend.
