super-productivity/docs/sync-and-op-log/operation-rules.md

Operation Log: Design Rules & Guidelines

Last Updated: July 2026 Related: Operation Log Architecture

This document establishes the core rules and principles for designing the Operation Log store and defining new Operations. Adherence to these rules ensures data integrity, synchronization reliability, and system performance.

1. Store Design Rules

1.1 Append-Only Persistence

• Rule: Operation payloads in the ops table are append-only. Lifecycle metadata may advance in place as described in 1.2.
• Reasoning: History preservation is critical for event sourcing and conflict resolution.
• Exception: Operations can only be deleted by the Compaction Service, and only if they are older than the retention window, baked into the current snapshot (seq <= lastAppliedOpSeq), and either:
  1. Successfully synced and application-complete; or
  2. Terminally rejected.
• Never compact: Non-rejected pending, archive_pending, and failed remote work is retained regardless of age.

1.2 Immutable History

• Rule: Once an operation is written to SUP_OPS, its operation ID, payload, action, vector clock, source, and sequence MUST NOT be modified.
• Reasoning: Modifying history breaks the cryptographic chain (if implemented later) and confuses sync peers who have already received the operation.
• Lifecycle exception: Local bookkeeping metadata advances in place: syncedAt, rejectedAt, reducerRejectedAt, applicationStatus, retryCount, lastRetryAt, and error. These fields describe local delivery/application state; they do not rewrite the operation peers exchange.
• Correction: If an operation was incorrect, append a new compensating operation (e.g., an undo or correction op) rather than editing the old one.

1.3 Single Source of Truth

• Rule: The Operation Log (SUP_OPS) is the ultimate source of truth for the application state.
• Context: The state_cache and runtime NgRx store are projections derived from the log.
• Implication: If the runtime state disagrees with the log replay, the log wins.

1.4 Snapshot Mandate

• Rule: The store must maintain a valid state_cache (snapshot).
• Frequency: Snapshots must be updated based on configurable thresholds:
  • Operation count: After N operations (default: 500, configurable).
  • Time-based: After T minutes of inactivity following changes.
  • Size-based: When tail ops exceed S kilobytes.
  • Event-triggered: Immediately after significant events (large imports, sync completion).
• Recovery: The system must be able to rebuild the state entirely from Snapshot + Tail Ops.

2. Operation Design Rules

2.1 Granularity & Atomicity

• Rule: Operations should be atomic and focused on a single entity where possible.
• Good: UPDATE_TASK { id: "A", changes: { title: "New" } }
• Bad: UPDATE_ALL_TASKS { [ ... entire tasks array ... ] }
• Reasoning: Granular ops reduce conflict probability. Large "dump" ops cause massive conflicts during sync.
• Exception: SYNC_IMPORT and BACKUP_IMPORT are allowed to replace large chunks of state but must be treated as special "reset" events.

2.2 Idempotency

• Rule: Applying the same operation twice must be safe.
• Implementation:
  • Use explicit IDs (UUID v7) for creation. CREATE with an existing ID must be ignored (not merged or updated). If updates are needed, a separate UPDATE operation must follow.
  • DELETE on a missing entity should be a no-op.
  • UPDATE on a missing entity should be queued for retry (see 3.4 Dependency Awareness).

2.3 Serializable Payload

• Rule: Operation payloads must be Pure JSON.
• Forbidden:
  • Date objects (use timestamp numbers).
  • Functions or class instances.
  • undefined (use null or omit the key, depending on semantics).
  • Circular references.

2.4 Causality Tracking

• Rule: Every operation MUST carry a vectorClock.
• Purpose: To determine if the operation is concurrent with others or if it causally follows them.
• Responsibility: The OperationLogEffects (or equivalent creator) captures the clock at the moment of creation.

2.5 Schema Versioning

• Rule: Every operation MUST carry a schemaVersion.
• Purpose: To allow future versions of the app to migrate or interpret old operations correctly.
• Default: Use CURRENT_SCHEMA_VERSION from SchemaMigrationService at the time of creation.
• Compatibility barrier: Bump the schema version when an old reducer would silently misinterpret a new payload semantic — but a bump alone does NOT stop released clients: v17.0.0–v18.14.0 receivers apply ops up to schema 5 unmigrated, and at ≥ 6 block them while still advancing the server cursor (permanent silent skip). Only post-v18.14.0 receivers block newer ops safely. New payload semantics must therefore degrade gracefully on old clients; see the Bump Policy in operation-log-architecture.md §A.7.11.

2.6 Explicit Intent (OpType)

• Rule: Use specific OpTypes (CRT, UPD, DEL, MOV) rather than a generic CHANGE.
• Reasoning: Specific types allow for smarter conflict resolution and UI feedback (e.g., "Task was deleted remotely" vs "Task was moved").

2.7 Causal Automatic Repair

• Rule: A new REPAIR snapshot sent to SuperSync must carry a top-level repairBaseServerSeq equal to the server cursor included in the repaired state.
• Server acceptance: The server checks the base before quota cleanup and again while holding the user_sync_state.lastSeq row lock in the upload transaction. A mismatch returns REPAIR_STALE; it must not prune history or consume quota.
• Replay: Operations at or below an accepted causal repair's base are already represented by its snapshot and must not be replayed. Concurrent operations that landed after the repair remain valid and apply on top.
• Stale rebase: Download and apply the missing suffix while ignoring the stale local repair as a filtering boundary. Then atomically mark the stale repair rejected and append a new repair containing the resulting state, updated vector clock, and downloaded server cursor.
• Capability negotiation: A SuperSync client may upload causal repairs only after GET /api/sync/ops advertises capabilities.causalRepairSnapshots: true. This makes a new client fail closed against an older server.
• Legacy compatibility: A markerless repair from an older client may be retained non-destructively, but it is not a proven causal boundary. It must not authorize download fast-forward, snapshot-cache trust, or history pruning; concurrent prefix work is replayed after that legacy repair. Because the server cannot reconstruct that ordering from the markerless row alone, it must also exclude the row from restore points and fail closed if server-side snapshot replay crosses it.

3. Interaction & Safety Rules

3.1 Validation First

• Rule: Validate operation payloads before appending to the log.
• Checkpoint: Structural validation (required fields) happens at the boundary. Deep semantic validation happens during application/replay.
• Failure: Reject malformed operations immediately; do not corrupt the log.

3.2 Robust Replay

• Rule: The replay mechanism (Hydrator) MUST NOT CRASH on invalid operations.
• Behavior: If an operation fails to apply (e.g., referencing a missing parent):
  1. Log a warning.
  2. Skip the operation (or queue for retry).
  3. Continue replaying the rest.
  4. Trigger a REPAIR cycle at the end if needed.

3.3 Sync Isolation

• Rule: The OperationLogStore should not contain logic specific to a sync provider (Dropbox, WebDAV).
• Separation: The store manages persistence. The Sync Services manage transport.
• Interface: The store exposes getUnsynced(), markSynced(), markRejected() as generic methods.

3.4 Dependency Awareness

• Rule: Operations creating dependent entities (e.g., Subtask) must ensure the dependency (Parent Task) exists.
• Handling: If a parent is missing during sync, the child creation op should be buffered in a DependencyQueue until the parent arrives.
• Safeguards:
  • Cycle detection: Before queuing, verify the dependency graph is acyclic. Reject operations that would create circular dependencies.
  • Buffer limits: The queue must enforce a maximum depth (default: 1000 pending ops) and timeout (default: 5 minutes). Operations exceeding limits should be logged and dropped.
  • Retry policy: Queued operations should be retried after each batch of new operations is applied, with exponential backoff for repeated failures.

3.5 Deletion & Tombstones

Status (December 2025): Tombstones are DEFERRED. After comprehensive evaluation, the current event-sourced architecture provides sufficient safeguards without explicit tombstones. See todo.md Item 1 for the full evaluation.

• Current Implementation: Deletions use DELETE operations in the event log (immutable events, not destructive).
• Alternative Safeguards in Place:
  • Vector clocks detect concurrent delete+update conflicts; user resolution UI is presented.
  • Tag sanitization filters non-existent taskIds at reducer level.
  • Subtask cascading deletes include all child tasks.
  • Auto-repair removes orphaned references and creates REPAIR operations.
• When to Revisit:
  • If undo/restore functionality is needed.
  • If audit compliance requires explicit "entity deleted at time X" records.
  • If cross-version sync (A.7.11) reveals edge cases not handled by current safeguards.

3.6 Operation Batching

• Rule: Normal operations should be batched with reasonable limits.
• Limits:
  • Max batch size: 25 operations per batch for normal sync uploads.
  • Max payload size: 1 MB per batch to prevent timeout issues.
• Exception: SYNC_IMPORT, BACKUP_IMPORT, and REPAIR bypass these limits but must be clearly marked as bulk operations. Imports trigger immediate snapshot creation; only causal (base-marked) repairs may be cached as server snapshots.

4. Effect Rules

4.1 LOCAL_ACTIONS for Side Effects

• Rule: All NgRx effects that perform side effects MUST use inject(LOCAL_ACTIONS) instead of inject(Actions).
• Reasoning: Effects should NEVER run for remote sync operations. Side effects (snackbars, API calls, sounds) happen exactly once on the originating client.
• Exception: Effects that only dispatch state-modifying actions (not side effects) may use regular Actions.

Example:

@Injectable()
export class MyEffects {
  private _actions$ = inject(LOCAL_ACTIONS); // ✅ Correct for side effects

  showSnack$ = createEffect(
    () =>
      this._actions$.pipe(
        ofType(completeTask),
        tap(() => this.snackService.show('Task completed!')),
      ),
    { dispatch: false },
  );
}

4.2 Avoid Selector-Based Effects That Dispatch Actions

• Rule: Prefer action-based effects (this._actions$.pipe(ofType(...))) over selector-based effects (this._store$.select(...)).
• Reasoning: Selector-based effects fire whenever the store changes, including during hydration and sync replay, bypassing LOCAL_ACTIONS filtering.
• Workaround: If you must use a selector-based effect that dispatches actions, guard it with HydrationStateService.isApplyingRemoteOps().

4.3 Archive Side Effects

• Rule: Archive operations (writing to IndexedDB) are handled by ArchiveOperationHandler, NOT by regular effects.
• Local operations: ArchiveOperationHandlerEffects routes through ArchiveOperationHandler (via LOCAL_ACTIONS)
• Remote operations: OperationApplierService calls ArchiveOperationHandler directly after dispatch

5. Multi-Entity Operation Rules

5.1 Use Meta-Reducers for Atomic Changes

• Rule: When one action affects multiple entities, use meta-reducers instead of effects.
• Reasoning: Meta-reducers ensure all changes happen in a single reducer pass, creating one operation in the sync log and preventing partial sync.
• Example: Deleting a tag also removes it from tasks → handled in tagSharedMetaReducer, not in an effect.

5.2 Capture Multi-Entity Changes

• Rule: The OperationCaptureService automatically captures all entity changes from a single action.
• Implementation: The operation-capture.meta-reducer calls OperationCaptureService.incrementPending() with the action; the persist effect computes the changes via OperationCaptureService.extractEntityChanges() and decrements the pending counter in a finally.
• Result: Single operation with entityChanges[] array containing all affected entities.

6. Configuration Constants

See operation-log.const.ts for all configurable values:

Constant	Value	Description
COMPACTION_TRIGGER	500 ops	Operations before automatic compaction
COMPACTION_RETENTION_MS	7 days	Synced ops older than this may be deleted
EMERGENCY_COMPACTION_RETENTION_MS	1 day	Shorter retention for quota exceeded
MAX_COMPACTION_FAILURES	3	Failures before user notification
MAX_DOWNLOAD_OPS_IN_MEMORY	50,000	Bounds memory during API download
REMOTE_OP_FILE_RETENTION_MS	14 days	Server-side operation file retention

7. Quick Reference Checklist

When adding a new persistent action:

• Add meta.isPersistent: true to the action
• Add meta.entityType and meta.opType
• Ensure related entity changes are in a meta-reducer (not effects)
• Effects with side effects use LOCAL_ACTIONS
• Archive operations route through ArchiveOperationHandler
• Add action to ACTION_AFFECTED_ENTITIES if multi-entity