mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-01-23 02:36:05 +00:00
feat(operation-log): add data validation and repair system
Integrate PFAPI's validation/repair with operation logs to prevent corruption and automatically recover from invalid states. Key features: - REPAIR operation type with full repaired state + summary - 4 validation checkpoints (A: before write, B: after snapshot load, C: after replay, D: after sync) - Payload validation before IndexedDB write (lenient structural checks) - Full Typia + cross-model validation at state checkpoints - User notification when auto-repair occurs - Infinite loop prevention during repair New files: - validate-state.service.ts: wraps PFAPI validation + repair - validate-operation-payload.ts: checkpoint A payload validation - repair-operation.service.ts: REPAIR operation creation
This commit is contained in:
Johannes Millan
parent
ae854c64f2
commit
d22fbe28b2
11 changed files with 1299 additions and 38 deletions
|
|
@ -1,22 +1,23 @@
|
|||
# Operation Log Architecture
|
||||
|
||||
**Status:** Parts A, B, C Implemented
|
||||
**Status:** Parts A, B, C, D Implemented
|
||||
**Branch:** `feat/operation-logs`
|
||||
**Last Updated:** December 3, 2025 (hydration optimizations, isSyncInProgress check)
|
||||
**Last Updated:** December 3, 2025 (validation & repair integration)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
The Operation Log serves **three distinct purposes**:
|
||||
The Operation Log serves **four distinct purposes**:
|
||||
|
||||
| Purpose | Description | Status |
|
||||
| ------------------------- | --------------------------------------------- | ----------- |
|
||||
| **A. Local Persistence** | Fast writes, crash recovery, event sourcing | Complete ✅ |
|
||||
| **B. Legacy Sync Bridge** | Vector clock updates for PFAPI sync detection | Complete ✅ |
|
||||
| **C. Server Sync** | Upload/download individual operations | Complete ✅ |
|
||||
| Purpose | Description | Status |
|
||||
| -------------------------- | --------------------------------------------- | ----------- |
|
||||
| **A. Local Persistence** | Fast writes, crash recovery, event sourcing | Complete ✅ |
|
||||
| **B. Legacy Sync Bridge** | Vector clock updates for PFAPI sync detection | Complete ✅ |
|
||||
| **C. Server Sync** | Upload/download individual operations | Complete ✅ |
|
||||
| **D. Validation & Repair** | Prevent corruption, auto-repair invalid state | Complete ✅ |
|
||||
|
||||
This document is structured around these three purposes. Most complexity lives in **Part A** (local persistence). **Part B** is a thin bridge to PFAPI. **Part C** handles operation-based sync with servers.
|
||||
This document is structured around these four purposes. Most complexity lives in **Part A** (local persistence). **Part B** is a thin bridge to PFAPI. **Part C** handles operation-based sync with servers. **Part D** integrates validation and automatic repair.
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
|
|
@ -140,7 +141,15 @@ interface Operation {
|
|||
schemaVersion: number; // For migrations
|
||||
}
|
||||
|
||||
type OpType = 'CRT' | 'UPD' | 'DEL' | 'MOV' | 'BATCH' | 'SYNC_IMPORT' | 'BACKUP_IMPORT';
|
||||
type OpType =
|
||||
| 'CRT'
|
||||
| 'UPD'
|
||||
| 'DEL'
|
||||
| 'MOV'
|
||||
| 'BATCH'
|
||||
| 'SYNC_IMPORT'
|
||||
| 'BACKUP_IMPORT'
|
||||
| 'REPAIR';
|
||||
```
|
||||
|
||||
### Persistent Action Pattern
|
||||
|
|
@ -746,6 +755,191 @@ interface OperationDependency {
|
|||
|
||||
---
|
||||
|
||||
# Part D: Data Validation & Repair
|
||||
|
||||
The operation log integrates with PFAPI's validation and repair system to prevent data corruption and automatically recover from invalid states.
|
||||
|
||||
## D.1 Validation Architecture
|
||||
|
||||
Four validation checkpoints ensure data integrity throughout the operation lifecycle:
|
||||
|
||||
| Checkpoint | Location | When | Action on Failure |
|
||||
| ---------- | ----------------------------------- | ------------------------- | ------------------------------------------ |
|
||||
| **A** | `operation-log.effects.ts` | Before IndexedDB write | Reject operation, log error, show snackbar |
|
||||
| **B** | `operation-log-hydrator.service.ts` | After loading snapshot | Attempt repair, create REPAIR op |
|
||||
| **C** | `operation-log-hydrator.service.ts` | After replaying tail ops | Attempt repair, create REPAIR op |
|
||||
| **D** | `operation-log-sync.service.ts` | After applying remote ops | Attempt repair, create REPAIR op |
|
||||
|
||||
## D.2 REPAIR Operation Type
|
||||
|
||||
When validation fails at checkpoints B, C, or D, the system attempts automatic repair using PFAPI's `dataRepair()` function. If repair succeeds, a REPAIR operation is created:
|
||||
|
||||
```typescript
|
||||
enum OpType {
|
||||
// ... existing types
|
||||
Repair = 'REPAIR', // Auto-repair operation with full repaired state
|
||||
}
|
||||
|
||||
interface RepairPayload {
|
||||
appDataComplete: AppDataCompleteNew; // Full repaired state
|
||||
repairSummary: RepairSummary; // What was fixed
|
||||
}
|
||||
|
||||
interface RepairSummary {
|
||||
entityStateFixed: number; // Fixed ids/entities array sync
|
||||
orphanedEntitiesRestored: number; // Tasks restored from archive
|
||||
invalidReferencesRemoved: number; // Non-existent project/tag IDs removed
|
||||
relationshipsFixed: number; // Project/tag ID consistency
|
||||
structureRepaired: number; // Menu tree, inbox project creation
|
||||
typeErrorsFixed: number; // Typia errors auto-fixed
|
||||
}
|
||||
```
|
||||
|
||||
### REPAIR Operation Behavior
|
||||
|
||||
- **During replay**: REPAIR operations load state directly (like SyncImport), skipping prior operations
|
||||
- **User notification**: Shows snackbar with count of issues fixed
|
||||
- **Audit trail**: REPAIR operations are visible in the operation log for debugging
|
||||
|
||||
## D.3 Checkpoint A: Payload Validation
|
||||
|
||||
Before writing to IndexedDB, operation payloads are validated in `validate-operation-payload.ts`:
|
||||
|
||||
```typescript
|
||||
validateOperationPayload(op: Operation): PayloadValidationResult {
|
||||
// 1. Structural validation - payload must be object
|
||||
// 2. OpType-specific validation:
|
||||
// - CREATE: entity with valid 'id' field required
|
||||
// - UPDATE: id + changes, or entity with id required
|
||||
// - DELETE: entityId/entityIds required
|
||||
// - MOVE: ids array required
|
||||
// - BATCH: non-empty payload required
|
||||
// - SYNC_IMPORT/BACKUP_IMPORT: appDataComplete structure required
|
||||
// - REPAIR: skip (internally generated)
|
||||
}
|
||||
```
|
||||
|
||||
This validation is **intentionally lenient** - it checks structural requirements rather than deep entity validation. Full Typia validation happens at state checkpoints.
|
||||
|
||||
## D.4 Checkpoints B & C: Hydration Validation
|
||||
|
||||
During hydration, state is validated at two points:
|
||||
|
||||
```
|
||||
App Startup
|
||||
│
|
||||
▼
|
||||
Load snapshot from state_cache
|
||||
│
|
||||
├──► CHECKPOINT B: Validate snapshot
|
||||
│ │
|
||||
│ └──► If invalid: repair + create REPAIR op
|
||||
│
|
||||
▼
|
||||
Dispatch loadAllData(snapshot)
|
||||
│
|
||||
▼
|
||||
Replay tail operations
|
||||
│
|
||||
└──► CHECKPOINT C: Validate current state
|
||||
│
|
||||
└──► If invalid: repair + create REPAIR op + dispatch repaired state
|
||||
```
|
||||
|
||||
### Implementation
|
||||
|
||||
```typescript
|
||||
// In operation-log-hydrator.service.ts
|
||||
private async _validateAndRepairState(state: AppDataCompleteNew): Promise<AppDataCompleteNew> {
|
||||
if (this._isRepairInProgress) return state; // Prevent infinite loops
|
||||
|
||||
const result = this.validateStateService.validateAndRepair(state);
|
||||
if (!result.wasRepaired) return state;
|
||||
|
||||
this._isRepairInProgress = true;
|
||||
try {
|
||||
await this.repairOperationService.createRepairOperation(
|
||||
result.repairedState,
|
||||
result.repairSummary,
|
||||
);
|
||||
return result.repairedState;
|
||||
} finally {
|
||||
this._isRepairInProgress = false;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## D.5 Checkpoint D: Post-Sync Validation
|
||||
|
||||
After applying remote operations, state is validated:
|
||||
|
||||
- In `operation-log-sync.service.ts` - after applying non-conflicting ops (when no conflicts)
|
||||
- In `conflict-resolution.service.ts` - after resolving all conflicts
|
||||
|
||||
This catches:
|
||||
|
||||
- State drift from remote operations
|
||||
- Corruption introduced during sync
|
||||
- Invalid operations from other clients
|
||||
|
||||
## D.6 ValidateStateService
|
||||
|
||||
Wraps PFAPI's validation and repair functionality:
|
||||
|
||||
```typescript
|
||||
@Injectable({ providedIn: 'root' })
|
||||
export class ValidateStateService {
|
||||
validateState(state: AppDataCompleteNew): StateValidationResult {
|
||||
// 1. Run Typia schema validation
|
||||
const typiaResult = validateAllData(state);
|
||||
|
||||
// 2. Run cross-model relationship validation
|
||||
const isRelatedValid = isRelatedModelDataValid(state);
|
||||
|
||||
return { isValid, typiaErrors, crossModelError };
|
||||
}
|
||||
|
||||
validateAndRepair(state: AppDataCompleteNew): ValidateAndRepairResult {
|
||||
// 1. Validate
|
||||
// 2. If invalid: run dataRepair()
|
||||
// 3. Re-validate repaired state
|
||||
// 4. Return repaired state + summary
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## D.7 RepairOperationService
|
||||
|
||||
Creates REPAIR operations and notifies the user:
|
||||
|
||||
```typescript
|
||||
@Injectable({ providedIn: 'root' })
|
||||
export class RepairOperationService {
|
||||
async createRepairOperation(
|
||||
repairedState: AppDataCompleteNew,
|
||||
repairSummary: RepairSummary,
|
||||
): Promise<void> {
|
||||
// 1. Create REPAIR operation with repaired state + summary
|
||||
// 2. Append to operation log
|
||||
// 3. Save state cache snapshot
|
||||
// 4. Show notification to user
|
||||
}
|
||||
|
||||
static createEmptyRepairSummary(): RepairSummary {
|
||||
return {
|
||||
entityStateFixed: 0,
|
||||
orphanedEntitiesRestored: 0,
|
||||
invalidReferencesRemoved: 0,
|
||||
relationshipsFixed: 0,
|
||||
structureRepaired: 0,
|
||||
typeErrorsFixed: 0,
|
||||
};
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Implementation Status
|
||||
|
||||
## Complete ✅
|
||||
|
|
@ -773,6 +967,13 @@ interface OperationDependency {
|
|||
- **Rejected operation tracking** (`rejectedAt` field, excluded from sync)
|
||||
- **Skip META_MODEL update during sync** (prevents lock errors when user makes changes during sync)
|
||||
- **Hydration optimizations** (skip replay for SyncImport, save snapshot after >10 ops replayed)
|
||||
- **Payload validation at write** (Checkpoint A - structural validation before IndexedDB write)
|
||||
- **State validation during hydration** (Checkpoints B & C - Typia + cross-model validation)
|
||||
- **Post-sync validation** (Checkpoint D - validation after applying remote ops)
|
||||
- **REPAIR operation type** (auto-repair with full state + repair summary)
|
||||
- **ValidateStateService** (wraps PFAPI validation + repair)
|
||||
- **RepairOperationService** (creates REPAIR ops, user notification)
|
||||
- **User notification on repair** (snackbar with issue count)
|
||||
|
||||
## Future Enhancements 🔮
|
||||
|
||||
|
|
@ -803,7 +1004,10 @@ src/app/core/persistence/operation-log/
|
|||
├── multi-tab-coordinator.service.ts # BroadcastChannel coordination
|
||||
├── schema-migration.service.ts # State schema migrations
|
||||
├── dependency-resolver.service.ts # Extract/check operation dependencies
|
||||
└── conflict-resolution.service.ts # Conflict UI presentation
|
||||
├── conflict-resolution.service.ts # Conflict UI presentation
|
||||
├── validate-state.service.ts # Typia + cross-model validation wrapper
|
||||
├── validate-operation-payload.ts # Checkpoint A - payload validation
|
||||
└── repair-operation.service.ts # REPAIR operation creation + notification
|
||||
|
||||
src/app/pfapi/
|
||||
├── pfapi-store-delegate.service.ts # Reads NgRx for sync (Part B)
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { inject, Injectable } from '@angular/core';
|
||||
import { Store } from '@ngrx/store';
|
||||
import { MatDialog, MatDialogRef } from '@angular/material/dialog';
|
||||
import { EntityConflict } from './operation.types';
|
||||
import { OperationApplierService } from './operation-applier.service';
|
||||
|
|
@ -11,6 +12,11 @@ import {
|
|||
import { firstValueFrom } from 'rxjs';
|
||||
import { SnackService } from '../../snack/snack.service';
|
||||
import { T } from '../../../t.const';
|
||||
import { ValidateStateService } from './validate-state.service';
|
||||
import { RepairOperationService } from './repair-operation.service';
|
||||
import { PfapiStoreDelegateService } from '../../../pfapi/pfapi-store-delegate.service';
|
||||
import { AppDataCompleteNew } from '../../../pfapi/pfapi-config';
|
||||
import { loadAllData } from '../../../root-store/meta/load-all-data.action';
|
||||
|
||||
/**
|
||||
* Service to manage conflict resolution, typically presenting a UI to the user.
|
||||
|
|
@ -21,9 +27,13 @@ import { T } from '../../../t.const';
|
|||
})
|
||||
export class ConflictResolutionService {
|
||||
private dialog = inject(MatDialog);
|
||||
private store = inject(Store);
|
||||
private operationApplier = inject(OperationApplierService);
|
||||
private opLogStore = inject(OperationLogStoreService);
|
||||
private snackService = inject(SnackService);
|
||||
private validateStateService = inject(ValidateStateService);
|
||||
private repairOperationService = inject(RepairOperationService);
|
||||
private storeDelegateService = inject(PfapiStoreDelegateService);
|
||||
|
||||
private _dialogRef?: MatDialogRef<DialogConflictResolutionComponent>;
|
||||
|
||||
|
|
@ -99,6 +109,50 @@ export class ConflictResolutionService {
|
|||
);
|
||||
}
|
||||
}
|
||||
|
||||
// CHECKPOINT D: Validate and repair state after conflict resolution
|
||||
await this._validateAndRepairAfterResolution();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates the current state after conflict resolution and repairs if necessary.
|
||||
* This is Checkpoint D in the validation architecture.
|
||||
*/
|
||||
private async _validateAndRepairAfterResolution(): Promise<void> {
|
||||
PFLog.normal('[ConflictResolutionService] Running post-resolution validation...');
|
||||
|
||||
// Get current state from NgRx
|
||||
const currentState =
|
||||
(await this.storeDelegateService.getAllSyncModelDataFromStore()) as AppDataCompleteNew;
|
||||
|
||||
// Validate and repair if needed
|
||||
const result = this.validateStateService.validateAndRepair(currentState);
|
||||
|
||||
if (!result.wasRepaired) {
|
||||
PFLog.normal('[ConflictResolutionService] State valid after conflict resolution');
|
||||
return;
|
||||
}
|
||||
|
||||
if (!result.repairedState || !result.repairSummary) {
|
||||
PFLog.err(
|
||||
'[ConflictResolutionService] Repair failed after conflict resolution:',
|
||||
result.error,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Create REPAIR operation
|
||||
await this.repairOperationService.createRepairOperation(
|
||||
result.repairedState,
|
||||
result.repairSummary,
|
||||
);
|
||||
|
||||
// Dispatch repaired state to NgRx
|
||||
this.store.dispatch(loadAllData({ appDataComplete: result.repairedState as any }));
|
||||
|
||||
PFLog.log(
|
||||
'[ConflictResolutionService] Created REPAIR operation after conflict resolution',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,11 +12,14 @@ import {
|
|||
import { PFLog } from '../../log';
|
||||
import { PfapiService } from '../../../pfapi/pfapi.service';
|
||||
import { PfapiStoreDelegateService } from '../../../pfapi/pfapi-store-delegate.service';
|
||||
import { Operation, OpType } from './operation.types';
|
||||
import { Operation, OpType, RepairPayload } from './operation.types';
|
||||
import { uuidv7 } from '../../../util/uuid-v7';
|
||||
import { incrementVectorClock } from '../../../pfapi/api/util/vector-clock';
|
||||
import { SnackService } from '../../snack/snack.service';
|
||||
import { T } from '../../../t.const';
|
||||
import { ValidateStateService } from './validate-state.service';
|
||||
import { RepairOperationService } from './repair-operation.service';
|
||||
import { AppDataCompleteNew } from '../../../pfapi/pfapi-config';
|
||||
|
||||
type StateCache = MigratableStateCache;
|
||||
|
||||
|
|
@ -36,6 +39,11 @@ export class OperationLogHydratorService {
|
|||
private pfapiService = inject(PfapiService);
|
||||
private storeDelegateService = inject(PfapiStoreDelegateService);
|
||||
private snackService = inject(SnackService);
|
||||
private validateStateService = inject(ValidateStateService);
|
||||
private repairOperationService = inject(RepairOperationService);
|
||||
|
||||
// Flag to prevent re-validation immediately after repair
|
||||
private _isRepairInProgress = false;
|
||||
|
||||
async hydrateStore(): Promise<void> {
|
||||
PFLog.normal('OperationLogHydratorService: Starting hydration...');
|
||||
|
|
@ -76,29 +84,37 @@ export class OperationLogHydratorService {
|
|||
PFLog.normal('OperationLogHydratorService: Snapshot found. Hydrating state...', {
|
||||
lastAppliedOpSeq: snapshot.lastAppliedOpSeq,
|
||||
});
|
||||
// 3. Hydrate NgRx with snapshot
|
||||
// We cast state to any because AllSyncModels type is complex and we trust the cache
|
||||
this.store.dispatch(loadAllData({ appDataComplete: snapshot.state as any }));
|
||||
|
||||
// CHECKPOINT B: Validate and repair snapshot state before dispatching
|
||||
let stateToLoad = snapshot.state as AppDataCompleteNew;
|
||||
if (!this._isRepairInProgress) {
|
||||
const validationResult = await this._validateAndRepairState(
|
||||
stateToLoad,
|
||||
'snapshot',
|
||||
);
|
||||
if (validationResult.wasRepaired && validationResult.repairedState) {
|
||||
stateToLoad = validationResult.repairedState;
|
||||
// Update snapshot with repaired state
|
||||
snapshot = { ...snapshot, state: stateToLoad };
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Hydrate NgRx with (possibly repaired) snapshot
|
||||
this.store.dispatch(loadAllData({ appDataComplete: stateToLoad as any }));
|
||||
|
||||
// 4. Replay tail operations
|
||||
const tailOps = await this.opLogStore.getOpsAfterSeq(snapshot.lastAppliedOpSeq);
|
||||
|
||||
if (tailOps.length > 0) {
|
||||
// Optimization: If last op is SyncImport, skip replay and load it directly
|
||||
// Optimization: If last op is SyncImport or Repair, skip replay and load directly
|
||||
const lastOp = tailOps[tailOps.length - 1].op;
|
||||
if (lastOp.opType === OpType.SyncImport && lastOp.payload) {
|
||||
const appData = this._extractFullStateFromOp(lastOp);
|
||||
if (appData) {
|
||||
PFLog.normal(
|
||||
`OperationLogHydratorService: Last of ${tailOps.length} tail ops is SyncImport, loading directly`,
|
||||
`OperationLogHydratorService: Last of ${tailOps.length} tail ops is ${lastOp.opType}, loading directly`,
|
||||
);
|
||||
const payload = lastOp.payload as { appDataComplete?: unknown } | unknown;
|
||||
const appData =
|
||||
typeof payload === 'object' &&
|
||||
payload !== null &&
|
||||
'appDataComplete' in payload
|
||||
? payload.appDataComplete
|
||||
: payload;
|
||||
this.store.dispatch(loadAllData({ appDataComplete: appData as any }));
|
||||
// No snapshot save needed - SyncImport already contains full state
|
||||
// No snapshot save needed - full state ops already contain complete state
|
||||
// Snapshot will be saved after next batch of regular operations
|
||||
} else {
|
||||
PFLog.normal(
|
||||
|
|
@ -116,6 +132,11 @@ export class OperationLogHydratorService {
|
|||
);
|
||||
await this._saveCurrentStateAsSnapshot();
|
||||
}
|
||||
|
||||
// CHECKPOINT C: Validate state after replaying tail operations
|
||||
if (!this._isRepairInProgress) {
|
||||
await this._validateAndRepairCurrentState('tail-replay');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -136,21 +157,15 @@ export class OperationLogHydratorService {
|
|||
return;
|
||||
}
|
||||
|
||||
// Optimization: If last op is SyncImport, skip replay and load it directly
|
||||
// Optimization: If last op is SyncImport or Repair, skip replay and load directly
|
||||
const lastOp = allOps[allOps.length - 1].op;
|
||||
if (lastOp.opType === OpType.SyncImport && lastOp.payload) {
|
||||
const appData = this._extractFullStateFromOp(lastOp);
|
||||
if (appData) {
|
||||
PFLog.normal(
|
||||
`OperationLogHydratorService: Last of ${allOps.length} ops is SyncImport, loading directly`,
|
||||
`OperationLogHydratorService: Last of ${allOps.length} ops is ${lastOp.opType}, loading directly`,
|
||||
);
|
||||
const payload = lastOp.payload as { appDataComplete?: unknown } | unknown;
|
||||
const appData =
|
||||
typeof payload === 'object' &&
|
||||
payload !== null &&
|
||||
'appDataComplete' in payload
|
||||
? payload.appDataComplete
|
||||
: payload;
|
||||
this.store.dispatch(loadAllData({ appDataComplete: appData as any }));
|
||||
// No snapshot save needed - SyncImport already contains full state
|
||||
// No snapshot save needed - full state ops already contain complete state
|
||||
} else {
|
||||
PFLog.normal(
|
||||
`OperationLogHydratorService: Replaying all ${allOps.length} operations.`,
|
||||
|
|
@ -165,6 +180,11 @@ export class OperationLogHydratorService {
|
|||
`OperationLogHydratorService: Saving snapshot after replaying ${allOps.length} ops`,
|
||||
);
|
||||
await this._saveCurrentStateAsSnapshot();
|
||||
|
||||
// CHECKPOINT C: Validate state after replaying all operations
|
||||
if (!this._isRepairInProgress) {
|
||||
await this._validateAndRepairCurrentState('full-replay');
|
||||
}
|
||||
}
|
||||
|
||||
PFLog.normal('OperationLogHydratorService: Full replay complete.');
|
||||
|
|
@ -218,6 +238,47 @@ export class OperationLogHydratorService {
|
|||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts full application state from operations that contain complete state.
|
||||
* Returns undefined for operations that don't contain full state (normal CRUD ops).
|
||||
*
|
||||
* Operations that contain full state:
|
||||
* - OpType.SyncImport: Full state from remote sync
|
||||
* - OpType.Repair: Full repaired state from auto-repair
|
||||
* - OpType.BackupImport: Full state from backup file restore
|
||||
*/
|
||||
private _extractFullStateFromOp(op: Operation): unknown | undefined {
|
||||
if (!op.payload) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// Handle full state operations
|
||||
if (
|
||||
op.opType === OpType.SyncImport ||
|
||||
op.opType === OpType.BackupImport ||
|
||||
op.opType === OpType.Repair
|
||||
) {
|
||||
const payload = op.payload as
|
||||
| { appDataComplete?: unknown }
|
||||
| RepairPayload
|
||||
| unknown;
|
||||
|
||||
// Check if payload has appDataComplete wrapper
|
||||
if (
|
||||
typeof payload === 'object' &&
|
||||
payload !== null &&
|
||||
'appDataComplete' in payload
|
||||
) {
|
||||
return (payload as { appDataComplete: unknown }).appDataComplete;
|
||||
}
|
||||
|
||||
// Legacy format: payload IS the appDataComplete
|
||||
return payload;
|
||||
}
|
||||
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Attempts to recover from a corrupted or missing SUP_OPS database.
|
||||
* Recovery strategy:
|
||||
|
|
@ -420,4 +481,64 @@ export class OperationLogHydratorService {
|
|||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates a state object and repairs it if necessary.
|
||||
* Used for validating snapshot state before dispatching.
|
||||
*
|
||||
* @param state - The state to validate
|
||||
* @param context - Context string for logging (e.g., 'snapshot', 'tail-replay')
|
||||
* @returns Validation result with optional repaired state
|
||||
*/
|
||||
private async _validateAndRepairState(
|
||||
state: AppDataCompleteNew,
|
||||
context: string,
|
||||
): Promise<{ wasRepaired: boolean; repairedState?: AppDataCompleteNew }> {
|
||||
const result = this.validateStateService.validateAndRepair(state);
|
||||
|
||||
if (!result.wasRepaired) {
|
||||
return { wasRepaired: false };
|
||||
}
|
||||
|
||||
if (!result.repairedState || !result.repairSummary) {
|
||||
PFLog.err(
|
||||
`[OperationLogHydratorService] Repair failed for ${context}:`,
|
||||
result.error,
|
||||
);
|
||||
return { wasRepaired: false };
|
||||
}
|
||||
|
||||
// Create REPAIR operation to persist the repaired state
|
||||
this._isRepairInProgress = true;
|
||||
try {
|
||||
await this.repairOperationService.createRepairOperation(
|
||||
result.repairedState,
|
||||
result.repairSummary,
|
||||
);
|
||||
PFLog.log(`[OperationLogHydratorService] Created REPAIR operation for ${context}`);
|
||||
} finally {
|
||||
this._isRepairInProgress = false;
|
||||
}
|
||||
|
||||
return { wasRepaired: true, repairedState: result.repairedState };
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates the current NgRx state and repairs it if necessary.
|
||||
* Used after replaying operations.
|
||||
*
|
||||
* @param context - Context string for logging
|
||||
*/
|
||||
private async _validateAndRepairCurrentState(context: string): Promise<void> {
|
||||
// Get current state from NgRx
|
||||
const currentState =
|
||||
(await this.storeDelegateService.getAllSyncModelDataFromStore()) as AppDataCompleteNew;
|
||||
|
||||
const result = await this._validateAndRepairState(currentState, context);
|
||||
|
||||
if (result.wasRepaired && result.repairedState) {
|
||||
// Dispatch the repaired state to NgRx
|
||||
this.store.dispatch(loadAllData({ appDataComplete: result.repairedState as any }));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
import { inject, Injectable } from '@angular/core';
|
||||
import { Store } from '@ngrx/store';
|
||||
import { OperationLogStoreService } from './operation-log-store.service';
|
||||
import { LockService } from './lock.service';
|
||||
import {
|
||||
|
|
@ -25,6 +26,11 @@ import {
|
|||
import { SyncProviderId } from '../../../pfapi/api/pfapi.const';
|
||||
import { OperationApplierService } from './operation-applier.service';
|
||||
import { ConflictResolutionService } from './conflict-resolution.service';
|
||||
import { ValidateStateService } from './validate-state.service';
|
||||
import { RepairOperationService } from './repair-operation.service';
|
||||
import { PfapiStoreDelegateService } from '../../../pfapi/pfapi-store-delegate.service';
|
||||
import { AppDataCompleteNew } from '../../../pfapi/pfapi-config';
|
||||
import { loadAllData } from '../../../root-store/meta/load-all-data.action';
|
||||
|
||||
const OPS_DIR = 'ops/';
|
||||
const MANIFEST_FILE_NAME = OPS_DIR + 'manifest.json';
|
||||
|
|
@ -51,10 +57,14 @@ const isOperationSyncCapable = (
|
|||
providedIn: 'root',
|
||||
})
|
||||
export class OperationLogSyncService {
|
||||
private store = inject(Store);
|
||||
private opLogStore = inject(OperationLogStoreService);
|
||||
private lockService = inject(LockService);
|
||||
private operationApplier = inject(OperationApplierService);
|
||||
private conflictResolutionService = inject(ConflictResolutionService);
|
||||
private validateStateService = inject(ValidateStateService);
|
||||
private repairOperationService = inject(RepairOperationService);
|
||||
private storeDelegateService = inject(PfapiStoreDelegateService);
|
||||
|
||||
private _getManifestFileName(): string {
|
||||
return MANIFEST_FILE_NAME;
|
||||
|
|
@ -412,7 +422,12 @@ export class OperationLogSyncService {
|
|||
`OperationLogSyncService: Detected ${conflicts.length} conflicts.`,
|
||||
conflicts,
|
||||
);
|
||||
await this.conflictResolutionService.presentConflicts(conflicts); // Call conflict resolution UI
|
||||
// Conflict resolution service will validate after resolving
|
||||
await this.conflictResolutionService.presentConflicts(conflicts);
|
||||
} else if (nonConflicting.length > 0) {
|
||||
// CHECKPOINT D: If no conflicts but we applied ops, validate state
|
||||
// (Conflict resolution handles validation when there are conflicts)
|
||||
await this._validateAfterSync();
|
||||
}
|
||||
|
||||
PFLog.normal(
|
||||
|
|
@ -484,7 +499,11 @@ export class OperationLogSyncService {
|
|||
`OperationLogSyncService: Detected ${conflicts.length} conflicts.`,
|
||||
conflicts,
|
||||
);
|
||||
// Conflict resolution service will validate after resolving
|
||||
await this.conflictResolutionService.presentConflicts(conflicts);
|
||||
} else if (nonConflicting.length > 0) {
|
||||
// CHECKPOINT D: If no conflicts but we applied ops, validate state
|
||||
await this._validateAfterSync();
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -578,4 +597,40 @@ export class OperationLogSyncService {
|
|||
}
|
||||
return { nonConflicting, conflicts };
|
||||
}
|
||||
|
||||
/**
|
||||
* CHECKPOINT D: Validates state after applying remote operations.
|
||||
* If validation fails, attempts repair and creates a REPAIR operation.
|
||||
*/
|
||||
private async _validateAfterSync(): Promise<void> {
|
||||
PFLog.normal('[OperationLogSyncService] Running post-sync validation...');
|
||||
|
||||
// Get current state from NgRx
|
||||
const currentState =
|
||||
(await this.storeDelegateService.getAllSyncModelDataFromStore()) as AppDataCompleteNew;
|
||||
|
||||
// Validate and repair if needed
|
||||
const result = this.validateStateService.validateAndRepair(currentState);
|
||||
|
||||
if (!result.wasRepaired) {
|
||||
PFLog.normal('[OperationLogSyncService] State valid after sync');
|
||||
return;
|
||||
}
|
||||
|
||||
if (!result.repairedState || !result.repairSummary) {
|
||||
PFLog.err('[OperationLogSyncService] Repair failed after sync:', result.error);
|
||||
return;
|
||||
}
|
||||
|
||||
// Create REPAIR operation
|
||||
await this.repairOperationService.createRepairOperation(
|
||||
result.repairedState,
|
||||
result.repairSummary,
|
||||
);
|
||||
|
||||
// Dispatch repaired state to NgRx
|
||||
this.store.dispatch(loadAllData({ appDataComplete: result.repairedState as any }));
|
||||
|
||||
PFLog.log('[OperationLogSyncService] Created REPAIR operation after sync');
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,6 +13,7 @@ import { OperationLogCompactionService } from './operation-log-compaction.servic
|
|||
import { PFLog } from '../../log';
|
||||
import { SnackService } from '../../snack/snack.service';
|
||||
import { T } from '../../../t.const';
|
||||
import { validateOperationPayload } from './validate-operation-payload';
|
||||
|
||||
const CURRENT_SCHEMA_VERSION = 1;
|
||||
const COMPACTION_THRESHOLD = 500;
|
||||
|
|
@ -81,6 +82,30 @@ export class OperationLogEffects {
|
|||
schemaVersion: CURRENT_SCHEMA_VERSION,
|
||||
};
|
||||
|
||||
// CHECKPOINT A: Validate payload before persisting
|
||||
const validationResult = validateOperationPayload(op);
|
||||
if (!validationResult.success) {
|
||||
PFLog.err('[OperationLogEffects] Invalid operation payload', {
|
||||
error: validationResult.error,
|
||||
actionType: action.type,
|
||||
opType: op.opType,
|
||||
entityType: op.entityType,
|
||||
});
|
||||
this.snackService.open({
|
||||
type: 'ERROR',
|
||||
msg: T.F.SYNC.S.INVALID_OPERATION_PAYLOAD,
|
||||
});
|
||||
return; // Skip persisting invalid operation
|
||||
}
|
||||
|
||||
// Log warnings if any (but still persist)
|
||||
if (validationResult.warnings?.length) {
|
||||
PFLog.warn('[OperationLogEffects] Operation payload warnings', {
|
||||
warnings: validationResult.warnings,
|
||||
actionType: action.type,
|
||||
});
|
||||
}
|
||||
|
||||
// 1. Write to SUP_OPS (Part A)
|
||||
await this.opLogStore.append(op);
|
||||
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ export enum OpType {
|
|||
Batch = 'BATCH', // For bulk operations (import, mass update)
|
||||
SyncImport = 'SYNC_IMPORT', // Full state import from remote sync
|
||||
BackupImport = 'BACKUP_IMPORT', // Full state import from backup file
|
||||
Repair = 'REPAIR', // Auto-repair operation with full repaired state
|
||||
}
|
||||
|
||||
export type EntityType =
|
||||
|
|
@ -83,3 +84,25 @@ export interface OperationLogManifest {
|
|||
lastCompactedSeq?: number; // The sequence number of the last op included in a full state snapshot
|
||||
lastCompactedSnapshotFile?: string; // Reference to the last full state snapshot file
|
||||
}
|
||||
|
||||
/**
|
||||
* Minimal summary of repairs performed, used in REPAIR operation payload.
|
||||
* Keeps repair log lightweight while providing debugging info.
|
||||
*/
|
||||
export interface RepairSummary {
|
||||
entityStateFixed: number; // Fixed ids/entities array sync
|
||||
orphanedEntitiesRestored: number; // Tasks restored from archive, orphaned notes fixed
|
||||
invalidReferencesRemoved: number; // Non-existent project/tag IDs removed
|
||||
relationshipsFixed: number; // Project/tag ID consistency, subtask parent relationships
|
||||
structureRepaired: number; // Menu tree, inbox project creation
|
||||
typeErrorsFixed: number; // Typia errors auto-fixed (type coercion)
|
||||
}
|
||||
|
||||
/**
|
||||
* Payload structure for REPAIR operations.
|
||||
* Contains the fully repaired state and a summary of what was fixed.
|
||||
*/
|
||||
export interface RepairPayload {
|
||||
appDataComplete: unknown; // AppDataCompleteNew - using unknown to avoid circular deps
|
||||
repairSummary: RepairSummary;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,135 @@
|
|||
import { Injectable, inject } from '@angular/core';
|
||||
import { OperationLogStoreService } from './operation-log-store.service';
|
||||
import { PfapiService } from '../../../pfapi/pfapi.service';
|
||||
import { Operation, OpType, RepairPayload, RepairSummary } from './operation.types';
|
||||
import { uuidv7 } from '../../../util/uuid-v7';
|
||||
import { incrementVectorClock } from '../../../pfapi/api/util/vector-clock';
|
||||
import { LockService } from './lock.service';
|
||||
import { SnackService } from '../../snack/snack.service';
|
||||
import { T } from '../../../t.const';
|
||||
import { PFLog } from '../../log';
|
||||
|
||||
const CURRENT_SCHEMA_VERSION = 1;
|
||||
|
||||
/**
|
||||
* Service responsible for creating REPAIR operations.
|
||||
* When validation fails and data is repaired, this service creates a REPAIR operation
|
||||
* containing the full repaired state and a summary of what was fixed.
|
||||
* REPAIR operations behave like SyncImport - they replace the entire state atomically.
|
||||
*/
|
||||
@Injectable({
|
||||
providedIn: 'root',
|
||||
})
|
||||
export class RepairOperationService {
|
||||
private opLogStore = inject(OperationLogStoreService);
|
||||
private pfapiService = inject(PfapiService);
|
||||
private lockService = inject(LockService);
|
||||
private snackService = inject(SnackService);
|
||||
|
||||
/**
|
||||
* Creates a REPAIR operation with the repaired state and saves it to the operation log.
|
||||
* Also updates the state cache to the repaired state for faster future hydration.
|
||||
*
|
||||
* @param repairedState - The fully repaired application state
|
||||
* @param repairSummary - Summary of what was repaired (counts by category)
|
||||
* @returns The sequence number of the created operation
|
||||
*/
|
||||
async createRepairOperation(
|
||||
repairedState: unknown,
|
||||
repairSummary: RepairSummary,
|
||||
): Promise<number> {
|
||||
const clientId = await this.pfapiService.pf.metaModel.loadClientId();
|
||||
if (!clientId) {
|
||||
throw new Error('Failed to load clientId - cannot create repair operation');
|
||||
}
|
||||
|
||||
const payload: RepairPayload = {
|
||||
appDataComplete: repairedState,
|
||||
repairSummary,
|
||||
};
|
||||
|
||||
let seq: number = 0;
|
||||
|
||||
await this.lockService.request('sp_op_log', async () => {
|
||||
const currentClock = await this.opLogStore.getCurrentVectorClock();
|
||||
const newClock = incrementVectorClock(currentClock, clientId);
|
||||
|
||||
const op: Operation = {
|
||||
id: uuidv7(),
|
||||
actionType: '[Repair] Auto Repair',
|
||||
opType: OpType.Repair,
|
||||
entityType: 'ALL',
|
||||
payload,
|
||||
clientId,
|
||||
vectorClock: newClock,
|
||||
timestamp: Date.now(),
|
||||
schemaVersion: CURRENT_SCHEMA_VERSION,
|
||||
};
|
||||
|
||||
// 1. Append REPAIR operation to log
|
||||
await this.opLogStore.append(op, 'local');
|
||||
|
||||
// 2. Save state cache with repaired state for fast hydration
|
||||
seq = await this.opLogStore.getLastSeq();
|
||||
await this.opLogStore.saveStateCache({
|
||||
state: repairedState,
|
||||
lastAppliedOpSeq: seq,
|
||||
vectorClock: newClock,
|
||||
compactedAt: Date.now(),
|
||||
schemaVersion: CURRENT_SCHEMA_VERSION,
|
||||
});
|
||||
|
||||
PFLog.log('[RepairOperationService] Created REPAIR operation', {
|
||||
seq,
|
||||
repairSummary,
|
||||
});
|
||||
});
|
||||
|
||||
// Notify user that repair happened
|
||||
this._notifyUser(repairSummary);
|
||||
|
||||
return seq;
|
||||
}
|
||||
|
||||
/**
|
||||
* Shows a snackbar notification to the user about the repair.
|
||||
*/
|
||||
private _notifyUser(summary: RepairSummary): void {
|
||||
const totalFixes = this._getTotalFixes(summary);
|
||||
if (totalFixes > 0) {
|
||||
this.snackService.open({
|
||||
type: 'SUCCESS',
|
||||
msg: T.F.SYNC.S.DATA_REPAIRED,
|
||||
translateParams: { count: totalFixes.toString() },
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Calculates the total number of fixes from a repair summary.
|
||||
*/
|
||||
private _getTotalFixes(summary: RepairSummary): number {
|
||||
return (
|
||||
summary.entityStateFixed +
|
||||
summary.orphanedEntitiesRestored +
|
||||
summary.invalidReferencesRemoved +
|
||||
summary.relationshipsFixed +
|
||||
summary.structureRepaired +
|
||||
summary.typeErrorsFixed
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates an empty repair summary (all counts at zero).
|
||||
*/
|
||||
static createEmptyRepairSummary(): RepairSummary {
|
||||
return {
|
||||
entityStateFixed: 0,
|
||||
orphanedEntitiesRestored: 0,
|
||||
invalidReferencesRemoved: 0,
|
||||
relationshipsFixed: 0,
|
||||
structureRepaired: 0,
|
||||
typeErrorsFixed: 0,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,323 @@
|
|||
import { Operation, EntityType, OpType } from './operation.types';
|
||||
import { PFLog } from '../../log';
|
||||
|
||||
/**
|
||||
* Result of validating an operation payload.
|
||||
*/
|
||||
export interface PayloadValidationResult {
|
||||
success: boolean;
|
||||
error?: string;
|
||||
warnings?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Maps EntityType to the expected payload key.
|
||||
*/
|
||||
const getEntityKeyFromType = (entityType: EntityType): string | null => {
|
||||
const mapping: Record<string, string> = {
|
||||
TASK: 'task',
|
||||
PROJECT: 'project',
|
||||
TAG: 'tag',
|
||||
NOTE: 'note',
|
||||
GLOBAL_CONFIG: 'globalConfig',
|
||||
SIMPLE_COUNTER: 'simpleCounter',
|
||||
WORK_CONTEXT: 'workContext',
|
||||
TASK_REPEAT_CFG: 'taskRepeatCfg',
|
||||
ISSUE_PROVIDER: 'issueProvider',
|
||||
PLANNER: 'planner',
|
||||
};
|
||||
return mapping[entityType] || null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Attempts to find an entity-like object in the payload.
|
||||
* Used when the entity key doesn't match the expected pattern.
|
||||
*/
|
||||
const findEntityInPayload = (payload: Record<string, unknown>): unknown => {
|
||||
const entityKeys = [
|
||||
'task',
|
||||
'project',
|
||||
'tag',
|
||||
'note',
|
||||
'simpleCounter',
|
||||
'workContext',
|
||||
'taskRepeatCfg',
|
||||
'issueProvider',
|
||||
];
|
||||
|
||||
for (const key of entityKeys) {
|
||||
if (payload[key] && typeof payload[key] === 'object') {
|
||||
return payload[key];
|
||||
}
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Checks if an object looks like AppDataCompleteNew.
|
||||
*/
|
||||
const isLikelyAppDataComplete = (obj: unknown): boolean => {
|
||||
if (!obj || typeof obj !== 'object') return false;
|
||||
const o = obj as Record<string, unknown>;
|
||||
// Check for a few key properties that AppDataCompleteNew should have
|
||||
return 'task' in o || 'project' in o || 'globalConfig' in o;
|
||||
};
|
||||
|
||||
/**
|
||||
* Validates CREATE operation payload.
|
||||
* Expects payload to contain the entity being created.
|
||||
*/
|
||||
const validateCreatePayload = (
|
||||
entityType: EntityType,
|
||||
payload: unknown,
|
||||
): PayloadValidationResult => {
|
||||
const p = payload as Record<string, unknown>;
|
||||
const entityKey = getEntityKeyFromType(entityType);
|
||||
const warnings: string[] = [];
|
||||
|
||||
// For create operations, we expect the entity to be in the payload
|
||||
const entity = entityKey ? p[entityKey] : findEntityInPayload(p);
|
||||
|
||||
if (!entity) {
|
||||
// Warning rather than error - some creates might have different shapes
|
||||
warnings.push(`CREATE payload missing expected entity (${entityType})`);
|
||||
PFLog.warn(`[ValidateOperationPayload] ${warnings[0]}`, payload);
|
||||
} else if (typeof entity === 'object' && entity !== null) {
|
||||
// Basic ID check for the entity
|
||||
const entityObj = entity as Record<string, unknown>;
|
||||
if (!entityObj.id || typeof entityObj.id !== 'string') {
|
||||
return {
|
||||
success: false,
|
||||
error: `CREATE entity missing valid 'id' field`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return { success: true, warnings: warnings.length > 0 ? warnings : undefined };
|
||||
};
|
||||
|
||||
/**
|
||||
* Validates UPDATE operation payload.
|
||||
* Expects payload to contain entity ID and changes, or a task/entity with changes.
|
||||
*/
|
||||
const validateUpdatePayload = (
|
||||
entityType: EntityType,
|
||||
payload: unknown,
|
||||
): PayloadValidationResult => {
|
||||
const p = payload as Record<string, unknown>;
|
||||
const warnings: string[] = [];
|
||||
|
||||
// Update payloads can have various shapes:
|
||||
// 1. { task: { id, changes } } or { project: { id, changes } }
|
||||
// 2. { id, changes }
|
||||
// 3. { task: Task, ...otherProps } - for convertToMainTask etc.
|
||||
// 4. { tasks: Task[] } - for batch updates like moveToArchive
|
||||
|
||||
const entityKey = getEntityKeyFromType(entityType);
|
||||
|
||||
// Check for nested update shape: { task: { id, changes } }
|
||||
if (entityKey && p[entityKey]) {
|
||||
const nested = p[entityKey] as Record<string, unknown>;
|
||||
if (nested.id && nested.changes) {
|
||||
return { success: true };
|
||||
}
|
||||
// Could be a full entity for updates that pass the whole entity
|
||||
if (nested.id) {
|
||||
return { success: true };
|
||||
}
|
||||
}
|
||||
|
||||
// Check for direct shape: { id, changes }
|
||||
if (p.id && typeof p.id === 'string') {
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
// Check for batch update shape: { tasks: Task[] }
|
||||
if (entityKey) {
|
||||
const pluralKey = entityKey + 's';
|
||||
if (Array.isArray(p[pluralKey])) {
|
||||
return { success: true };
|
||||
}
|
||||
}
|
||||
|
||||
// Check if there's any entity-like object in payload
|
||||
const entity = findEntityInPayload(p);
|
||||
if (entity && typeof entity === 'object' && (entity as Record<string, unknown>).id) {
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
// Allow through with warning - updates have many shapes
|
||||
warnings.push(`UPDATE payload has unusual structure for ${entityType}`);
|
||||
PFLog.warn(`[ValidateOperationPayload] ${warnings[0]}`, payload);
|
||||
|
||||
return { success: true, warnings };
|
||||
};
|
||||
|
||||
/**
|
||||
* Validates DELETE operation payload.
|
||||
* Expects entityId/entityIds in operation or in payload.
|
||||
*/
|
||||
const validateDeletePayload = (
|
||||
entityType: EntityType,
|
||||
payload: unknown,
|
||||
entityId?: string,
|
||||
entityIds?: string[],
|
||||
): PayloadValidationResult => {
|
||||
// entityId or entityIds should be on the operation itself
|
||||
if (entityId && typeof entityId === 'string') {
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
if (entityIds && Array.isArray(entityIds) && entityIds.length > 0) {
|
||||
const allStrings = entityIds.every((id) => typeof id === 'string');
|
||||
if (!allStrings) {
|
||||
return { success: false, error: 'DELETE entityIds must all be strings' };
|
||||
}
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
// Check payload for task/taskIds
|
||||
const p = payload as Record<string, unknown>;
|
||||
if (p.taskIds && Array.isArray(p.taskIds)) {
|
||||
return { success: true };
|
||||
}
|
||||
if (p.task && typeof p.task === 'object' && (p.task as Record<string, unknown>).id) {
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
// Allow through with warning
|
||||
PFLog.warn(
|
||||
`[ValidateOperationPayload] DELETE missing entityId/entityIds for ${entityType}`,
|
||||
payload,
|
||||
);
|
||||
return { success: true, warnings: ['DELETE missing entityId/entityIds'] };
|
||||
};
|
||||
|
||||
/**
|
||||
* Validates MOVE operation payload.
|
||||
* Expects entityIds array for reordering.
|
||||
*/
|
||||
const validateMovePayload = (
|
||||
payload: unknown,
|
||||
entityIds?: string[],
|
||||
): PayloadValidationResult => {
|
||||
// entityIds should be on the operation
|
||||
if (entityIds && Array.isArray(entityIds)) {
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
// Or in the payload
|
||||
const p = payload as Record<string, unknown>;
|
||||
if (p.ids && Array.isArray(p.ids)) {
|
||||
return { success: true };
|
||||
}
|
||||
if (p.taskIds && Array.isArray(p.taskIds)) {
|
||||
return { success: true };
|
||||
}
|
||||
|
||||
PFLog.warn('[ValidateOperationPayload] MOVE missing ids array', payload);
|
||||
return { success: true, warnings: ['MOVE missing ids array'] };
|
||||
};
|
||||
|
||||
/**
|
||||
* Validates BATCH operation payload.
|
||||
* Allows various batch structures through with minimal validation.
|
||||
*/
|
||||
const validateBatchPayload = (payload: unknown): PayloadValidationResult => {
|
||||
const p = payload as Record<string, unknown>;
|
||||
|
||||
// Batch operations can have many shapes
|
||||
// Just ensure payload is not empty
|
||||
if (Object.keys(p).length === 0) {
|
||||
return { success: false, error: 'BATCH payload cannot be empty' };
|
||||
}
|
||||
|
||||
return { success: true };
|
||||
};
|
||||
|
||||
/**
|
||||
* Validates SYNC_IMPORT/BACKUP_IMPORT payload.
|
||||
* Expects appDataComplete structure.
|
||||
*/
|
||||
const validateFullStatePayload = (payload: unknown): PayloadValidationResult => {
|
||||
const p = payload as Record<string, unknown>;
|
||||
|
||||
// Full state imports should have appDataComplete
|
||||
if (!p.appDataComplete && !isLikelyAppDataComplete(p)) {
|
||||
return {
|
||||
success: false,
|
||||
error: 'Full state import missing appDataComplete',
|
||||
};
|
||||
}
|
||||
|
||||
const data = (p.appDataComplete || p) as Record<string, unknown>;
|
||||
|
||||
// Check for at least some expected keys
|
||||
const expectedKeys = ['task', 'project', 'tag', 'globalConfig'];
|
||||
const hasExpectedKeys = expectedKeys.some((key) => key in data);
|
||||
|
||||
if (!hasExpectedKeys) {
|
||||
return {
|
||||
success: false,
|
||||
error: 'Full state import missing expected data keys (task, project, tag, etc.)',
|
||||
};
|
||||
}
|
||||
|
||||
return { success: true };
|
||||
};
|
||||
|
||||
/**
|
||||
* Validates an operation payload before persisting to IndexedDB.
|
||||
*
|
||||
* This is Checkpoint A in the validation architecture.
|
||||
* - For CREATE/UPDATE operations: validates payload has required entity/id structure
|
||||
* - For DELETE operations: validates IDs are present
|
||||
* - For SYNC_IMPORT/BACKUP_IMPORT: validates appDataComplete structure exists
|
||||
* - For REPAIR: skips validation (internally generated)
|
||||
*
|
||||
* NOTE: This validation is intentionally lenient to start.
|
||||
* It checks structural requirements rather than deep entity validation.
|
||||
* Full Typia validation happens at state checkpoints (B, C, D).
|
||||
*/
|
||||
export const validateOperationPayload = (op: Operation): PayloadValidationResult => {
|
||||
// 1. Basic structural validation
|
||||
if (op.payload === null || op.payload === undefined) {
|
||||
return { success: false, error: 'Payload cannot be null or undefined' };
|
||||
}
|
||||
|
||||
if (typeof op.payload !== 'object') {
|
||||
return { success: false, error: 'Payload must be an object' };
|
||||
}
|
||||
|
||||
// 2. Validate based on operation type
|
||||
switch (op.opType) {
|
||||
case OpType.Create:
|
||||
return validateCreatePayload(op.entityType, op.payload);
|
||||
|
||||
case OpType.Update:
|
||||
return validateUpdatePayload(op.entityType, op.payload);
|
||||
|
||||
case OpType.Delete:
|
||||
return validateDeletePayload(op.entityType, op.payload, op.entityId, op.entityIds);
|
||||
|
||||
case OpType.Move:
|
||||
return validateMovePayload(op.payload, op.entityIds);
|
||||
|
||||
case OpType.Batch:
|
||||
return validateBatchPayload(op.payload);
|
||||
|
||||
case OpType.SyncImport:
|
||||
case OpType.BackupImport:
|
||||
return validateFullStatePayload(op.payload);
|
||||
|
||||
case OpType.Repair:
|
||||
// Repair operations are internally generated - skip validation
|
||||
return { success: true };
|
||||
|
||||
default:
|
||||
PFLog.warn(
|
||||
`[ValidateOperationPayload] Unknown opType: ${op.opType}, allowing through`,
|
||||
);
|
||||
return { success: true };
|
||||
}
|
||||
};
|
||||
317
src/app/core/persistence/operation-log/validate-state.service.ts
Normal file
317
src/app/core/persistence/operation-log/validate-state.service.ts
Normal file
|
|
@ -0,0 +1,317 @@
|
|||
import { Injectable } from '@angular/core';
|
||||
import { validateAllData } from '../../../pfapi/validate/validation-fn';
|
||||
import {
|
||||
isRelatedModelDataValid,
|
||||
getLastValidityError,
|
||||
} from '../../../pfapi/validate/is-related-model-data-valid';
|
||||
import { dataRepair } from '../../../pfapi/repair/data-repair';
|
||||
import { isDataRepairPossible } from '../../../pfapi/repair/is-data-repair-possible.util';
|
||||
import { AppDataCompleteNew } from '../../../pfapi/pfapi-config';
|
||||
import { RepairSummary } from './operation.types';
|
||||
import { PFLog } from '../../log';
|
||||
import { RepairOperationService } from './repair-operation.service';
|
||||
|
||||
/**
|
||||
* Result of validating application state.
|
||||
*/
|
||||
export interface StateValidationResult {
|
||||
isValid: boolean;
|
||||
typiaErrors: unknown[];
|
||||
crossModelError?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of validating and repairing application state.
|
||||
*/
|
||||
export interface ValidateAndRepairResult {
|
||||
isValid: boolean;
|
||||
wasRepaired: boolean;
|
||||
repairedState?: AppDataCompleteNew;
|
||||
repairSummary?: RepairSummary;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Service for validating and repairing application state.
|
||||
* Wraps PFAPI's validation (Typia + cross-model) and repair functionality.
|
||||
*
|
||||
* Validation happens at key checkpoints:
|
||||
* - Checkpoint B: After loading snapshot during hydration
|
||||
* - Checkpoint C: After replaying tail operations during hydration
|
||||
* - Checkpoint D: After applying remote operations during sync
|
||||
*/
|
||||
@Injectable({
|
||||
providedIn: 'root',
|
||||
})
|
||||
export class ValidateStateService {
|
||||
/**
|
||||
* Validates application state using both Typia schema validation
|
||||
* and cross-model relationship validation.
|
||||
*/
|
||||
validateState(state: AppDataCompleteNew): StateValidationResult {
|
||||
const result: StateValidationResult = {
|
||||
isValid: true,
|
||||
typiaErrors: [],
|
||||
};
|
||||
|
||||
// 1. Run Typia schema validation
|
||||
const typiaResult = validateAllData(state);
|
||||
if (!typiaResult.success) {
|
||||
result.isValid = false;
|
||||
result.typiaErrors = typiaResult.errors || [];
|
||||
PFLog.warn('[ValidateStateService] Typia validation failed', {
|
||||
errorCount: result.typiaErrors.length,
|
||||
});
|
||||
}
|
||||
|
||||
// 2. Run cross-model relationship validation
|
||||
const isRelatedValid = isRelatedModelDataValid(state);
|
||||
if (!isRelatedValid) {
|
||||
result.isValid = false;
|
||||
result.crossModelError = getLastValidityError();
|
||||
PFLog.warn('[ValidateStateService] Cross-model validation failed', {
|
||||
error: result.crossModelError,
|
||||
});
|
||||
}
|
||||
|
||||
if (result.isValid) {
|
||||
PFLog.normal('[ValidateStateService] State validation passed');
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates state and repairs if necessary.
|
||||
* Returns the (possibly repaired) state and repair summary.
|
||||
*/
|
||||
validateAndRepair(state: AppDataCompleteNew): ValidateAndRepairResult {
|
||||
// First, validate the state
|
||||
const validationResult = this.validateState(state);
|
||||
|
||||
if (validationResult.isValid) {
|
||||
return {
|
||||
isValid: true,
|
||||
wasRepaired: false,
|
||||
};
|
||||
}
|
||||
|
||||
// State is invalid - attempt repair
|
||||
PFLog.log('[ValidateStateService] State invalid, attempting repair...');
|
||||
|
||||
// Check if repair is possible
|
||||
if (!isDataRepairPossible(state)) {
|
||||
PFLog.err('[ValidateStateService] Data repair not possible - state too corrupted');
|
||||
return {
|
||||
isValid: false,
|
||||
wasRepaired: false,
|
||||
error: 'Data repair not possible - state too corrupted',
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
// Run repair
|
||||
const typiaErrors = validationResult.typiaErrors as any[];
|
||||
const repairedState = dataRepair(state, typiaErrors);
|
||||
|
||||
// Create repair summary based on validation errors
|
||||
const repairSummary = this._createRepairSummary(
|
||||
validationResult,
|
||||
state,
|
||||
repairedState,
|
||||
);
|
||||
|
||||
// Validate the repaired state to confirm it's now valid
|
||||
const revalidationResult = this.validateState(repairedState);
|
||||
if (!revalidationResult.isValid) {
|
||||
PFLog.err('[ValidateStateService] State still invalid after repair');
|
||||
return {
|
||||
isValid: false,
|
||||
wasRepaired: true,
|
||||
repairedState,
|
||||
repairSummary,
|
||||
error: 'State still invalid after repair',
|
||||
};
|
||||
}
|
||||
|
||||
PFLog.log('[ValidateStateService] State successfully repaired', {
|
||||
repairSummary,
|
||||
});
|
||||
|
||||
return {
|
||||
isValid: true,
|
||||
wasRepaired: true,
|
||||
repairedState,
|
||||
repairSummary,
|
||||
};
|
||||
} catch (e) {
|
||||
PFLog.err('[ValidateStateService] Error during repair', e);
|
||||
return {
|
||||
isValid: false,
|
||||
wasRepaired: false,
|
||||
error: `Repair failed: ${e instanceof Error ? e.message : String(e)}`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a repair summary by analyzing what changed between original and repaired state.
|
||||
* This is an approximation based on what we can detect changed.
|
||||
*/
|
||||
private _createRepairSummary(
|
||||
validationResult: StateValidationResult,
|
||||
original: AppDataCompleteNew,
|
||||
repaired: AppDataCompleteNew,
|
||||
): RepairSummary {
|
||||
const summary = RepairOperationService.createEmptyRepairSummary();
|
||||
|
||||
// Count typia errors as type errors fixed
|
||||
summary.typeErrorsFixed = validationResult.typiaErrors.length;
|
||||
|
||||
// Detect entity state changes (ids array sync)
|
||||
summary.entityStateFixed = this._countEntityStateChanges(original, repaired);
|
||||
|
||||
// Detect relationship fixes by looking at task/project/tag counts
|
||||
summary.relationshipsFixed = this._countRelationshipChanges(original, repaired);
|
||||
|
||||
// Detect orphaned entity changes
|
||||
summary.orphanedEntitiesRestored = this._countOrphanedEntityChanges(
|
||||
original,
|
||||
repaired,
|
||||
);
|
||||
|
||||
// Detect invalid reference removals
|
||||
summary.invalidReferencesRemoved = this._countInvalidReferenceRemovals(
|
||||
original,
|
||||
repaired,
|
||||
);
|
||||
|
||||
// Structure repairs (menu tree, inbox project)
|
||||
summary.structureRepaired = this._countStructureRepairs(original, repaired);
|
||||
|
||||
return summary;
|
||||
}
|
||||
|
||||
/**
|
||||
* Counts changes in entity state consistency (ids array matching entities object).
|
||||
*/
|
||||
private _countEntityStateChanges(
|
||||
original: AppDataCompleteNew,
|
||||
repaired: AppDataCompleteNew,
|
||||
): number {
|
||||
let count = 0;
|
||||
const models = ['task', 'project', 'tag', 'note', 'simpleCounter'] as const;
|
||||
|
||||
for (const model of models) {
|
||||
const origIds = (original[model] as any)?.ids?.length || 0;
|
||||
const repairedIds = (repaired[model] as any)?.ids?.length || 0;
|
||||
if (origIds !== repairedIds) {
|
||||
count += Math.abs(origIds - repairedIds);
|
||||
}
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Counts relationship changes (task-project, task-tag assignments).
|
||||
*/
|
||||
private _countRelationshipChanges(
|
||||
original: AppDataCompleteNew,
|
||||
repaired: AppDataCompleteNew,
|
||||
): number {
|
||||
let count = 0;
|
||||
|
||||
// Check if task-project assignments changed
|
||||
const origTasks = Object.values((original.task as any)?.entities || {});
|
||||
|
||||
for (const origTask of origTasks as any[]) {
|
||||
const repairedTask = (repaired.task as any)?.entities?.[origTask.id];
|
||||
if (repairedTask) {
|
||||
if (origTask.projectId !== repairedTask.projectId) {
|
||||
count++;
|
||||
}
|
||||
// Check tag changes
|
||||
const origTags = origTask.tagIds || [];
|
||||
const repairedTags = repairedTask.tagIds || [];
|
||||
if (origTags.length !== repairedTags.length) {
|
||||
count++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Counts orphaned entity restorations.
|
||||
*/
|
||||
private _countOrphanedEntityChanges(
|
||||
original: AppDataCompleteNew,
|
||||
repaired: AppDataCompleteNew,
|
||||
): number {
|
||||
let count = 0;
|
||||
|
||||
// Check archive changes
|
||||
const origArchiveCount =
|
||||
((original.archiveYoung?.task as any)?.ids?.length || 0) +
|
||||
((original.archiveOld?.task as any)?.ids?.length || 0);
|
||||
const repairedArchiveCount =
|
||||
((repaired.archiveYoung?.task as any)?.ids?.length || 0) +
|
||||
((repaired.archiveOld?.task as any)?.ids?.length || 0);
|
||||
|
||||
if (origArchiveCount !== repairedArchiveCount) {
|
||||
count += Math.abs(origArchiveCount - repairedArchiveCount);
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Counts invalid reference removals.
|
||||
*/
|
||||
private _countInvalidReferenceRemovals(
|
||||
original: AppDataCompleteNew,
|
||||
repaired: AppDataCompleteNew,
|
||||
): number {
|
||||
let count = 0;
|
||||
|
||||
// Check for tasks with removed projectIds
|
||||
const origTasks = Object.values((original.task as any)?.entities || {});
|
||||
|
||||
for (const origTask of origTasks as any[]) {
|
||||
const repairedTask = (repaired.task as any)?.entities?.[origTask.id];
|
||||
if (!repairedTask && origTask.projectId) {
|
||||
count++;
|
||||
}
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Counts structure repairs (menu tree, inbox project creation).
|
||||
*/
|
||||
private _countStructureRepairs(
|
||||
original: AppDataCompleteNew,
|
||||
repaired: AppDataCompleteNew,
|
||||
): number {
|
||||
let count = 0;
|
||||
|
||||
// Check if inbox project was created
|
||||
const origProjectCount = (original.project as any)?.ids?.length || 0;
|
||||
const repairedProjectCount = (repaired.project as any)?.ids?.length || 0;
|
||||
if (repairedProjectCount > origProjectCount) {
|
||||
count++;
|
||||
}
|
||||
|
||||
// Check menu tree changes
|
||||
const origMenuItems = (original.menuTree as any)?.items?.length || 0;
|
||||
const repairedMenuItems = (repaired.menuTree as any)?.items?.length || 0;
|
||||
if (origMenuItems !== repairedMenuItems) {
|
||||
count++;
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
}
|
||||
|
|
@ -1194,6 +1194,8 @@ const T = {
|
|||
HYDRATION_FAILED: 'F.SYNC.S.HYDRATION_FAILED',
|
||||
COMPACTION_FAILED: 'F.SYNC.S.COMPACTION_FAILED',
|
||||
CONFLICT_RESOLUTION_FAILED: 'F.SYNC.S.CONFLICT_RESOLUTION_FAILED',
|
||||
DATA_REPAIRED: 'F.SYNC.S.DATA_REPAIRED',
|
||||
INVALID_OPERATION_PAYLOAD: 'F.SYNC.S.INVALID_OPERATION_PAYLOAD',
|
||||
UNKNOWN_ERROR: 'F.SYNC.S.UNKNOWN_ERROR',
|
||||
UPLOAD_ERROR: 'F.SYNC.S.UPLOAD_ERROR',
|
||||
},
|
||||
|
|
|
|||
|
|
@ -1176,6 +1176,8 @@
|
|||
"HYDRATION_FAILED": "Failed to load data. Please reload the app.",
|
||||
"COMPACTION_FAILED": "Database cleanup failed. App may slow down.",
|
||||
"CONFLICT_RESOLUTION_FAILED": "Sync conflict resolution failed. Please reload.",
|
||||
"DATA_REPAIRED": "Data automatically repaired ({{count}} issues fixed)",
|
||||
"INVALID_OPERATION_PAYLOAD": "Invalid operation payload. Action not saved.",
|
||||
"UNKNOWN_ERROR": "Unknown Sync Error: {{err}}",
|
||||
"UPLOAD_ERROR": "Unknown Upload Error (Settings correct?): {{err}}"
|
||||
},
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue