mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-07-18 00:46:45 +00:00
refactor(sync-core): configure full-state op classification
This commit is contained in:
parent
00658ac67a
commit
d2be63c9c4
11 changed files with 376 additions and 111 deletions
|
|
@ -1,9 +1,9 @@
|
|||
# `@sp/sync-core` Extraction Plan
|
||||
|
||||
> **Status: In progress - PR 1, PR 2 guardrails/logger adapter work, PR 3a
|
||||
> vector-clock ownership, and the PR 3b pure helper slices are present on this
|
||||
> branch. Remaining PR 2 cleanup is PR text alignment plus targeted future
|
||||
> `SyncLogger` routing for files as they move.**
|
||||
> vector-clock ownership, full-state op classification config, and the PR 3b
|
||||
> pure helper slices are present on this branch. Remaining cleanup is PR text
|
||||
> alignment plus targeted future `SyncLogger` routing for files as they move.**
|
||||
|
||||
**Goal:** Carve the sync engine out of `src/app/op-log/` into a reusable,
|
||||
framework-agnostic, **domain-agnostic** `@sp/sync-core` package, plus a sibling
|
||||
|
|
@ -103,15 +103,22 @@ groundwork:
|
|||
- `eslint.config.js` applies `no-restricted-imports` and a dynamic-import ban to
|
||||
`packages/sync-core/**/*.ts`.
|
||||
- The package currently exports operation primitives, apply types, LWW helper
|
||||
factory, entity-key helpers, `SyncStateCorruptedError`, entity-registry
|
||||
contracts, and the privacy-aware logger port.
|
||||
factory, full-state op-type helper factory, entity-key helpers,
|
||||
`SyncStateCorruptedError`, entity-registry contracts, and the privacy-aware
|
||||
logger port.
|
||||
- The app registry now has `buildEntityRegistry()` and an `ENTITY_REGISTRY`
|
||||
injection token. Existing helper functions still read the app-side
|
||||
`ENTITY_CONFIGS` singleton for compatibility.
|
||||
|
||||
Current extraction state and remaining immediate debt:
|
||||
|
||||
- `FULL_STATE_OP_TYPES` still lives in `@sp/sync-core` as compatibility debt.
|
||||
- Full-state operation classification is now host-configured via
|
||||
`createFullStateOpTypeHelpers()`. The SP-facing
|
||||
`src/app/op-log/core/operation.types.ts` shim instantiates its own
|
||||
`FULL_STATE_OP_TYPES` and `isFullStateOpType`; the package root keeps
|
||||
deprecated SP compatibility exports for existing consumers. `OpType.SyncImport`,
|
||||
`OpType.BackupImport`, and `OpType.Repair` remain in `@sp/sync-core` only as
|
||||
host-defined compatibility strings.
|
||||
- Vector-clock compare/merge/prune now lives in `@sp/sync-core`, with
|
||||
`@sp/shared-schema` re-exporting it for existing client/server imports.
|
||||
- `SyncLogger` exists, but movable app code still mostly calls `OpLog` directly.
|
||||
|
|
@ -137,14 +144,12 @@ Current extraction state and remaining immediate debt:
|
|||
Suggested next order:
|
||||
|
||||
1. Finish PR 2 documentation and verification.
|
||||
2. Add the app-side `SyncLogger` adapter before moving `OpLog`-using files.
|
||||
2. Continue targeted `SyncLogger` routing for files as they become movable.
|
||||
3. Treat the PR 3b pure conflict-resolution and sync-import slices as complete
|
||||
for this round.
|
||||
4. Make full-state operation classification configurable or explicitly mark
|
||||
those op types as host-defined compatibility strings.
|
||||
5. Clean up logger/config dependencies before moving compression, prefix, or
|
||||
4. Clean up logger/config dependencies before moving compression, prefix, or
|
||||
error helpers.
|
||||
6. Defer port/orchestration work until the remaining pure/config boundaries are
|
||||
5. Defer port/orchestration work until the remaining pure/config boundaries are
|
||||
settled.
|
||||
|
||||
## PR 1 - Thin First Slice (#7546)
|
||||
|
|
@ -176,8 +181,15 @@ Source: `packages/sync-core/src/`. All exports come through `index.ts`.
|
|||
- `OperationLogEntry`, `EntityConflict`, `ConflictResult`, `EntityChange`,
|
||||
`MultiEntityPayload`.
|
||||
- `VectorClock = Record<string, number>`.
|
||||
- `FULL_STATE_OP_TYPES`, `isFullStateOpType`, `isMultiEntityPayload`,
|
||||
`extractActionPayload`.
|
||||
- `isMultiEntityPayload`, `extractActionPayload`.
|
||||
|
||||
**Full-state op-type helper factory** (`full-state-op-types.ts`):
|
||||
|
||||
- `createFullStateOpTypeHelpers<TOpType>(fullStateOpTypes)` returns the
|
||||
host-owned `FULL_STATE_OP_TYPES` set and `isFullStateOpType` predicate.
|
||||
- The package keeps deprecated SP compatibility exports for
|
||||
`FULL_STATE_OP_TYPES` / `isFullStateOpType`, but reusable hosts should
|
||||
instantiate their own helper instead of using those defaults.
|
||||
|
||||
**LWW factory** (`lww-update-action-types.ts`):
|
||||
|
||||
|
|
@ -202,7 +214,8 @@ Each previously-public symbol path keeps working via thin shims:
|
|||
|
||||
- `src/app/op-log/core/operation.types.ts` re-exports generic symbols and
|
||||
redeclares SP-narrowed `Operation`, `OperationLogEntry`, `EntityChange`,
|
||||
`EntityConflict`, `ConflictResult`, and `MultiEntityPayload`.
|
||||
`EntityConflict`, `ConflictResult`, and `MultiEntityPayload`. It also
|
||||
instantiates `createFullStateOpTypeHelpers()` with SP's full-state op strings.
|
||||
- `src/app/op-log/core/types/apply.types.ts` redeclares app-narrowed apply
|
||||
result/options types.
|
||||
- `src/app/op-log/core/lww-update-action-types.ts` instantiates the LWW helper
|
||||
|
|
@ -222,8 +235,8 @@ Each previously-public symbol path keeps working via thin shims:
|
|||
- Fix comments that imply `sync-core` depends on `shared-schema`. The build may
|
||||
run after `shared-schema`, but the package dependency direction must remain
|
||||
absent.
|
||||
- Decide whether `FULL_STATE_OP_TYPES` is acceptable compatibility debt for PR 1
|
||||
or whether it should already become app-configurable.
|
||||
- Resolved in follow-up: `FULL_STATE_OP_TYPES` is now app-configured via
|
||||
`createFullStateOpTypeHelpers()`.
|
||||
|
||||
### Verification
|
||||
|
||||
|
|
@ -277,6 +290,8 @@ Already present:
|
|||
- `EncryptAndCompressHandlerService` now accepts a `SyncLogger` constructor
|
||||
argument and uses the app adapter by default, proving the direct-constructor
|
||||
path for package-level classes without changing sync behavior.
|
||||
- `op-log/encryption/compression-handler.ts` now routes compression failures
|
||||
through `SyncLogger` + `toSyncLogError()` and logs only safe length metadata.
|
||||
- A deliberate bad-import check was run with a temporary
|
||||
`packages/sync-core/src/__boundary-check__.ts` importing `@angular/core`;
|
||||
`npm run lint:file -- packages/sync-core/src/__boundary-check__.ts` failed on
|
||||
|
|
@ -391,9 +406,9 @@ Initial candidate-file audit:
|
|||
|
||||
- `op-log/encryption/encrypt-and-compress-handler.service.ts`: safe prefix and
|
||||
flag metadata now goes through `SyncLogger`.
|
||||
- `op-log/encryption/compression-handler.ts`: still logs raw errors directly;
|
||||
before moving, route failures through `toSyncLogError()` and preserve only
|
||||
safe counts such as compressed input length.
|
||||
- `op-log/encryption/compression-handler.ts`: routes failures through
|
||||
`toSyncLogError()` and preserves only safe counts such as input length. It
|
||||
remains app-side until compression errors are split from app diagnostics.
|
||||
- `op-log/core/errors/sync-errors.ts`: not move-ready. Several constructors log
|
||||
validation params, raw samples, or additional objects; split generic error
|
||||
classes from app-specific diagnostics before routing through `SyncLogger`.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { Operation } from './operation.types';
|
||||
import type { Operation } from './operation.types';
|
||||
|
||||
/**
|
||||
* Result of applying a batch of operations.
|
||||
|
|
@ -6,16 +6,16 @@ import { Operation } from './operation.types';
|
|||
* Allows callers to handle partial success scenarios where some operations
|
||||
* were applied before an error occurred.
|
||||
*/
|
||||
export interface ApplyOperationsResult {
|
||||
export interface ApplyOperationsResult<TOperation extends Operation<string> = Operation> {
|
||||
/** Operations that were successfully applied. */
|
||||
appliedOps: Operation[];
|
||||
appliedOps: TOperation[];
|
||||
|
||||
/**
|
||||
* If an error occurred, this contains the failed operation and the error.
|
||||
* Operations after this one in the batch were NOT applied.
|
||||
*/
|
||||
failedOp?: {
|
||||
op: Operation;
|
||||
op: TOperation;
|
||||
error: Error;
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -24,7 +24,7 @@ export interface DeepEqualOptions {
|
|||
}
|
||||
|
||||
export interface EntityFrontierContext {
|
||||
localOpsForEntity: Pick<Operation, 'vectorClock'>[];
|
||||
localOpsForEntity: Pick<Operation<string>, 'vectorClock'>[];
|
||||
appliedFrontier: VectorClock | undefined;
|
||||
snapshotVectorClock: VectorClock | undefined;
|
||||
snapshotEntityKeys: ReadonlySet<string> | undefined;
|
||||
|
|
@ -41,7 +41,7 @@ export interface ClockCorruptionAdjustmentOptions {
|
|||
}
|
||||
|
||||
export interface LwwConflictResolutionPlan<
|
||||
TConflict extends EntityConflict = EntityConflict,
|
||||
TConflict extends EntityConflictLike<Operation<string>> = EntityConflict,
|
||||
> {
|
||||
conflict: TConflict;
|
||||
winner: LwwConflictResolutionWinner;
|
||||
|
|
@ -51,25 +51,29 @@ export interface LwwConflictResolutionPlan<
|
|||
remoteMaxTimestamp?: number;
|
||||
}
|
||||
|
||||
export interface LwwConflictResolutionPlanningOptions {
|
||||
isArchiveAction: (op: Operation) => boolean;
|
||||
export interface LwwConflictResolutionPlanningOptions<
|
||||
TOperation extends Operation<string> = Operation,
|
||||
> {
|
||||
isArchiveAction: (op: TOperation) => boolean;
|
||||
toEntityKey?: (entityType: string, entityId: string) => string;
|
||||
}
|
||||
|
||||
export interface LocalDeleteRemoteUpdateConversionOptions {
|
||||
export interface LocalDeleteRemoteUpdateConversionOptions<
|
||||
TOperation extends Operation<string> = Operation,
|
||||
> {
|
||||
payloadKey: string | ((entityType: string) => string);
|
||||
toLwwUpdateActionType: (entityType: string) => string;
|
||||
isSingletonEntityId?: (entityId: string) => boolean;
|
||||
onMissingBaseEntity?: (ctx: {
|
||||
conflict: EntityConflict;
|
||||
localDeleteOp: Operation | undefined;
|
||||
remoteOp: Operation;
|
||||
conflict: EntityConflictLike<TOperation>;
|
||||
localDeleteOp: TOperation | undefined;
|
||||
remoteOp: TOperation;
|
||||
localDeletePayloadKeys: string[] | undefined;
|
||||
}) => void;
|
||||
}
|
||||
|
||||
export type EntityConflictLike<TOperation extends Operation> = Omit<
|
||||
EntityConflict,
|
||||
export type EntityConflictLike<TOperation extends Operation<string>> = Omit<
|
||||
EntityConflict<Operation<string>>,
|
||||
'localOps' | 'remoteOps'
|
||||
> & {
|
||||
localOps: TOperation[];
|
||||
|
|
@ -77,7 +81,7 @@ export type EntityConflictLike<TOperation extends Operation> = Omit<
|
|||
};
|
||||
|
||||
export interface LwwResolvedConflict<
|
||||
TOperation extends Operation = Operation,
|
||||
TOperation extends Operation<string> = Operation,
|
||||
TConflict extends EntityConflictLike<TOperation> = EntityConflictLike<TOperation>,
|
||||
> {
|
||||
conflict: TConflict;
|
||||
|
|
@ -86,14 +90,16 @@ export interface LwwResolvedConflict<
|
|||
}
|
||||
|
||||
export interface LwwResolutionPartitionOptions<
|
||||
TOperation extends Operation,
|
||||
TOperation extends Operation<string>,
|
||||
TConflict extends EntityConflictLike<TOperation> = EntityConflictLike<TOperation>,
|
||||
> {
|
||||
processRemoteWinnerOps?: (conflict: TConflict) => TOperation[];
|
||||
toEntityKey?: (entityType: string, entityId: string) => string;
|
||||
}
|
||||
|
||||
export interface LwwResolutionPartitions<TOperation extends Operation = Operation> {
|
||||
export interface LwwResolutionPartitions<
|
||||
TOperation extends Operation<string> = Operation,
|
||||
> {
|
||||
localWinsCount: number;
|
||||
remoteWinsCount: number;
|
||||
remoteWinsOps: TOperation[];
|
||||
|
|
@ -106,7 +112,7 @@ export interface LwwResolutionPartitions<TOperation extends Operation = Operatio
|
|||
|
||||
const resolvePayloadKey = (
|
||||
entityType: string,
|
||||
payloadKey: LocalDeleteRemoteUpdateConversionOptions['payloadKey'],
|
||||
payloadKey: LocalDeleteRemoteUpdateConversionOptions<Operation<string>>['payloadKey'],
|
||||
): string => (typeof payloadKey === 'function' ? payloadKey(entityType) : payloadKey);
|
||||
|
||||
const getPayloadKeys = (payload: unknown): string[] | undefined => {
|
||||
|
|
@ -166,10 +172,10 @@ export const extractUpdateChanges = (
|
|||
* and singleton-id semantics so the core remains domain-agnostic.
|
||||
*/
|
||||
export const convertLocalDeleteRemoteUpdatesToLww = <
|
||||
TOperation extends Operation = Operation,
|
||||
TOperation extends Operation<string> = Operation,
|
||||
>(
|
||||
conflict: EntityConflictLike<TOperation>,
|
||||
options: LocalDeleteRemoteUpdateConversionOptions,
|
||||
options: LocalDeleteRemoteUpdateConversionOptions<TOperation>,
|
||||
): TOperation[] => {
|
||||
const localDeleteOp = conflict.localOps.find((op) => op.opType === OpType.Delete);
|
||||
|
||||
|
|
@ -281,7 +287,7 @@ export const deepEqual = (
|
|||
* the same resulting state.
|
||||
*/
|
||||
export const isIdenticalConflict = (
|
||||
conflict: EntityConflict,
|
||||
conflict: EntityConflict<Operation<string>>,
|
||||
logger: SyncLogger = NOOP_SYNC_LOGGER,
|
||||
): boolean => {
|
||||
const { localOps, remoteOps } = conflict;
|
||||
|
|
@ -324,9 +330,9 @@ export const isIdenticalConflict = (
|
|||
/**
|
||||
* Suggests a conservative default conflict resolution for the UI/orchestrator.
|
||||
*/
|
||||
export const suggestConflictResolution = (
|
||||
localOps: Operation[],
|
||||
remoteOps: Operation[],
|
||||
export const suggestConflictResolution = <TOperation extends Operation<string>>(
|
||||
localOps: TOperation[],
|
||||
remoteOps: TOperation[],
|
||||
): ConflictResolutionSuggestion => {
|
||||
if (localOps.length === 0) return 'remote';
|
||||
if (remoteOps.length === 0) return 'local';
|
||||
|
|
@ -363,10 +369,11 @@ export const suggestConflictResolution = (
|
|||
* be created and which app-side factory should create it.
|
||||
*/
|
||||
export const planLwwConflictResolutions = <
|
||||
TConflict extends EntityConflict = EntityConflict,
|
||||
TOperation extends Operation<string> = Operation,
|
||||
TConflict extends EntityConflictLike<TOperation> = EntityConflictLike<TOperation>,
|
||||
>(
|
||||
conflicts: TConflict[],
|
||||
options: LwwConflictResolutionPlanningOptions,
|
||||
options: LwwConflictResolutionPlanningOptions<TOperation>,
|
||||
): Array<LwwConflictResolutionPlan<TConflict>> => {
|
||||
const toEntityKey =
|
||||
options.toEntityKey ??
|
||||
|
|
@ -443,7 +450,7 @@ export const planLwwConflictResolutions = <
|
|||
* pending local ops are superseded.
|
||||
*/
|
||||
export const partitionLwwResolutions = <
|
||||
TOperation extends Operation = Operation,
|
||||
TOperation extends Operation<string> = Operation,
|
||||
TConflict extends EntityConflictLike<TOperation> = EntityConflictLike<TOperation>,
|
||||
TResolution extends LwwResolvedConflict<TOperation, TConflict> = LwwResolvedConflict<
|
||||
TOperation,
|
||||
|
|
|
|||
43
packages/sync-core/src/full-state-op-types.ts
Normal file
43
packages/sync-core/src/full-state-op-types.ts
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
import { OpType } from './operation.types';
|
||||
|
||||
export interface FullStateOpTypeHelpers<TOpType extends string = string> {
|
||||
FULL_STATE_OP_TYPES: ReadonlySet<TOpType>;
|
||||
isFullStateOpType: (opType: string) => opType is TOpType;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates host-owned helpers for classifying operations that replace full state.
|
||||
*
|
||||
* Full-state operation names are application wire conventions, so sync-core
|
||||
* only provides the classification helper and the host supplies the op strings.
|
||||
*/
|
||||
export const createFullStateOpTypeHelpers = <TOpType extends string>(
|
||||
fullStateOpTypes: readonly TOpType[],
|
||||
): FullStateOpTypeHelpers<TOpType> => {
|
||||
const fullStateOpTypeSet: ReadonlySet<TOpType> = new Set(fullStateOpTypes);
|
||||
|
||||
return {
|
||||
FULL_STATE_OP_TYPES: fullStateOpTypeSet,
|
||||
isFullStateOpType: (opType: string): opType is TOpType =>
|
||||
fullStateOpTypeSet.has(opType as TOpType),
|
||||
};
|
||||
};
|
||||
|
||||
const compatibilityFullStateOpTypeHelpers = createFullStateOpTypeHelpers<OpType>([
|
||||
OpType.SyncImport,
|
||||
OpType.BackupImport,
|
||||
OpType.Repair,
|
||||
]);
|
||||
|
||||
/**
|
||||
* @deprecated Super Productivity compatibility export. Hosts should instantiate
|
||||
* their own helper with `createFullStateOpTypeHelpers()`.
|
||||
*/
|
||||
export const FULL_STATE_OP_TYPES =
|
||||
compatibilityFullStateOpTypeHelpers.FULL_STATE_OP_TYPES;
|
||||
|
||||
/**
|
||||
* @deprecated Super Productivity compatibility export. Hosts should instantiate
|
||||
* their own helper with `createFullStateOpTypeHelpers()`.
|
||||
*/
|
||||
export const isFullStateOpType = compatibilityFullStateOpTypeHelpers.isFullStateOpType;
|
||||
|
|
@ -1,11 +1,5 @@
|
|||
// Operation log primitives — the generic, app-agnostic core of the sync engine.
|
||||
export {
|
||||
OpType,
|
||||
FULL_STATE_OP_TYPES,
|
||||
isFullStateOpType,
|
||||
isMultiEntityPayload,
|
||||
extractActionPayload,
|
||||
} from './operation.types';
|
||||
export { OpType, isMultiEntityPayload, extractActionPayload } from './operation.types';
|
||||
export type {
|
||||
VectorClock,
|
||||
Operation,
|
||||
|
|
@ -35,6 +29,14 @@ export type {
|
|||
SyncImportFilterKeepReason,
|
||||
} from './sync-import-filter';
|
||||
|
||||
// Full-state operation classification helper. Hosts supply their own op strings.
|
||||
export {
|
||||
FULL_STATE_OP_TYPES,
|
||||
createFullStateOpTypeHelpers,
|
||||
isFullStateOpType,
|
||||
} from './full-state-op-types';
|
||||
export type { FullStateOpTypeHelpers } from './full-state-op-types';
|
||||
|
||||
// LWW (Last-Writer-Wins) update action-type helpers — factory parameterized by
|
||||
// the host application's entity-type list, so the lib stays domain-agnostic.
|
||||
export { createLwwUpdateActionTypeHelpers } from './lww-update-action-types';
|
||||
|
|
|
|||
|
|
@ -8,15 +8,24 @@ export enum OpType {
|
|||
Delete = 'DEL',
|
||||
Move = 'MOV', // For list reordering
|
||||
Batch = 'BATCH', // For bulk operations (import, mass update)
|
||||
/** Replaces entire app state from remote sync. All concurrent ops are discarded. */
|
||||
/**
|
||||
* Host-defined full-state compatibility string used by Super Productivity.
|
||||
* Reusable core logic must receive full-state classification from the host.
|
||||
*/
|
||||
SyncImport = 'SYNC_IMPORT',
|
||||
/** Replaces entire app state from backup file. All concurrent ops are discarded. */
|
||||
/**
|
||||
* Host-defined full-state compatibility string used by Super Productivity.
|
||||
* Reusable core logic must receive full-state classification from the host.
|
||||
*/
|
||||
BackupImport = 'BACKUP_IMPORT',
|
||||
/** Auto-repair operation containing full repaired state. */
|
||||
/**
|
||||
* Host-defined full-state compatibility string used by Super Productivity.
|
||||
* Reusable core logic must receive full-state classification from the host.
|
||||
*/
|
||||
Repair = 'REPAIR',
|
||||
}
|
||||
|
||||
export interface Operation {
|
||||
export interface Operation<TOpType extends string = OpType> {
|
||||
/**
|
||||
* Unique identifier for the operation.
|
||||
* Should be a UUID v7 (time-ordered) to allow for rough chronological sorting
|
||||
|
|
@ -36,7 +45,7 @@ export interface Operation {
|
|||
* High-level operation category (Create, Update, Delete, etc.).
|
||||
* Used for broad logic handling like persistence strategies or conflict resolution rules.
|
||||
*/
|
||||
opType: OpType;
|
||||
opType: TOpType;
|
||||
|
||||
// SCOPE
|
||||
/**
|
||||
|
|
@ -92,7 +101,7 @@ export interface Operation {
|
|||
schemaVersion: number;
|
||||
}
|
||||
|
||||
export interface OperationLogEntry {
|
||||
export interface OperationLogEntry<TOperation extends Operation<string> = Operation> {
|
||||
/**
|
||||
* Local, monotonic auto-increment integer (IndexedDB primary key).
|
||||
* Strictly orders operations as they arrived or were generated on THIS specific device.
|
||||
|
|
@ -102,7 +111,7 @@ export interface OperationLogEntry {
|
|||
/**
|
||||
* The operation data itself (the synchronized part).
|
||||
*/
|
||||
op: Operation;
|
||||
op: TOperation;
|
||||
|
||||
/**
|
||||
* Local timestamp (epoch ms) indicating when this operation was written to the LOCAL database.
|
||||
|
|
@ -141,39 +150,20 @@ export interface OperationLogEntry {
|
|||
retryCount?: number;
|
||||
}
|
||||
|
||||
export interface EntityConflict {
|
||||
export interface EntityConflict<TOperation extends Operation<string> = Operation> {
|
||||
entityType: string;
|
||||
entityId: string;
|
||||
localOps: Operation[];
|
||||
remoteOps: Operation[];
|
||||
localOps: TOperation[];
|
||||
remoteOps: TOperation[];
|
||||
suggestedResolution: 'local' | 'remote' | 'merge' | 'manual';
|
||||
mergedPayload?: unknown;
|
||||
}
|
||||
|
||||
export interface ConflictResult {
|
||||
nonConflicting: Operation[];
|
||||
conflicts: EntityConflict[];
|
||||
export interface ConflictResult<TOperation extends Operation<string> = Operation> {
|
||||
nonConflicting: TOperation[];
|
||||
conflicts: EntityConflict<TOperation>[];
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// FULL-STATE OPERATION HELPERS
|
||||
// =============================================================================
|
||||
|
||||
/**
|
||||
* OpTypes that contain full application state in their payload.
|
||||
*/
|
||||
export const FULL_STATE_OP_TYPES = new Set<OpType>([
|
||||
OpType.SyncImport,
|
||||
OpType.BackupImport,
|
||||
OpType.Repair,
|
||||
]);
|
||||
|
||||
/**
|
||||
* Type guard to check if an operation is a full-state operation.
|
||||
*/
|
||||
export const isFullStateOpType = (opType: OpType | string): boolean =>
|
||||
FULL_STATE_OP_TYPES.has(opType as OpType);
|
||||
|
||||
// =============================================================================
|
||||
// MULTI-ENTITY OPERATIONS
|
||||
// =============================================================================
|
||||
|
|
@ -181,10 +171,10 @@ export const isFullStateOpType = (opType: OpType | string): boolean =>
|
|||
/**
|
||||
* Represents a single entity change within a multi-entity operation.
|
||||
*/
|
||||
export interface EntityChange {
|
||||
export interface EntityChange<TOpType extends string = OpType> {
|
||||
entityType: string;
|
||||
entityId: string;
|
||||
opType: OpType;
|
||||
opType: TOpType;
|
||||
/**
|
||||
* The actual changes:
|
||||
* - For Create: Full entity object
|
||||
|
|
@ -198,9 +188,9 @@ export interface EntityChange {
|
|||
* Payload wrapper for multi-entity operations.
|
||||
* Contains the original action payload plus all entity changes computed from state diff.
|
||||
*/
|
||||
export interface MultiEntityPayload {
|
||||
export interface MultiEntityPayload<TOpType extends string = OpType> {
|
||||
actionPayload: Record<string, unknown>;
|
||||
entityChanges: EntityChange[];
|
||||
entityChanges: EntityChange<TOpType>[];
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
104
packages/sync-core/tests/full-state-op-types.spec.ts
Normal file
104
packages/sync-core/tests/full-state-op-types.spec.ts
Normal file
|
|
@ -0,0 +1,104 @@
|
|||
import { describe, expect, it } from 'vitest';
|
||||
import {
|
||||
FULL_STATE_OP_TYPES,
|
||||
createFullStateOpTypeHelpers,
|
||||
isFullStateOpType,
|
||||
} from '../src/full-state-op-types';
|
||||
import {
|
||||
partitionLwwResolutions,
|
||||
planLwwConflictResolutions,
|
||||
} from '../src/conflict-resolution';
|
||||
import { OpType } from '../src/operation.types';
|
||||
import type {
|
||||
ConflictResult,
|
||||
EntityConflict,
|
||||
MultiEntityPayload,
|
||||
Operation,
|
||||
OperationLogEntry,
|
||||
} from '../src/operation.types';
|
||||
|
||||
describe('createFullStateOpTypeHelpers', () => {
|
||||
it('classifies only the host-supplied op types as full-state operations', () => {
|
||||
const helpers = createFullStateOpTypeHelpers([
|
||||
'SYNC_IMPORT',
|
||||
'BACKUP_IMPORT',
|
||||
'REPAIR',
|
||||
] as const);
|
||||
|
||||
expect(helpers.FULL_STATE_OP_TYPES.has('SYNC_IMPORT')).toBe(true);
|
||||
expect(helpers.FULL_STATE_OP_TYPES.has('BACKUP_IMPORT')).toBe(true);
|
||||
expect(helpers.FULL_STATE_OP_TYPES.has('REPAIR')).toBe(true);
|
||||
expect(helpers.isFullStateOpType('SYNC_IMPORT')).toBe(true);
|
||||
expect(helpers.isFullStateOpType('CRT')).toBe(false);
|
||||
});
|
||||
|
||||
it('does not hard-code Super Productivity full-state op names', () => {
|
||||
const helpers = createFullStateOpTypeHelpers(['HOST_FULL_REPLACE'] as const);
|
||||
|
||||
expect(helpers.FULL_STATE_OP_TYPES.has('HOST_FULL_REPLACE')).toBe(true);
|
||||
expect(helpers.isFullStateOpType('HOST_FULL_REPLACE')).toBe(true);
|
||||
expect(helpers.isFullStateOpType('SYNC_IMPORT')).toBe(false);
|
||||
});
|
||||
|
||||
it('keeps deprecated Super Productivity compatibility exports', () => {
|
||||
expect(FULL_STATE_OP_TYPES.has(OpType.SyncImport)).toBe(true);
|
||||
expect(FULL_STATE_OP_TYPES.has(OpType.BackupImport)).toBe(true);
|
||||
expect(FULL_STATE_OP_TYPES.has(OpType.Repair)).toBe(true);
|
||||
expect(isFullStateOpType(OpType.SyncImport)).toBe(true);
|
||||
expect(isFullStateOpType(OpType.Create)).toBe(false);
|
||||
});
|
||||
|
||||
it('allows host-specific operation opType strings across exported wrappers', () => {
|
||||
type HostOpType = OpType | 'HOST_FULL_REPLACE';
|
||||
type HostOperation = Operation<HostOpType>;
|
||||
|
||||
const hostOp: HostOperation = {
|
||||
id: 'op-1',
|
||||
actionType: '[Host] Replace State',
|
||||
opType: 'HOST_FULL_REPLACE',
|
||||
entityType: 'HOST_ENTITY',
|
||||
payload: {},
|
||||
clientId: 'client-1',
|
||||
vectorClock: { client1: 1 },
|
||||
timestamp: 1,
|
||||
schemaVersion: 1,
|
||||
};
|
||||
const entry: OperationLogEntry<HostOperation> = {
|
||||
seq: 1,
|
||||
op: hostOp,
|
||||
appliedAt: 1,
|
||||
source: 'local',
|
||||
};
|
||||
const conflict: EntityConflict<HostOperation> = {
|
||||
entityType: 'HOST_ENTITY',
|
||||
entityId: 'entity-1',
|
||||
localOps: [hostOp],
|
||||
remoteOps: [hostOp],
|
||||
suggestedResolution: 'manual',
|
||||
};
|
||||
const result: ConflictResult<HostOperation> = {
|
||||
nonConflicting: [hostOp],
|
||||
conflicts: [conflict],
|
||||
};
|
||||
const multiEntityPayload: MultiEntityPayload<HostOpType> = {
|
||||
actionPayload: {},
|
||||
entityChanges: [
|
||||
{
|
||||
entityType: 'HOST_ENTITY',
|
||||
entityId: 'entity-1',
|
||||
opType: 'HOST_FULL_REPLACE',
|
||||
changes: {},
|
||||
},
|
||||
],
|
||||
};
|
||||
const plans = planLwwConflictResolutions<HostOperation>([conflict], {
|
||||
isArchiveAction: (op: HostOperation): boolean => op.actionType === '[Host] Archive',
|
||||
});
|
||||
const partitions = partitionLwwResolutions<HostOperation>(plans);
|
||||
|
||||
expect(entry.op.opType).toBe('HOST_FULL_REPLACE');
|
||||
expect(result.conflicts[0].localOps[0].opType).toBe('HOST_FULL_REPLACE');
|
||||
expect(multiEntityPayload.entityChanges[0].opType).toBe('HOST_FULL_REPLACE');
|
||||
expect(partitions.remoteWinsOps[0].opType).toBe('HOST_FULL_REPLACE');
|
||||
});
|
||||
});
|
||||
|
|
@ -14,18 +14,26 @@ import type {
|
|||
} from '@sp/sync-core';
|
||||
import type { EntityType as SharedEntityType } from '@sp/shared-schema';
|
||||
import { ENTITY_TYPES } from '@sp/shared-schema';
|
||||
import {
|
||||
createFullStateOpTypeHelpers,
|
||||
isMultiEntityPayload as libIsMultiEntityPayload,
|
||||
OpType,
|
||||
} from '@sp/sync-core';
|
||||
import { ActionType } from './action-types.enum';
|
||||
|
||||
export {
|
||||
OpType,
|
||||
FULL_STATE_OP_TYPES,
|
||||
isFullStateOpType,
|
||||
extractActionPayload,
|
||||
} from '@sp/sync-core';
|
||||
import { isMultiEntityPayload as libIsMultiEntityPayload } from '@sp/sync-core';
|
||||
export { OpType, extractActionPayload } from '@sp/sync-core';
|
||||
export type { VectorClock };
|
||||
export { ENTITY_TYPES, ActionType };
|
||||
|
||||
const fullStateOpTypeHelpers = createFullStateOpTypeHelpers<OpType>([
|
||||
OpType.SyncImport,
|
||||
OpType.BackupImport,
|
||||
OpType.Repair,
|
||||
]);
|
||||
|
||||
export const FULL_STATE_OP_TYPES = fullStateOpTypeHelpers.FULL_STATE_OP_TYPES;
|
||||
export const isFullStateOpType = fullStateOpTypeHelpers.isFullStateOpType;
|
||||
|
||||
/**
|
||||
* Entity type — Super Productivity's domain set, sourced from `@sp/shared-schema`
|
||||
* so client and server agree on the union.
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import { NOOP_SYNC_LOGGER } from '@sp/sync-core';
|
||||
import { OpLog } from '../../core/log';
|
||||
import { DecompressError } from '../core/errors/sync-errors';
|
||||
import {
|
||||
|
|
@ -9,6 +10,7 @@ import {
|
|||
describe('compression-handler', () => {
|
||||
beforeEach(() => {
|
||||
spyOn(OpLog, 'err').and.stub();
|
||||
spyOn(OpLog, 'log').and.stub();
|
||||
spyOn(console, 'log').and.stub();
|
||||
});
|
||||
|
||||
|
|
@ -46,6 +48,45 @@ describe('compression-handler', () => {
|
|||
);
|
||||
});
|
||||
|
||||
it('logs only sanitized error metadata on decompression failure', async () => {
|
||||
(OpLog.err as jasmine.Spy).calls.reset();
|
||||
const invalidGzip = btoa('not gzip data');
|
||||
|
||||
await expectAsync(decompressGzipFromString(invalidGzip)).toBeRejectedWithError(
|
||||
DecompressError,
|
||||
);
|
||||
|
||||
expect(OpLog.err).toHaveBeenCalledOnceWith(
|
||||
'[compression-handler] gzip decompression failed',
|
||||
jasmine.objectContaining({ name: jasmine.any(String) }),
|
||||
{
|
||||
inputLength: invalidGzip.length,
|
||||
sanitizedInputLength: invalidGzip.length,
|
||||
},
|
||||
);
|
||||
expect((OpLog.err as jasmine.Spy).calls.mostRecent().args[1] instanceof Error)
|
||||
.withContext('logged error identity must be sanitized')
|
||||
.toBeFalse();
|
||||
const additionalLogText = (OpLog.log as jasmine.Spy).calls
|
||||
.allArgs()
|
||||
.flat()
|
||||
.join('\n');
|
||||
expect(additionalLogText).not.toContain('not gzip data');
|
||||
});
|
||||
|
||||
it('preserves DecompressError when the injected logger throws', async () => {
|
||||
const throwingLogger = {
|
||||
...NOOP_SYNC_LOGGER,
|
||||
err: () => {
|
||||
throw new Error('logger failed');
|
||||
},
|
||||
};
|
||||
|
||||
await expectAsync(
|
||||
decompressGzipFromString(btoa('not gzip data'), throwingLogger),
|
||||
).toBeRejectedWithError(DecompressError);
|
||||
});
|
||||
|
||||
// Issue #6581: decompressGzipFromString should sanitize base64 input
|
||||
// to handle data corruption from WebDAV servers, partial uploads, etc.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,11 @@
|
|||
import { CompressError, DecompressError } from '../core/errors/sync-errors';
|
||||
import { OpLog } from '../../core/log';
|
||||
import type { SyncLogMeta, SyncLogger } from '@sp/sync-core';
|
||||
import { toSyncLogError } from '@sp/sync-core';
|
||||
import {
|
||||
CompressError,
|
||||
DecompressError,
|
||||
extractErrorMessage,
|
||||
} from '../core/errors/sync-errors';
|
||||
import { OP_LOG_SYNC_LOGGER } from '../core/sync-logger.adapter';
|
||||
|
||||
/**
|
||||
* Reads all bytes from a ReadableStream without using the Response constructor,
|
||||
|
|
@ -29,12 +35,43 @@ const readAllBytes = async (
|
|||
return result;
|
||||
};
|
||||
|
||||
const logCompressionError = (
|
||||
logger: SyncLogger,
|
||||
message: string,
|
||||
error: unknown,
|
||||
meta: SyncLogMeta,
|
||||
): void => {
|
||||
try {
|
||||
logger.err(message, toSyncLogError(error), meta);
|
||||
} catch {
|
||||
// Logging is best-effort and must not mask the original compression error.
|
||||
}
|
||||
};
|
||||
|
||||
const toSafeErrorForWrapper = (error: unknown): Error => {
|
||||
const syncLogError = toSyncLogError(error);
|
||||
const safeError = new Error(extractErrorMessage(error) ?? syncLogError.name);
|
||||
safeError.name = syncLogError.name;
|
||||
|
||||
if (syncLogError.code !== undefined) {
|
||||
Object.defineProperty(safeError, 'code', {
|
||||
value: syncLogError.code,
|
||||
enumerable: false,
|
||||
});
|
||||
}
|
||||
|
||||
return safeError;
|
||||
};
|
||||
|
||||
/**
|
||||
* Compresses a string using gzip and returns the raw bytes.
|
||||
* Use this for binary transmission (e.g., HTTP with Content-Encoding: gzip).
|
||||
*/
|
||||
// eslint-disable-next-line prefer-arrow/prefer-arrow-functions
|
||||
export async function compressWithGzip(input: string): Promise<Uint8Array> {
|
||||
export async function compressWithGzip(
|
||||
input: string,
|
||||
logger: SyncLogger = OP_LOG_SYNC_LOGGER,
|
||||
): Promise<Uint8Array> {
|
||||
try {
|
||||
const stream = new CompressionStream('gzip');
|
||||
const writer = stream.writable.getWriter();
|
||||
|
|
@ -47,8 +84,10 @@ export async function compressWithGzip(input: string): Promise<Uint8Array> {
|
|||
const [, compressed] = await Promise.all([writePromise, readPromise]);
|
||||
return compressed;
|
||||
} catch (error) {
|
||||
OpLog.err(error);
|
||||
throw new CompressError(error);
|
||||
logCompressionError(logger, '[compression-handler] gzip compression failed', error, {
|
||||
inputLength: input.length,
|
||||
});
|
||||
throw new CompressError(toSafeErrorForWrapper(error));
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -57,7 +96,10 @@ export async function compressWithGzip(input: string): Promise<Uint8Array> {
|
|||
* Use this for JSON payloads where binary data needs string encoding.
|
||||
*/
|
||||
// eslint-disable-next-line prefer-arrow/prefer-arrow-functions
|
||||
export async function compressWithGzipToString(input: string): Promise<string> {
|
||||
export async function compressWithGzipToString(
|
||||
input: string,
|
||||
logger: SyncLogger = OP_LOG_SYNC_LOGGER,
|
||||
): Promise<string> {
|
||||
try {
|
||||
const stream = new CompressionStream('gzip');
|
||||
const writer = stream.writable.getWriter();
|
||||
|
|
@ -79,8 +121,15 @@ export async function compressWithGzipToString(input: string): Promise<string> {
|
|||
reader.readAsDataURL(new Blob([compressed as unknown as BlobPart]));
|
||||
});
|
||||
} catch (error) {
|
||||
OpLog.err(error);
|
||||
throw new CompressError(error);
|
||||
logCompressionError(
|
||||
logger,
|
||||
'[compression-handler] gzip string compression failed',
|
||||
error,
|
||||
{
|
||||
inputLength: input.length,
|
||||
},
|
||||
);
|
||||
throw new CompressError(toSafeErrorForWrapper(error));
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -108,6 +157,7 @@ const sanitizeBase64 = (input: string): string => {
|
|||
// eslint-disable-next-line prefer-arrow/prefer-arrow-functions
|
||||
export async function decompressGzipFromString(
|
||||
compressedBase64: string,
|
||||
logger: SyncLogger = OP_LOG_SYNC_LOGGER,
|
||||
): Promise<string> {
|
||||
try {
|
||||
const sanitized = sanitizeBase64(compressedBase64);
|
||||
|
|
@ -128,10 +178,15 @@ export async function decompressGzipFromString(
|
|||
const decoded = new TextDecoder().decode(decompressed);
|
||||
return decoded;
|
||||
} catch (error) {
|
||||
OpLog.err(error);
|
||||
if (compressedBase64) {
|
||||
OpLog.err('base64 input length:', compressedBase64.length);
|
||||
}
|
||||
throw new DecompressError(error);
|
||||
logCompressionError(
|
||||
logger,
|
||||
'[compression-handler] gzip decompression failed',
|
||||
error,
|
||||
{
|
||||
inputLength: compressedBase64.length,
|
||||
sanitizedInputLength: sanitizeBase64(compressedBase64).length,
|
||||
},
|
||||
);
|
||||
throw new DecompressError(toSafeErrorForWrapper(error));
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -86,7 +86,7 @@ export class EncryptAndCompressHandlerService {
|
|||
);
|
||||
let dataStr = JSON.stringify(data);
|
||||
if (isCompress) {
|
||||
dataStr = await compressWithGzipToString(dataStr);
|
||||
dataStr = await compressWithGzipToString(dataStr, this._logger);
|
||||
}
|
||||
if (isEncrypt) {
|
||||
if (!encryptKey || encryptKey.length === 0) {
|
||||
|
|
@ -152,7 +152,7 @@ export class EncryptAndCompressHandlerService {
|
|||
}
|
||||
|
||||
if (isCompressed) {
|
||||
outStr = await decompressGzipFromString(outStr);
|
||||
outStr = await decompressGzipFromString(outStr, this._logger);
|
||||
}
|
||||
|
||||
let parsedData: T;
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue