Commit graph

183 commits

Author SHA1 Message Date
Johannes Millan
1c85979792 docs: add plan 2025-12-28 13:17:16 +01:00
Johannes Millan
c29913aeac test(time-tracking): add comprehensive tests and documentation
Add extensive tests and documentation for TimeTrackingState sync:

Unit tests (~22 new):
- merge-time-tracking-states.spec.ts: partial data merging, null
  handling, merge priority, scale testing
- sort-data-to-flush.spec.ts: flush completeness, LWW behavior,
  date boundaries
- time-tracking.reducer.spec.ts: context preservation, immutability

Integration tests (9 new):
- time-tracking-sync.integration.spec.ts: operation creation,
  SYNC_IMPORT with timeTracking, multi-client sync, archive flush

Server tests (18 new):
- time-tracking-operations.spec.ts: operation upload, conflict
  detection, snapshot generation, LWW semantics

Documentation:
- JSDoc for merge-time-tracking-states.ts and sort-data-to-flush.ts
- Section E.6 "Time Tracking Sync Semantics" in architecture docs
2025-12-28 13:16:07 +01:00
Johannes Millan
402c67a5b1 docs: add TODAY_TAG architecture and improve timing constants docs
- Create docs/ai/today-tag-architecture.md documenting the virtual tag
  pattern, explaining why TODAY_TAG must never be in task.tagIds and
  how membership is determined by task.dueDay instead
- Add reference section to operation-log.const.ts pointing to related
  timing constants in other domain files (sync.const.ts, meta-sync)

This addresses documentation gaps identified in the maintainability review.
2025-12-27 19:13:56 +01:00
Johannes Millan
d6dcc86ca0 refactor: reorganize operation-log files into src/app/op-log/
Move operation-log from src/app/core/persistence/operation-log/ to
src/app/op-log/ with improved subdirectory organization:

- core/: Types, constants, errors, entity registry
- capture/: Write path (Actions → Operations)
  - operation-capture.meta-reducer.ts
  - operation-capture.service.ts
  - operation-log.effects.ts
- apply/: Read path (Operations → State)
  - bulk-hydration.action.ts/.meta-reducer.ts
  - operation-applier.service.ts
  - operation-converter.util.ts
  - hydration-state.service.ts
  - archive-operation-handler.service.ts/.effects.ts
- store/: IndexedDB persistence
  - operation-log-store.service.ts
  - operation-log-hydrator.service.ts
  - operation-log-compaction.service.ts
  - schema-migration.service.ts
- sync/: Server sync (SuperSync)
  - operation-log-sync.service.ts
  - operation-log-upload.service.ts
  - operation-log-download.service.ts
  - conflict-resolution.service.ts
  - sync-import-filter.service.ts
  - vector-clock.service.ts
  - operation-encryption.service.ts
- validation/: State validation
  - validate-state.service.ts
  - validate-operation-payload.ts
- util/: Shared utilities
  - entity-key.util.ts
  - client-id.provider.ts
- testing/: Integration tests and benchmarks

This reorganization:
- Places op-log at the same level as pfapi for better visibility
- Groups files by responsibility (write path vs read path)
- Makes the sync architecture more discoverable
- Improves navigation for developers new to the codebase
2025-12-27 17:52:11 +01:00
Johannes Millan
693e71deeb docs(sync): complete quick-reference with Areas 8-12
Add visual summaries for remaining areas:
- Area 8: Meta-Reducers (atomic multi-entity changes)
- Area 9: Compaction (snapshot + garbage collection)
- Area 10: Bulk Application (single dispatch optimization)
- Area 11: Encryption (AES-256-GCM + Argon2id)
- Area 12: Legacy PFAPI Bridge (WebDAV/Dropbox)

Removes "Coming Soon" placeholder. All 12 areas now documented.
2025-12-27 17:10:18 +01:00
Johannes Millan
30fa62b9a4 test(sync): add canary test for entity type coverage
Add safeguards to prevent silent failures when new entity types are
added without proper registration:

- Add canary test that fails if EntityType union grows but test arrays
  aren't updated (verifies count matches expected 20 types)
- Create developer checklist at docs/ai/adding-new-entity-type-checklist.md
  documenting all required changes when adding new entity types
2025-12-27 17:08:53 +01:00
Johannes Millan
e221afe81c docs(sync): add quick-reference.md with visual summaries
Create a quick reference document with ASCII diagrams summarizing:
- Area 1: Write Path
- Area 2: Read Path (Hydration)
- Area 3: Server Sync (SuperSync)
- Area 4: Conflict Detection
- Area 5: Conflict Resolution (LWW)
- Area 6: SYNC_IMPORT Filtering
- Area 7: Archive Handling

Includes decision tables, key invariants, and file references.
Areas 8-12 placeholders for future additions.
2025-12-27 16:42:38 +01:00
Johannes Millan
330ecce2c9 docs(sync): fix LWW timestamp and SYNC_IMPORT filtering sections
- C.5: Correct "Current timestamp" to "Preserved maximum timestamp
  from local ops" - critical for correct LWW semantics
- C.7: Rewrite "Late-Joiner Replay" as "SYNC_IMPORT Filtering" to
  reflect actual "Clean Slate Semantics" implementation:
  - CONCURRENT ops are now DROPPED, not replayed
  - Uses SyncImportFilterService, not removed _replayLocalSyncedOpsAfterImport()
  - Vector clock comparison determines causality, not UUIDv7 timestamps
2025-12-27 15:59:00 +01:00
Johannes Millan
8fff1325b4 docs: fix SYNC_IMPORT filtering section to match implementation
- Fix CLAUDE.md path reference from docs/op-log/ to docs/sync-and-op-log/
- Rewrite section 2c in architecture diagrams to reflect actual implementation:
  - Was: UUIDv7 timestamp replay (removed feature)
  - Now: Vector clock filtering with clean slate semantics
- Update service reference from removed _replayLocalSyncedOpsAfterImport()
  to SyncImportFilterService.filterOpsInvalidatedBySyncImport()
- Clarify that CONCURRENT ops are dropped, not replayed
2025-12-27 13:15:47 +01:00
Johannes Millan
c1575d7669 docs(sync): update operation-log docs to reflect current implementation
- Update README.md with correct links and implementation status
- Add E2EE to implementation status (now complete)
- Update e2e-encryption-plan.md to mark as implemented
- Update operation-log-architecture.md:
  - Add missing service files to file reference section
  - Add E2EE to Part C and "Recently Completed" sections
  - Fix broken references, update last modified date
- Remove reference to non-existent tiered-archive-proposal.md
2025-12-27 11:44:30 +01:00
Johannes Millan
39d6e15446 docs: improve structure 2025-12-27 11:32:54 +01:00
Johannes Millan
7227749ec4 test(sync): add tests for bulk dispatch edge cases
Add comprehensive tests for operation applier testing gaps:

- Partial archive failure: verifies that when archive handling fails
  midway through a batch, previously processed ops are reported
  as successful while remaining ops are not applied
- Effects isolation: confirms that only bulkApplyOperations action
  is dispatched (not individual action types), which is the key
  architectural benefit preventing effects from firing on remote ops
- Multiple archive-affecting ops: ensures remoteArchiveDataApplied
  is dispatched exactly once when batch contains multiple
  archive-affecting operations

Total: 4 new test cases, 24 tests now passing
2025-12-27 11:31:51 +01:00
Johannes Millan
b4ce9d5da6 docs: reorganize sync and operation-log documentation
Move scattered architecture docs into centralized locations:

- Move operation-log docs from src/app/core/persistence/operation-log/docs/
  to docs/op-log/
- Flatten docs/sync/sync/ nested structure to docs/sync/
- Move supersync-encryption-architecture.md from docs/ai/ to docs/sync/
- Copy pfapi sync README to docs/sync/pfapi-sync-overview.md
- Update all cross-references to use new paths

This improves discoverability and keeps architecture documentation
separate from source code.
2025-12-27 10:54:13 +01:00
Johannes Millan
5138b46546 fix(sync): temporarily disable data repair to debug archive subtask loss
Disable automatic data repair during sync/hydration to:
- Expose validation failures instead of silently "fixing" them
- Prevent REPAIR operations from propagating potentially corrupted data
- Help identify the root cause of archived tasks with subtasks being lost

When validation fails, an alert is now shown with error details.
The original repair logic is preserved in comments for easy re-enabling.
2025-12-26 10:54:54 +01:00
Johannes Millan
064c2452ca fix(sync): handle piggyback limit in high-volume sync
When syncing 100+ operations, the server's piggyback limit (100 ops)
was causing Client B to miss operations. The server returned 100 piggybacked
ops but latestSeq was set to the actual server sequence (e.g., 199).
Client B then updated lastServerSeq to 199, so subsequent download got 0 ops.

Changes:
- Server: Add hasMorePiggyback flag to UploadOpsResponse when piggyback limit
  is reached and more ops exist
- Client: When hasMorePiggyback is true, store lastServerSeq as the max
  piggybacked op's serverSeq instead of latestSeq, ensuring subsequent
  download fetches remaining ops
- Effects: Change switchMap to mergeMap in autoAddTodayTagOnMarkAsDone to
  ensure ALL mark-as-done actions trigger planTasksForToday
- Flush service: Implement two-phase wait strategy (poll queue + acquire lock)
  to ensure all pending writes complete before upload
- Add diagnostic logging for operation counts at key stages

Test: High volume sync with 50 tasks + 49 mark-as-done (197 ops total)
now correctly syncs all done states via piggyback (100) + download (97).
2025-12-22 20:31:35 +01:00
Johannes Millan
c4cc32da29 feat(sync): add encryption password change feature for SuperSync
Implements the ability to change the encryption password by deleting
all server data and uploading a fresh snapshot with the new password.

Server changes:
- Add DELETE /api/sync/data endpoint to delete all user sync data
- Add deleteAllUserData() method to SyncService

Client changes:
- Add deleteAllData() to OperationSyncCapable interface
- Implement deleteAllData() in SuperSync provider
- Add EncryptionPasswordChangeService to orchestrate password change
- Add DialogChangeEncryptionPasswordComponent with validation
- Add "Change Encryption Password" button to sync settings (visible
  when encryption is enabled)
- Add translations for all new UI strings

Testing:
- Add 10 unit tests for EncryptionPasswordChangeService
- Add 14 unit tests for DialogChangeEncryptionPasswordComponent
- Add 5 E2E tests for complete password change flow
- Add changeEncryptionPassword() helper to SuperSyncPage

Also fixes:
- Add missing deleteAllData() to MockOperationSyncProvider
- Fix typo S_FINISH_DAY_SYNC_ERROR -> FINISH_DAY_SYNC_ERROR
2025-12-22 13:31:19 +01:00
Johannes Millan
155e03b84c docs: remove outdated sync documentation and deprecated file
- Delete docs/ai/sync/server-sync-architecture.md which incorrectly
  stated "Status: Not Started" when server sync is fully implemented
- Delete deprecated src/app/features/time-tracking/store/archive.effects.ts
  which was empty and marked for removal
- Update local-actions.token.ts comment to reference the correct
  archive-operation-handler.effects.ts file
2025-12-20 11:09:49 +01:00
Johannes Millan
f928e6b18e docs: cleanup 2025-12-16 12:28:04 +01:00
Johannes Millan
642f8d3984 docs(sync): improve and consolidate operation log documentation
- Update hybrid-manifest-architecture.md status to Implemented
- Add documentation index (README.md) for operation-log docs
- Add index for docs/ai/sync/ with document categorization
- Replace duplicate docs with redirect stubs to canonical locations
- Add effect rules, multi-entity rules, and config constants to operation-rules.md
- Update vector-clocks.md with operation log integration section
- Expand super-sync-server README with API details and security features
- Update pfapi sync README with operation log integration info
2025-12-16 10:45:20 +01:00
Johannes Millan
04043e3964 docs(sync): document late-joiner replay with vector clock dominance
- Add Section 2c to architecture diagrams with mermaid diagrams
  explaining the late-joiner problem and solution
- Add Section C.7 to main architecture doc with code examples
- Add Example 4 to vector-clocks.md explaining dominance filtering
- Update Last Updated dates to December 12, 2025
2025-12-12 20:48:40 +01:00
Johannes Millan
37a5101da3 docs(sync): remove separate review doc, architecture documented in code 2025-12-12 20:48:40 +01:00
Johannes Millan
e647455114 docs(sync): restore E2E encryption review document 2025-12-12 20:48:40 +01:00
Johannes Millan
f47e9496a9 docs(sync): add architecture context to operation log services
Add class-level documentation explaining:
- Only SuperSync uses operation log sync (API-based)
- Legacy providers skip operation log sync entirely
- File-based methods exist for future extensibility but are never called
- If file-based sync is ever enabled, encryption/decryption must be added
2025-12-12 20:48:40 +01:00
Johannes Millan
c79144c4b8 docs(sync): update E2E encryption review with correct architecture
The original review incorrectly identified file-based operation log sync
as having "critical gaps". This update corrects the analysis:

- File-based operation log sync is dead code, never called
- Only SuperSync uses operation log sync (API-based)
- Legacy providers skip operation log sync entirely and use pfapi LWW

Status changed from "NEEDS FIXES" to "PRODUCTION READY"
2025-12-12 20:48:40 +01:00
Johannes Millan
37b5e94525 docs(sync): update implementation review - all test gaps now covered
Mark all previously identified test gaps as resolved:
- Race conditions: 6 new tests in compaction service
- Schema migration: 6 version mismatch tests in hydrator service
- Download retry: 4 tests now passing (was pending())
- All critical and missing scenarios now have test coverage
2025-12-12 20:48:40 +01:00
Johannes Millan
d6e8fe9ced docs(sync): update implementation review and E2E test plan
- Add December 11 security review results (all issues verified as addressed)
- Document newly added tests (3-way conflict, delete vs update, large conflict sets)
- Update test coverage status table with completed items
- Mark supersync E2E test plan as IMPLEMENTED
- List all test files and their coverage areas
2025-12-12 20:48:40 +01:00
Johannes Millan
7ad78333be docs(sync): update architecture docs for recent refactors
Update documentation to reflect recent simplifications:
- OperationCaptureService now uses simple FIFO queue
- OperationApplierService uses fail-fast approach
- ArchiveOperationHandler is unified for local and remote operations
- Update diagrams and file references
- Add "Recently Completed" items for December 2025
2025-12-12 20:48:13 +01:00
Johannes Millan
5ffc109943 refactor(sync): unify archive operation handling
Consolidate all archive-affecting operations into ArchiveOperationHandler
to eliminate duplicate code and create a single source of truth.

Changes:
- Add ArchiveOperationHandlerEffects for local operations
- Rename handleRemoteOperation to handleOperation
- Add isArchiveAffectingAction helper for action filtering
- Make isIgnoreDBLock conditional on action.meta.isRemote
- Remove archive logic from individual effect files
- Update feature-stores.module.ts to use new effect
- Add archive write points documentation

This ensures archive operations (moveToArchive, restoreTask,
flushYoungToOld, deleteProject, deleteTag, deleteTaskRepeatCfg,
deleteIssueProvider) all go through one handler for both local
and remote operations.
2025-12-12 20:48:12 +01:00
Johannes Millan
320e57e622 refactor(tasks): use satisfies instead of as for PersistentActionMeta
Improves type safety by using satisfies which validates the object
matches the type at compile time, rather than as which is just an
assertion.
2025-12-12 20:48:12 +01:00
Johannes Millan
69f3602927 docs(sync): update review with completed fixes
Mark as fixed:
- Bounds checking in planner-shared.reducer.ts
- Error handling in tag.effects.ts (async IndexedDB operations)
2025-12-12 20:47:48 +01:00
Johannes Millan
8187313e0c docs(sync): mark TaskDueEffects race conditions as false positive
Analysis shows the alleged race conditions are intentional defensive patterns:
- Double sync wait ensures stability during date changes
- Effects are complementary with staggered timing, not competing
- Infinite loop scenario explicitly handled in code

Note: Tests exist but are disabled due to Dropbox SDK mocking issues.
2025-12-12 20:47:48 +01:00
Johannes Millan
89066aca51 docs(sync): mark subtask orphan detection as false positive
Re-evaluated the "missing subtask cascade in tag deletion" issue.
The current behavior is correct:
- A subtask with a surviving parent is NOT orphaned - it's accessible
  through the parent in the UI
- If a parent is orphaned, its subtasks ARE correctly deleted
- Subtasks always have parentId, so they're never directly orphaned

This is intentional behavior, not a bug.
2025-12-12 20:47:48 +01:00
Johannes Millan
e6025b4178 fix(sync): validate state before saving snapshot in hydrator
The snapshot was being saved before validation ran. If validation found
corruption and repaired it, the snapshot was stale/wrong. This fix
moves validation (CHECKPOINT C) BEFORE saving snapshot in both:
- Tail replay path (after replaying ops following a snapshot)
- Full replay path (when no snapshot exists)

Now the snapshot contains validated/repaired state.

Includes regression tests that verify validation happens before snapshot
save in both code paths.
2025-12-12 20:47:48 +01:00
Johannes Millan
bbd50c223a fix(sync): persist compaction counter when no cache exists
The incrementCompactionCounter() function returned 1 without persisting
when no state cache existed. This caused the compaction threshold to
never be reached, leading to unbounded operation log growth.

Also updated review docs:
- Marked compaction counter bug as fixed
- Identified memory leak in TaskElectronEffects as false positive
  (take(1) ensures automatic cleanup)
- Identified TODAY_TAG "board-style" claim as false positive
  (code correctly REMOVES TODAY_TAG from task.tagIds)
- Identified cache invalidation claim as false positive
  (sequence-based invalidation works correctly)
- Identified repair mutex race condition as false positive
  (JavaScript single-threaded, no await between assignment)
2025-12-12 20:47:48 +01:00
Johannes Millan
78c65acf4d test(e2e): add sd:today to tasks for TODAY view visibility
Tasks created without explicit dueDay don't appear in TODAY view
due to recent selector changes. Add sd:today short syntax to ensure
tasks have dueDay set and appear correctly in tests.
2025-12-12 20:47:48 +01:00
Johannes Millan
60bc0e78d7 docs(sync): replace PFAPI with operation log sync for all providers 2025-12-12 20:47:47 +01:00
Johannes Millan
e62f2f86fa feat(e2e): add SuperSync E2E test infrastructure for multi-client sync
- Add test mode to super-sync-server (TEST_MODE=true env var)
- Add /api/test/create-user endpoint for auto-verified test users
- Add /api/test/cleanup endpoint for wiping test data
- Create SuperSyncPage page object for sync configuration
- Create supersync-helpers.ts with test utilities
- Add 6 E2E sync scenarios:
  - 2.1: Client A creates task, Client B downloads
  - 2.2: Both clients create different tasks
  - 1.3: Update propagates between clients
  - 1.4: Delete propagates between clients
  - 3.1: Concurrent edits handled gracefully
  - 2.3: Parent/subtask relationships sync correctly
2025-12-12 20:47:41 +01:00
Johannes Millan
c977ad3edd test(oplog): add integration tests for multi-client sync and state consistency
Add Karma-based integration tests for the operation log system using real
IndexedDB. Tests cover multi-client synchronization scenarios and state
consistency verification.

New test files:
- multi-client-sync.integration.spec.ts (16 tests)
- state-consistency.integration.spec.ts (17 tests)

Test utilities:
- TestClient helper for simulating multiple clients with independent vector clocks
- Operation factory helpers for creating test operations
2025-12-12 20:46:27 +01:00
Johannes Millan
3129c1dbca test(oplog): add persistent-action tests and fix compilation/lint errors
- Add unit tests for isPersistentAction type guard.

- Fix compilation errors in task scheduling components caused by removed reminderId/removeReminderFromTask.

- Fix type error in create-sorted-blocker-blocks.spec.ts.

- Fix lint errors in various files.
2025-12-12 20:46:27 +01:00
Johannes Millan
f661de10de feat(sync): implement time tracking sync and deterministic archive flush
- Add syncTimeSpent persistent action for batched time tracking sync
  - Accumulates duration per task, syncs every 5 minutes or on task stop
  - Local dispatch: no-op (state already updated by second-by-second ticks)
  - Remote dispatch: applies batched duration to task.timeSpentOnDay

- Fix remote archive handling to include time tracking data
  - writeTasksToArchiveForRemoteSync now moves historical time tracking
    data to archiveYoung alongside tasks

- Add flushYoungToOld persistent action for deterministic archive state
  - Replaces inline flush logic with action-based approach
  - Remote clients replay the flush operation to maintain consistent
    archiveOld state without syncing large files
  - Vector clock ordering ensures flush happens after preceding operations
2025-12-12 20:46:26 +01:00
Johannes Millan
fd44db3852 docs(sync): update planning doc 2025-12-12 20:46:26 +01:00
Johannes Millan
410bcde114 feat(sync): implement receiver-side migration shield and fix tests
- Implement processRemoteOps migration logic in OperationLogSyncService using SchemaMigrationService.
- Add MAX_VERSION_SKIP check and user warning via SnackService.
- Fix Log class to support dynamic level switching (fixes PFAPI tests).
- Fix operation-log-stress imports.
- Skip disabled MultiTabCoordinatorService tests.
- Add unit tests for migration logic.
2025-12-12 20:46:26 +01:00
Johannes Millan
4e018a8064 docs: update operation log architecture paths 2025-12-12 20:46:26 +01:00
Johannes Millan
d79dc153fe docs: add architectural introduction to operation log documentation 2025-12-12 20:46:26 +01:00
Johannes Millan
04728267e2 docs: update operation log architecture docs to reflect recent changes
- Update ValidateStateService documentation to reflect better error handling for isRelatedModelDataValid.
- Update Part C status to Complete as Server Sync is implemented.
- List implemented components for Part C.
- Fix lint error in startup.service.ts
2025-12-12 20:46:26 +01:00
Johannes Millan
a70273597f refactor(oplog): split sync service and extract vector clock utilities
- Split OperationLogSyncService (678→289 lines) into focused services:
  - OperationLogManifestService: manifest file operations
  - OperationLogUploadService: upload pending operations
  - OperationLogDownloadService: download remote operations
  - operation-sync.util.ts: shared type guards and converters

- Extract VectorClockService with clear documentation:
  - getCurrentVectorClock(): merged clock from snapshot + ops
  - getEntityFrontier(): per-entity vector clocks
  - getSnapshotVectorClock(): baseline for conflict detection

- Remove dead code:
  - Remove markApplied() no-op from store service
  - Move stress tests to benchmarks/ folder

- Fix type safety:
  - Replace 'as any' casts with proper types
  - Add TaskOperationPayload interface
2025-12-12 20:46:26 +01:00
Johannes Millan
377800caf5 docs(oplog): add refactoring plan for operation log system 2025-12-12 20:46:26 +01:00
Johannes Millan
6818ae8d9f feat(oplog): add persistent compaction counter and syncedAt index
- Store compaction counter in state_cache to share across tabs/restarts
- Add bySyncedAt index for faster getUnsynced() queries
- Remove in-memory opsSinceCompaction counter from effects
- Reset counter in compaction service after successful compaction
2025-12-12 20:46:26 +01:00
Johannes Millan
b73769aa15 feat(oplog): implement migration safety and tail ops migration
Implements A.7.12, A.7.13, and A.7.15 from the operation log architecture:

- A.7.12 Migration Safety: Backup state_cache before migration, restore
  on failure, detect interrupted migrations on startup
- A.7.13 Tail Ops Migration: Migrate operations to current schema before
  replay during hydration
- A.7.15 Unified Interface: Extend SchemaMigration with migrateOperation
  and requiresOperationMigration for explicit migration decisions

Changes:
- operation-log-store: Add backup/restore methods for state_cache
- schema-migration: Add operation migration support and validation
- operation-log-hydrator: Integrate backup safety and tail ops migration
- Update architecture docs to reflect implementation status
2025-12-12 20:46:26 +01:00
Johannes Millan
36dd2ee38b docs(sync): replace hybrid manifest mindmap with detailed flow diagrams
Replace the single mindmap with comprehensive diagrams:
- 6.1 Data Lifecycle: Hot → Cold → Frozen
- 6.2 Manifest File Structure
- 6.3 Write Path: Buffer vs Overflow Decision
- 6.4 Read Path: Reconstructing State
- 6.5 Compaction: Freezing State
- 6.6 Request Count Comparison table
2025-12-12 20:46:26 +01:00