Commit graph

105 commits

Author SHA1 Message Date
Johannes Millan
9f0adbb95c docs: fix outdated file paths and types in diagrams
- Fix FileBasedSyncData type: remove non-existent lastSeq, add clientId
- Fix file paths: op-log/processing → op-log/apply
- Fix file paths: features/time-tracking → features/archive
- Fix file path: super-sync not supersync
- Fix vector-clock path: now in op-log/sync/
- Remove non-existent state-capture.meta-reducer.ts reference
- Remove pfapi-migration.service.ts (no longer exists)

docs: remove outdated .bak file references from diagrams

The backup file (sync-data.json.bak) is no longer created during
upload. It's only deleted as cleanup from legacy implementations.

docs: add sync comparison and simple sync flow diagrams

- Add 07-supersync-vs-file-based.md comparing the two sync approaches
- Add 08-sync-flow-explained.md with step-by-step sync explanation
- Remove consolidated unified-oplog-sync-diagrams.md
- Update diagrams README with new entries

docs(sync): reorganize diagrams into subfolder and update for unified architecture

- Create docs/sync-and-op-log/diagrams/ with topic-based diagram files
- Remove outdated PFAPI Legacy Bridge references from diagrams
- Update archive diagrams to use generic "Archive Database" naming
- Fix file paths from sync/providers/ to sync-providers/
- Update quick-reference Area 12 to show unified file-based sync
- Update README to reference new diagram locations

docs: update architecture docs to reflect PFAPI elimination

- Delete obsolete PFAPI documentation:
  - docs/sync-and-op-log/pfapi-sync-persistence-architecture.md
  - docs/sync-and-op-log/pfapi-sync-overview.md
  - docs/plans/pfapi-elimination-status.md

- Update sync-and-op-log/README.md:
  - Describe unified operation log architecture
  - Document file-based sync (Part B) and server sync (Part C)
  - Update file structure to reflect sync-providers location

- Update operation-log-architecture.md:
  - Rewrite Part B from "Legacy Sync Bridge" to "File-Based Sync"
  - Remove all PFAPI code examples and references
  - Update IndexedDB structure diagram (single SUP_OPS database)
  - Update architecture overview to show current provider structure
  - Add notes about PFAPI elimination (January 2026)

- Mark completed implementation plans:
  - replace-pfapi-with-oplog-plan.md - marked as COMPLETED
  - file-based-oplog-sync-implementation-plan.md - marked as COMPLETED

Also includes fix for file-based sync gap detection to handle
snapshot replacement (when "Use Local" is chosen in conflict resolution).
2026-01-08 11:10:29 +01:00
Johannes Millan
1f8fe61c84 refactor: integrate pfapi into oplog 1 2026-01-07 13:30:09 +01:00
Johannes Millan
402c67a5b1 docs: add TODAY_TAG architecture and improve timing constants docs
- Create docs/ai/today-tag-architecture.md documenting the virtual tag
  pattern, explaining why TODAY_TAG must never be in task.tagIds and
  how membership is determined by task.dueDay instead
- Add reference section to operation-log.const.ts pointing to related
  timing constants in other domain files (sync.const.ts, meta-sync)

This addresses documentation gaps identified in the maintainability review.
2025-12-27 19:13:56 +01:00
Johannes Millan
d6dcc86ca0 refactor: reorganize operation-log files into src/app/op-log/
Move operation-log from src/app/core/persistence/operation-log/ to
src/app/op-log/ with improved subdirectory organization:

- core/: Types, constants, errors, entity registry
- capture/: Write path (Actions → Operations)
  - operation-capture.meta-reducer.ts
  - operation-capture.service.ts
  - operation-log.effects.ts
- apply/: Read path (Operations → State)
  - bulk-hydration.action.ts/.meta-reducer.ts
  - operation-applier.service.ts
  - operation-converter.util.ts
  - hydration-state.service.ts
  - archive-operation-handler.service.ts/.effects.ts
- store/: IndexedDB persistence
  - operation-log-store.service.ts
  - operation-log-hydrator.service.ts
  - operation-log-compaction.service.ts
  - schema-migration.service.ts
- sync/: Server sync (SuperSync)
  - operation-log-sync.service.ts
  - operation-log-upload.service.ts
  - operation-log-download.service.ts
  - conflict-resolution.service.ts
  - sync-import-filter.service.ts
  - vector-clock.service.ts
  - operation-encryption.service.ts
- validation/: State validation
  - validate-state.service.ts
  - validate-operation-payload.ts
- util/: Shared utilities
  - entity-key.util.ts
  - client-id.provider.ts
- testing/: Integration tests and benchmarks

This reorganization:
- Places op-log at the same level as pfapi for better visibility
- Groups files by responsibility (write path vs read path)
- Makes the sync architecture more discoverable
- Improves navigation for developers new to the codebase
2025-12-27 17:52:11 +01:00
Johannes Millan
30fa62b9a4 test(sync): add canary test for entity type coverage
Add safeguards to prevent silent failures when new entity types are
added without proper registration:

- Add canary test that fails if EntityType union grows but test arrays
  aren't updated (verifies count matches expected 20 types)
- Create developer checklist at docs/ai/adding-new-entity-type-checklist.md
  documenting all required changes when adding new entity types
2025-12-27 17:08:53 +01:00
Johannes Millan
b4ce9d5da6 docs: reorganize sync and operation-log documentation
Move scattered architecture docs into centralized locations:

- Move operation-log docs from src/app/core/persistence/operation-log/docs/
  to docs/op-log/
- Flatten docs/sync/sync/ nested structure to docs/sync/
- Move supersync-encryption-architecture.md from docs/ai/ to docs/sync/
- Copy pfapi sync README to docs/sync/pfapi-sync-overview.md
- Update all cross-references to use new paths

This improves discoverability and keeps architecture documentation
separate from source code.
2025-12-27 10:54:13 +01:00
Johannes Millan
5138b46546 fix(sync): temporarily disable data repair to debug archive subtask loss
Disable automatic data repair during sync/hydration to:
- Expose validation failures instead of silently "fixing" them
- Prevent REPAIR operations from propagating potentially corrupted data
- Help identify the root cause of archived tasks with subtasks being lost

When validation fails, an alert is now shown with error details.
The original repair logic is preserved in comments for easy re-enabling.
2025-12-26 10:54:54 +01:00
Johannes Millan
064c2452ca fix(sync): handle piggyback limit in high-volume sync
When syncing 100+ operations, the server's piggyback limit (100 ops)
was causing Client B to miss operations. The server returned 100 piggybacked
ops but latestSeq was set to the actual server sequence (e.g., 199).
Client B then updated lastServerSeq to 199, so subsequent download got 0 ops.

Changes:
- Server: Add hasMorePiggyback flag to UploadOpsResponse when piggyback limit
  is reached and more ops exist
- Client: When hasMorePiggyback is true, store lastServerSeq as the max
  piggybacked op's serverSeq instead of latestSeq, ensuring subsequent
  download fetches remaining ops
- Effects: Change switchMap to mergeMap in autoAddTodayTagOnMarkAsDone to
  ensure ALL mark-as-done actions trigger planTasksForToday
- Flush service: Implement two-phase wait strategy (poll queue + acquire lock)
  to ensure all pending writes complete before upload
- Add diagnostic logging for operation counts at key stages

Test: High volume sync with 50 tasks + 49 mark-as-done (197 ops total)
now correctly syncs all done states via piggyback (100) + download (97).
2025-12-22 20:31:35 +01:00
Johannes Millan
c4cc32da29 feat(sync): add encryption password change feature for SuperSync
Implements the ability to change the encryption password by deleting
all server data and uploading a fresh snapshot with the new password.

Server changes:
- Add DELETE /api/sync/data endpoint to delete all user sync data
- Add deleteAllUserData() method to SyncService

Client changes:
- Add deleteAllData() to OperationSyncCapable interface
- Implement deleteAllData() in SuperSync provider
- Add EncryptionPasswordChangeService to orchestrate password change
- Add DialogChangeEncryptionPasswordComponent with validation
- Add "Change Encryption Password" button to sync settings (visible
  when encryption is enabled)
- Add translations for all new UI strings

Testing:
- Add 10 unit tests for EncryptionPasswordChangeService
- Add 14 unit tests for DialogChangeEncryptionPasswordComponent
- Add 5 E2E tests for complete password change flow
- Add changeEncryptionPassword() helper to SuperSyncPage

Also fixes:
- Add missing deleteAllData() to MockOperationSyncProvider
- Fix typo S_FINISH_DAY_SYNC_ERROR -> FINISH_DAY_SYNC_ERROR
2025-12-22 13:31:19 +01:00
Johannes Millan
155e03b84c docs: remove outdated sync documentation and deprecated file
- Delete docs/ai/sync/server-sync-architecture.md which incorrectly
  stated "Status: Not Started" when server sync is fully implemented
- Delete deprecated src/app/features/time-tracking/store/archive.effects.ts
  which was empty and marked for removal
- Update local-actions.token.ts comment to reference the correct
  archive-operation-handler.effects.ts file
2025-12-20 11:09:49 +01:00
Johannes Millan
f928e6b18e docs: cleanup 2025-12-16 12:28:04 +01:00
Johannes Millan
642f8d3984 docs(sync): improve and consolidate operation log documentation
- Update hybrid-manifest-architecture.md status to Implemented
- Add documentation index (README.md) for operation-log docs
- Add index for docs/ai/sync/ with document categorization
- Replace duplicate docs with redirect stubs to canonical locations
- Add effect rules, multi-entity rules, and config constants to operation-rules.md
- Update vector-clocks.md with operation log integration section
- Expand super-sync-server README with API details and security features
- Update pfapi sync README with operation log integration info
2025-12-16 10:45:20 +01:00
Johannes Millan
37a5101da3 docs(sync): remove separate review doc, architecture documented in code 2025-12-12 20:48:40 +01:00
Johannes Millan
e647455114 docs(sync): restore E2E encryption review document 2025-12-12 20:48:40 +01:00
Johannes Millan
f47e9496a9 docs(sync): add architecture context to operation log services
Add class-level documentation explaining:
- Only SuperSync uses operation log sync (API-based)
- Legacy providers skip operation log sync entirely
- File-based methods exist for future extensibility but are never called
- If file-based sync is ever enabled, encryption/decryption must be added
2025-12-12 20:48:40 +01:00
Johannes Millan
c79144c4b8 docs(sync): update E2E encryption review with correct architecture
The original review incorrectly identified file-based operation log sync
as having "critical gaps". This update corrects the analysis:

- File-based operation log sync is dead code, never called
- Only SuperSync uses operation log sync (API-based)
- Legacy providers skip operation log sync entirely and use pfapi LWW

Status changed from "NEEDS FIXES" to "PRODUCTION READY"
2025-12-12 20:48:40 +01:00
Johannes Millan
37b5e94525 docs(sync): update implementation review - all test gaps now covered
Mark all previously identified test gaps as resolved:
- Race conditions: 6 new tests in compaction service
- Schema migration: 6 version mismatch tests in hydrator service
- Download retry: 4 tests now passing (was pending())
- All critical and missing scenarios now have test coverage
2025-12-12 20:48:40 +01:00
Johannes Millan
d6e8fe9ced docs(sync): update implementation review and E2E test plan
- Add December 11 security review results (all issues verified as addressed)
- Document newly added tests (3-way conflict, delete vs update, large conflict sets)
- Update test coverage status table with completed items
- Mark supersync E2E test plan as IMPLEMENTED
- List all test files and their coverage areas
2025-12-12 20:48:40 +01:00
Johannes Millan
7ad78333be docs(sync): update architecture docs for recent refactors
Update documentation to reflect recent simplifications:
- OperationCaptureService now uses simple FIFO queue
- OperationApplierService uses fail-fast approach
- ArchiveOperationHandler is unified for local and remote operations
- Update diagrams and file references
- Add "Recently Completed" items for December 2025
2025-12-12 20:48:13 +01:00
Johannes Millan
5ffc109943 refactor(sync): unify archive operation handling
Consolidate all archive-affecting operations into ArchiveOperationHandler
to eliminate duplicate code and create a single source of truth.

Changes:
- Add ArchiveOperationHandlerEffects for local operations
- Rename handleRemoteOperation to handleOperation
- Add isArchiveAffectingAction helper for action filtering
- Make isIgnoreDBLock conditional on action.meta.isRemote
- Remove archive logic from individual effect files
- Update feature-stores.module.ts to use new effect
- Add archive write points documentation

This ensures archive operations (moveToArchive, restoreTask,
flushYoungToOld, deleteProject, deleteTag, deleteTaskRepeatCfg,
deleteIssueProvider) all go through one handler for both local
and remote operations.
2025-12-12 20:48:12 +01:00
Johannes Millan
320e57e622 refactor(tasks): use satisfies instead of as for PersistentActionMeta
Improves type safety by using satisfies which validates the object
matches the type at compile time, rather than as which is just an
assertion.
2025-12-12 20:48:12 +01:00
Johannes Millan
69f3602927 docs(sync): update review with completed fixes
Mark as fixed:
- Bounds checking in planner-shared.reducer.ts
- Error handling in tag.effects.ts (async IndexedDB operations)
2025-12-12 20:47:48 +01:00
Johannes Millan
8187313e0c docs(sync): mark TaskDueEffects race conditions as false positive
Analysis shows the alleged race conditions are intentional defensive patterns:
- Double sync wait ensures stability during date changes
- Effects are complementary with staggered timing, not competing
- Infinite loop scenario explicitly handled in code

Note: Tests exist but are disabled due to Dropbox SDK mocking issues.
2025-12-12 20:47:48 +01:00
Johannes Millan
89066aca51 docs(sync): mark subtask orphan detection as false positive
Re-evaluated the "missing subtask cascade in tag deletion" issue.
The current behavior is correct:
- A subtask with a surviving parent is NOT orphaned - it's accessible
  through the parent in the UI
- If a parent is orphaned, its subtasks ARE correctly deleted
- Subtasks always have parentId, so they're never directly orphaned

This is intentional behavior, not a bug.
2025-12-12 20:47:48 +01:00
Johannes Millan
e6025b4178 fix(sync): validate state before saving snapshot in hydrator
The snapshot was being saved before validation ran. If validation found
corruption and repaired it, the snapshot was stale/wrong. This fix
moves validation (CHECKPOINT C) BEFORE saving snapshot in both:
- Tail replay path (after replaying ops following a snapshot)
- Full replay path (when no snapshot exists)

Now the snapshot contains validated/repaired state.

Includes regression tests that verify validation happens before snapshot
save in both code paths.
2025-12-12 20:47:48 +01:00
Johannes Millan
bbd50c223a fix(sync): persist compaction counter when no cache exists
The incrementCompactionCounter() function returned 1 without persisting
when no state cache existed. This caused the compaction threshold to
never be reached, leading to unbounded operation log growth.

Also updated review docs:
- Marked compaction counter bug as fixed
- Identified memory leak in TaskElectronEffects as false positive
  (take(1) ensures automatic cleanup)
- Identified TODAY_TAG "board-style" claim as false positive
  (code correctly REMOVES TODAY_TAG from task.tagIds)
- Identified cache invalidation claim as false positive
  (sequence-based invalidation works correctly)
- Identified repair mutex race condition as false positive
  (JavaScript single-threaded, no await between assignment)
2025-12-12 20:47:48 +01:00
Johannes Millan
78c65acf4d test(e2e): add sd:today to tasks for TODAY view visibility
Tasks created without explicit dueDay don't appear in TODAY view
due to recent selector changes. Add sd:today short syntax to ensure
tasks have dueDay set and appear correctly in tests.
2025-12-12 20:47:48 +01:00
Johannes Millan
60bc0e78d7 docs(sync): replace PFAPI with operation log sync for all providers 2025-12-12 20:47:47 +01:00
Johannes Millan
e62f2f86fa feat(e2e): add SuperSync E2E test infrastructure for multi-client sync
- Add test mode to super-sync-server (TEST_MODE=true env var)
- Add /api/test/create-user endpoint for auto-verified test users
- Add /api/test/cleanup endpoint for wiping test data
- Create SuperSyncPage page object for sync configuration
- Create supersync-helpers.ts with test utilities
- Add 6 E2E sync scenarios:
  - 2.1: Client A creates task, Client B downloads
  - 2.2: Both clients create different tasks
  - 1.3: Update propagates between clients
  - 1.4: Delete propagates between clients
  - 3.1: Concurrent edits handled gracefully
  - 2.3: Parent/subtask relationships sync correctly
2025-12-12 20:47:41 +01:00
Johannes Millan
c977ad3edd test(oplog): add integration tests for multi-client sync and state consistency
Add Karma-based integration tests for the operation log system using real
IndexedDB. Tests cover multi-client synchronization scenarios and state
consistency verification.

New test files:
- multi-client-sync.integration.spec.ts (16 tests)
- state-consistency.integration.spec.ts (17 tests)

Test utilities:
- TestClient helper for simulating multiple clients with independent vector clocks
- Operation factory helpers for creating test operations
2025-12-12 20:46:27 +01:00
Johannes Millan
f661de10de feat(sync): implement time tracking sync and deterministic archive flush
- Add syncTimeSpent persistent action for batched time tracking sync
  - Accumulates duration per task, syncs every 5 minutes or on task stop
  - Local dispatch: no-op (state already updated by second-by-second ticks)
  - Remote dispatch: applies batched duration to task.timeSpentOnDay

- Fix remote archive handling to include time tracking data
  - writeTasksToArchiveForRemoteSync now moves historical time tracking
    data to archiveYoung alongside tasks

- Add flushYoungToOld persistent action for deterministic archive state
  - Replaces inline flush logic with action-based approach
  - Remote clients replay the flush operation to maintain consistent
    archiveOld state without syncing large files
  - Vector clock ordering ensures flush happens after preceding operations
2025-12-12 20:46:26 +01:00
Johannes Millan
fd44db3852 docs(sync): update planning doc 2025-12-12 20:46:26 +01:00
Johannes Millan
410bcde114 feat(sync): implement receiver-side migration shield and fix tests
- Implement processRemoteOps migration logic in OperationLogSyncService using SchemaMigrationService.
- Add MAX_VERSION_SKIP check and user warning via SnackService.
- Fix Log class to support dynamic level switching (fixes PFAPI tests).
- Fix operation-log-stress imports.
- Skip disabled MultiTabCoordinatorService tests.
- Add unit tests for migration logic.
2025-12-12 20:46:26 +01:00
Johannes Millan
4e018a8064 docs: update operation log architecture paths 2025-12-12 20:46:26 +01:00
Johannes Millan
d79dc153fe docs: add architectural introduction to operation log documentation 2025-12-12 20:46:26 +01:00
Johannes Millan
04728267e2 docs: update operation log architecture docs to reflect recent changes
- Update ValidateStateService documentation to reflect better error handling for isRelatedModelDataValid.
- Update Part C status to Complete as Server Sync is implemented.
- List implemented components for Part C.
- Fix lint error in startup.service.ts
2025-12-12 20:46:26 +01:00
Johannes Millan
6818ae8d9f feat(oplog): add persistent compaction counter and syncedAt index
- Store compaction counter in state_cache to share across tabs/restarts
- Add bySyncedAt index for faster getUnsynced() queries
- Remove in-memory opsSinceCompaction counter from effects
- Reset counter in compaction service after successful compaction
2025-12-12 20:46:26 +01:00
Johannes Millan
b73769aa15 feat(oplog): implement migration safety and tail ops migration
Implements A.7.12, A.7.13, and A.7.15 from the operation log architecture:

- A.7.12 Migration Safety: Backup state_cache before migration, restore
  on failure, detect interrupted migrations on startup
- A.7.13 Tail Ops Migration: Migrate operations to current schema before
  replay during hydration
- A.7.15 Unified Interface: Extend SchemaMigration with migrateOperation
  and requiresOperationMigration for explicit migration decisions

Changes:
- operation-log-store: Add backup/restore methods for state_cache
- schema-migration: Add operation migration support and validation
- operation-log-hydrator: Integrate backup safety and tail ops migration
- Update architecture docs to reflect implementation status
2025-12-12 20:46:26 +01:00
Johannes Millan
36dd2ee38b docs(sync): replace hybrid manifest mindmap with detailed flow diagrams
Replace the single mindmap with comprehensive diagrams:
- 6.1 Data Lifecycle: Hot → Cold → Frozen
- 6.2 Manifest File Structure
- 6.3 Write Path: Buffer vs Overflow Decision
- 6.4 Read Path: Reconstructing State
- 6.5 Compaction: Freezing State
- 6.6 Request Count Comparison table
2025-12-12 20:46:26 +01:00
Johannes Millan
4554c33074 docs(sync): add A.7.15 unified state and operation migrations
Document the relationship between state migrations (for snapshots) and
operation migrations (for tail ops/sync). Key points:

- Single SchemaMigration interface with both migrateState and migrateOperation
- requiresOperationMigration flag forces explicit decision
- Startup validation catches missing operation migrations
- Examples for field rename and feature removal migrations
- Explains why auto-derivation doesn't work (UPDATE payloads differ from entities)
- Updated implementation status table with A.7.15
2025-12-12 20:46:26 +01:00
Johannes Millan
05ecf23b69 docs(sync): expand hybrid manifest architecture with detailed workflows
- Add complete type definitions (HybridManifest, SnapshotFile, etc.)
- Add ASCII flow diagrams for upload, download, and snapshot workflows
- Document edge cases and failure modes:
  - Concurrent uploads with ETag/rev handling
  - Manifest corruption recovery
  - Missing snapshot handling
  - Schema version mismatch
  - Large pending operations batching
- Add configuration constants reference
- Add implementation plan with 4 phases
- Add file reference for remote storage layout
2025-12-12 20:46:26 +01:00
Johannes Millan
7362eb13dd docs(sync): improve architecture doc accuracy and add missing considerations
- Fix status claims: Part A "Functional" (not Complete), Part C "Design Only"
- Add Known Gaps callout for critical unimplemented features (A.7.12, A.7.13)
- Fix code bug in C.5 conflict resolution (variable out of scope)
- Fix broken markdown (4 backticks instead of 3)
- Add "Edge Cases & Missing Considerations" section:
  - IndexedDB quota exhaustion handling
  - Compaction trigger coordination between tabs
  - Genesis migration with partial data
- Restructure Implementation Status by Part (A, B, C, D)
- Add note to diagrams doc clarifying planned vs implemented features
- Fix mermaid syntax errors in hybrid manifest diagram
2025-12-12 20:46:26 +01:00
Johannes Millan
6489760466 docs: add hybrid manifest mindmap to architecture diagrams 2025-12-12 20:46:25 +01:00
Johannes Millan
0b5f2c7002 docs(sync): update planning doc 2025-12-12 20:46:25 +01:00
Johannes Millan
386c297ade docs: Document Hybrid Manifest & Snapshot Architecture
Introduces a new documentation file detailing the Hybrid Manifest and Snapshot architecture
designed to improve efficiency for file-based sync providers like WebDAV and Dropbox.
This approach reduces API requests and file proliferation by embedding small operations
directly into the manifest file and periodically consolidating changes into snapshots.

Also updates the architecture diagrams to include a visualization of this new hybrid approach.

Additionally, a "Conflict-Aware Migration Strategy" section was added to the existing
operation-log-architecture.md during initial codebase investigation to provide further
context on managing schema version differences during sync.
2025-12-12 20:46:25 +01:00
Johannes Millan
fb590c7fd6 docs: improve mermaid diagrams with file names 2025-12-12 20:46:25 +01:00
Johannes Millan
4cfd73cbd6 docs: add mermaid diagrams for operation log architecture and sync 2025-12-12 20:46:25 +01:00
Johannes Millan
a31de900db docs: clean up execution plan by removing completed tasks and focusing on future enhancements 2025-12-12 20:46:25 +01:00
Johannes Millan
546fd54573 docs: clarify state model in migration guide to AllSyncModels 2025-12-12 20:46:25 +01:00
Johannes Millan
3d00507fa1 docs: add guide on creating new migrations to operation log architecture 2025-12-12 20:46:25 +01:00