Commit graph

322 commits

Author SHA1 Message Date
Johannes Millan
b68c6c2497 docs: clarify relation between legacy CROSS_MODEL_MIGRATION and operation log migrations 2025-12-12 20:46:25 +01:00
Johannes Millan
4408fbbbb0 docs(sync): update planning doc 2025-12-12 20:46:25 +01:00
Johannes Millan
03e3c3677d docs(operation-log): expand schema migration documentation
Add comprehensive documentation covering:
- Schema version concepts and migration triggers
- Local snapshot migration during app updates
- Operation replay with mixed versions
- Cross-version client sync scenarios
- Full state import handling (SYNC_IMPORT/BACKUP_IMPORT)
- Migration implementation patterns with examples
- Version handling in operations
- Design principles for writing migrations
- Handling unsupported versions (too old/too new)
- Future considerations for migration enhancements
2025-12-12 20:46:25 +01:00
Johannes Millan
d22fbe28b2 feat(operation-log): add data validation and repair system
Integrate PFAPI's validation/repair with operation logs to prevent
corruption and automatically recover from invalid states.

Key features:
- REPAIR operation type with full repaired state + summary
- 4 validation checkpoints (A: before write, B: after snapshot load,
  C: after replay, D: after sync)
- Payload validation before IndexedDB write (lenient structural checks)
- Full Typia + cross-model validation at state checkpoints
- User notification when auto-repair occurs
- Infinite loop prevention during repair

New files:
- validate-state.service.ts: wraps PFAPI validation + repair
- validate-operation-payload.ts: checkpoint A payload validation
- repair-operation.service.ts: REPAIR operation creation
2025-12-12 20:46:25 +01:00
Johannes Millan
37b8ec392f docs(sync): update planning doc 2025-12-12 20:46:25 +01:00
Johannes Millan
73ac516608 docs(sync): update 2025-12-12 20:46:21 +01:00
Johannes Millan
c87e7ec218 refactor(sync): use uuidv7 library for UUID generation
Replace custom UUID v7 implementations with the standard uuidv7 library.
The custom implementation had a bug with JavaScript bitwise operators
not handling 48-bit timestamps correctly.
2025-12-12 20:46:19 +01:00
Johannes Millan
c3ef2436dc fix(effects): skip remote actions in task effects
Add filterLocalAction() operator to filter out remote actions from sync.
Effects that handle UI feedback (snackbars, sounds) or derive additional
actions should not run for operations replayed from other clients.
2025-12-12 20:46:17 +01:00
Johannes Millan
94c1123c1f docs(op-log): update architecture doc and fix compaction service
- Update architecture doc to reflect actual implementation state:
  - Replace action blacklist with isPersistent pattern
  - Mark Part C (server sync) as complete (was incorrectly "not started")
  - Update file reference section with all current files
  - Update compaction config to show 7-day retention window
  - Remove obsolete "models requiring migration" section
- Fix compaction service:
  - Use PfapiStoreDelegateService for consistency
  - Add schemaVersion to state cache saves
2025-12-12 20:46:13 +01:00
Johannes Millan
cdaedf4fb8 feat(sync-server): implement operation log sync API
Add server-side operation log synchronization:

- Database schema: operations, user_sync_state, sync_devices, tombstones
- SyncService: upload/download ops, snapshots, rate limiting, validation
- REST API: POST/GET /api/sync/ops, /snapshot, /status, /devices/:id/ack
- Cleanup jobs: daily tombstone/old ops cleanup, hourly stale device cleanup
- Compression: gzip for snapshots
- Security: timestamp validation, payload size limits, idempotency

Endpoints:
- POST /api/sync/ops - Upload operations batch
- GET /api/sync/ops?sinceSeq=N - Download ops since sequence
- GET /api/sync/snapshot - Get full state snapshot
- POST /api/sync/snapshot - Upload full state (migration/recovery)
- GET /api/sync/status - Get sync status
- POST /api/sync/devices/:clientId/ack - Acknowledge sequences
2025-12-12 20:46:12 +01:00
Johannes Millan
5d7bc5d7fb feat(op-log): add disaster recovery and schema migration
A.6: Disaster Recovery
- Add snapshot validation in hydrateStore()
- Implement _attemptRecovery() to restore from legacy 'pf' database
- Add _hasUsableData() and _recoverFromLegacyData() helpers
- Add 'RECOVERY' entity type for recovery operations

A.7: Schema Migration Service
- Create SchemaMigrationService with migration infrastructure
- Define MigratableStateCache interface with schemaVersion
- Integrate migrations into hydration flow
- Update state_cache schema to include schemaVersion

Also updates architecture doc to reflect Part A/B completion.
2025-12-12 20:46:12 +01:00
Johannes Millan
302eec4758 docs: plan operation logs server 1 2025-12-12 20:46:12 +01:00
Johannes Millan
aa05c1638f docs: plan operation logs again 13 2025-12-12 20:46:12 +01:00
Johannes Millan
7f1ee6ea74 docs: plan operation logs again 13 2025-12-12 20:46:12 +01:00
Johannes Millan
73cf144ed4 docs: plan operation logs again 12 2025-12-12 20:46:12 +01:00
Johannes Millan
e53c581621 docs: plan operation logs again 11 2025-12-12 20:46:12 +01:00
Johannes Millan
2e7bd3a660 docs: update operation log docs with B-Lite implementation status
Mark Phase 0.1, 0.2, and 0.2.5 as complete:
- Legacy sync stale data: Implemented PfapiStoreDelegateService
- Feature flag: useOperationLogSync already existed
- SaveToDbEffects gating: Now filtered by feature flag

Update architecture to reflect B-Lite delegate pattern:
- Exclusive persistence based on feature flag
- PfapiStoreDelegateService reads from NgRx store
- Backward compatibility flush when disabling op-log mode
2025-12-12 20:46:12 +01:00
Johannes Millan
9f70a28bf5 feat(sync): implement B-Lite PFAPI integration for operation log sync
When useOperationLogSync is enabled:
- SaveToDbEffects is completely disabled (no writes to 'pf' database)
- Legacy sync (WebDAV/Dropbox) reads from NgRx store via delegate
- Non-NgRx models (reminders, archives, plugins) are loaded from 'pf' DB on-demand

Key changes:
- Add PfapiStoreDelegateService to read sync data from NgRx store
- Wire up delegate in PfapiService based on useOperationLogSync config
- Flush NgRx state to 'pf' database when disabling op-log mode

This approach simplifies PFAPI/OpLog integration by making SUP_OPS
the write path while keeping PFAPI functional for legacy sync.
2025-12-12 20:46:12 +01:00
Johannes Millan
88a987b184 docs: plan operation logs again 10 2025-12-12 20:46:12 +01:00
Johannes Millan
d174eae989 docs: plan operation logs again 9 2025-12-12 20:46:06 +01:00
Johannes Millan
4ef436be74 feat(sync): implement Phase 0 critical blockers for operation log
- Add useOperationLogSync feature flag to SyncConfig (default: false)
- Restore and gate SaveToDbEffects by feature flag
- Add provider gating in sync.service.ts (op-log disabled for all providers)
- Add delegate pattern in Pfapi.getAllSyncModelData() for NgRx reads
- Add dependency retry queue in OperationApplierService (max 3 retries)
- Update tests to provide required mocks

When useOperationLogSync is false (default), legacy sync behavior is
preserved. This ensures backward compatibility while laying groundwork
for future operation log sync support.
2025-12-12 20:46:05 +01:00
Johannes Millan
849a6cac4b docs: plan operation logs again 8 2025-12-12 20:46:05 +01:00
Johannes Millan
254df59615 docs: plan operation logs again 7 2025-12-12 20:46:05 +01:00
Johannes Millan
0f33d14df1 docs: plan operation logs again 6 2025-12-12 20:46:05 +01:00
Johannes Millan
f9464b347a docs: plan operation logs again 5 2025-12-12 20:46:05 +01:00
Johannes Millan
a270e7fda8 docs: plan operation logs again 4 2025-12-12 20:46:05 +01:00
Johannes Millan
9c12165e46 docs: plan operation logs again 3 2025-12-12 20:46:05 +01:00
Johannes Millan
9f1f0bc7a4 docs: plan operation logs again 2 2025-12-12 20:46:05 +01:00
Johannes Millan
f8e0160801 docs: plan operation logs again 1 2025-12-12 20:46:05 +01:00
Johannes Millan
0dd40a70c0 docs: analysis of delta sync 3 2025-12-12 20:46:05 +01:00
Johannes Millan
cb03bba281 docs: analysis of delta sync 2 2025-12-12 20:46:05 +01:00
Johannes Millan
ad31746db9 docs: analysis of delta sync 2025-12-12 20:46:05 +01:00
Johannes Millan
e1679d4b16 docs: plan operation log sync 14 2025-12-12 20:46:05 +01:00
Johannes Millan
911051fbf2 docs: plan operation log sync 12 2025-12-12 20:46:05 +01:00
Johannes Millan
7d47f37803 docs: plan operation log sync 11 2025-12-12 20:46:05 +01:00
Johannes Millan
22fa68e609 docs: plan operation log sync 10 2025-12-12 20:46:05 +01:00
Johannes Millan
17f1395d1b docs: plan operation log sync 9 2025-12-12 20:46:05 +01:00
Johannes Millan
f459f1e0c7 docs: plan operation log sync 6 2025-12-12 20:46:05 +01:00
Johannes Millan
21e1af25ef docs: plan operation log sync 3 2025-12-12 20:46:05 +01:00
Johannes Millan
5d69ba4761 docs: plan operation log sync 2 2025-12-12 20:46:05 +01:00
Johannes Millan
4679439084 docs: plan operation log sync 2025-12-12 20:46:05 +01:00
Johannes Millan
93c76d5cdb feat(syncServer): implement JWT authentication and user registration/login functionality 2025-12-12 20:46:04 +01:00
Johannes Millan
8e65d7263b feat(syncServer): refactor server functions to use arrow function syntax and improve error handling 2025-12-12 20:46:04 +01:00
Johannes Millan
ed1bc59a0a feat(syncServer): update plan 2025-12-12 20:46:04 +01:00
Johannes Millan
dfc84a2f88 feat(syncServer): add configuration file and enhance server setup 2025-12-12 20:46:04 +01:00
Johannes Millan
648ce47249 docs: add cool new plans 2025-12-07 13:41:55 +01:00
Mustache Games
273a1077d7 docs(plugin-dev): improve whitespaces
- Blank lines in Quick Start JS (after log/notification/counter)
- Spacing in Data Operations subsections (Tasks/Projects/Tags/Simple Counters)
2025-11-17 21:08:02 +02:00
Mustache Games
afa2f60450 docs(plugin-dev): document simple counters API
- Add "Simple Counters" subsection with basic/full model tables, validation/examples (regex, dates)
- Integrate counter demo in Quick Start (`plugin.js`: set/inc on button, snack update)
- Tie-in Hooks ACTION example: increment counter on task add
- List community plugins in Resources (`counter-tester-plugin`, `sp-reporter`)
- Clean Data Operations formatting
2025-11-17 20:55:11 +02:00
Johannes Millan
05f088a900 docs: add performance report 2025-11-13 18:23:50 +01:00
johannesjo
a0761732c2 docs: clarify certificate selection with 'For use in Xcode 11 or later' description 2025-11-10 16:06:58 +01:00