mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-07-18 17:05:48 +00:00
4c239e5691
* docs(sync): add SQLite migration plan + Phase A adapter port skeleton Documents the op-log persistence migration off WebView IndexedDB into app-private SQLite on native (Capacitor), addressing the data-loss class where Android can evict WebView storage when no sync is configured. Phase A skeleton (no behavior change, not yet wired in): - OpLogDbAdapter / OpLogTx: backend-agnostic persistence port with a callback-based transaction() as the atomicity seam both IndexedDB and SQLite map onto. - OP_LOG_DB_SCHEMA: declarative SUP_OPS schema descriptor (mirrors db-upgrade.ts v6) that both backends can consume. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * feat(op-log): add IndexedDbOpLogAdapter implementing the persistence port Phase A continuation of the SQLite migration (docs/sync-and-op-log/ sqlite-migration.md). Implements the IndexedDB backend behind the OpLogDbAdapter port: open-retry with the existing budgets, versionchange/ close re-open handling, IndexedDBOpenError wrapping, index/range queries, and a callback-based transaction() that commits on resolve and aborts on throw — the atomicity seam both backends share. Extends the port with cursor-style iterate() (continue/stop/delete/ delete-stop) to cover the latest-entry lookups and predicate pruning the store does today, plus a close() teardown hook. Spec exercises CRUD, the unique byId index, range queries, cursor direction/stop/delete, and — critically — multi-store transaction commit and rollback against fake-indexeddb. 10/10 pass. Not yet wired into OperationLogStoreService; additive scaffolding only. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): harden persistence port after multi-agent review Addresses blocking fidelity gaps found reviewing the adapter against the real store's usage, so the upcoming store refactor can be behavior- preserving: - iterate() visitor is now synchronous and receives the primary key. An async visitor could await real I/O mid-cursor, letting the IDB transaction auto-commit and the next continue() throw TransactionInactiveError. Synchronous-only also lets a buffered SQLite backend honor it without materializing the whole result set. - DbIterateOptions.query positions an index cursor at an exact key (clearFullStateOpsExcept's keyed delete). - getAll()/count() take an optional primary-key range (getOpsAfterSeq and the getUnsynced/getAppliedOpIds incremental caches use getAll(OPS, lowerBound(seq))). - getKeyFromIndex() for cheap existence probes (appendBatchSkipDuplicates' getKey, avoids deserializing the value). Tests expanded 10 -> 26: destructive clear()+delete() rollback, abort on inner-op rejection, transactional reads/index/cursor, readonly mode, keyed index iteration, compound-index match, getAll/count ranges, the close() re-open cliff, and the open-retry budgets via the _openDbOnce seam. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): route import-backup methods through the persistence adapter First method group of the Phase A store migration. Adds an adoptConnection() seam so IndexedDbOpLogAdapter operates on the store's existing connection rather than opening a second one to SUP_OPS (avoiding versionchange deadlocks and doubled close/upgrade handling during the transition). The store adopts/releases the connection alongside its own _db in init()/close/versionchange. saveImportBackup / loadImportBackup / clearImportBackup / hasImportBackup now go through the adapter. Behavior is identical — same connection, same store, same keys. Verified: 170 store unit specs, 26 adapter specs, 3 archive specs, and the import-sync integration spec all green. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): route state_cache + compaction methods through the adapter Second method group of the Phase A store migration. saveStateCache, loadStateCache, the migration-safety backup methods (save/load/clear/has/ restore), and the compaction counter (get/increment/reset) now go through the shared adapter. The two atomic read-modify-write methods (incrementCompactionCounter, resetCompactionCounter) use the adapter's callback transaction(), preserving their single-transaction semantics. Introduces a StateCacheEntry type; `id` is optional so the read-side return types stay assignable from the looser snapshot shapes callers construct (the pre-migration return types didn't surface `id`). Verified: 170 store unit, 53 compaction unit, 27 vector-clock, 20 compaction integration specs all green; full tsc clean. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * docs(sync): track Phase A migration progress in sqlite-migration.md https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): route ops-table append + markApplied through the adapter Third method group of the Phase A store migration — the higher-risk write path. append, appendBatch, appendBatchSkipDuplicates and markApplied now go through the shared adapter. The batch methods use the adapter's callback transaction() (one atomic unit, same as before); the TOCTOU-free duplicate guard uses tx.getKeyFromIndex (the byId unique index probe, issue #6343). ConstraintError->DUPLICATE and QuotaExceededError-> StorageQuotaExceededError mappings are preserved — the adapter rethrows the original DOMException so the store's catch blocks still fire. Verified: 170 store unit + 367 op-log integration specs green; tsc clean. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): route ops-table reads + full-state clears through adapter Fourth method group. getPendingRemoteOps (compound-index match expressed as a degenerate [k,k] range, with the pre-v3 fallback scan preserved), hasOp, getOpById, getOpsAfterSeq (primary-key range), the two reverse- cursor latest-full-state lookups, and clearFullStateOps/ clearFullStateOpsExcept now go through the adapter's iterate()/getAll()/ getAllFromIndex(). The keyed-index-cursor delete is factored into a _deleteOpsByIds() helper using iterate({index, query}) + delete-stop in a single atomic transaction, matching the prior behavior (no-op + no cache invalidation on empty list). Verified: 170 store unit + 367 op-log integration specs green; tsc clean. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): route unsynced/applied caches + mark methods through adapter Fifth method group. The getUnsynced/getAppliedOpIds incremental cache builds (getAll with a primary-key range), getFailedRemoteOps (compound index), markSynced/markRejected/clearUnsyncedOps/markFailed (transactional get+put loops), deleteOpsWhere (predicate cursor delete) and getLastSeq (reverse cursor reading the primary key via the iterate visitor's key arg) now go through the adapter. markFailed keeps its original behavior of NOT invalidating the unsynced cache. Verified: 170 store unit + 367 op-log integration specs green; tsc clean. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): route remaining store methods through adapter Final OperationLogStoreService group — every method now goes through the persistence adapter; no direct `this.db` calls remain. Covers hasSyncedOps (bySyncedAt index cursor), clearAllOperations, _clearAllDataForTesting (multi-store clear in one transaction), the vector-clock accessors, and the two flagship atomic flows: - appendWithVectorClockUpdate (OPS + VECTOR_CLOCK in one transaction) - runDestructiveStateReplacement (OPS + STATE_CACHE + VECTOR_CLOCK + CLIENT_ID + archive). The hand-rolled try/abort is replaced by the adapter's commit-on-resolve / abort-on-throw transaction(); success-only cache + clientId-cache invalidation now runs after the resolved transaction. The #7709 interrupt atomicity tests still pass — the adapter operates on the same adopted connection the tests spy on, so a poisoned opsStore.add still aborts and unwinds the queued clientId rotation. Verified: 170 store unit + 367 op-log integration specs (incl. the 3 clean-slate-interrupt atomicity tests) green; tsc clean. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(op-log): route ArchiveStoreService through the persistence adapter Completes the Phase A store/archive migration. ArchiveStoreService gets its own IndexedDbOpLogAdapter that adopts its independent SUP_OPS connection (released on close/versionchange and on the iOS connection-closing retry path in _withRetryOnClose). All six accessors plus saveArchivesAtomic/_clearAllDataForTesting now go through the adapter; the dead `db` getter and its unused error constant are removed. No direct `this.db` calls remain in either persistence service. Verified: 3 archive unit + 170 store unit + 367 op-log integration specs green; tsc clean. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * refactor(sync): fix readonly cursor regression + multi-review findings Multi-agent review of the Phase A op-log adapter (post full store/cursor migration). Addresses one live regression plus hardening; no functional behavior change. W1 (live regression fix): the migrated read-only cursor methods — getLastSeq, hasSyncedOps, getLatestFullStateOp(Entry) — ran through iterate(), which always opened a 'readwrite' transaction, so pure reads on the hot ops store took an exclusive write lock and serialized against appends (pre-migration they were 'readonly'). Add `mode` to DbIterateOptions (default 'readwrite' so delete-walks keep working) and pass `mode:'readonly'` from those four readers; clearFullStateOps* delete-walks stay readwrite. W2: op-log-db-schema reuses DB_NAME/DB_VERSION from db-keys.const instead of re-literaling 'SUP_OPS'/6 (no third source of truth), and a new op-log-db-schema.spec.ts asserts the descriptor matches both DB_VERSION and the stores/indexes runDbUpgrade actually creates (the contract Phase B builds on). W3: test the adoptConnection seam (both branches) — ops route onto an adopted external connection, and adoptConnection(undefined) returns to the not-initialized cliff (the store's close/versionchange path). W4: convert the two open-retry specs from real ~8s backoff sleeps to fakeAsync + tick (adapter spec ~0.04s vs ~8s) and assert exact attempt budgets; add a full-lock-budget case. Gates: adapter 30 + schema 2 + store 170 unit, and 59 op-log integration specs (race-conditions, multi-entity-atomicity, compaction, server-migration, clean-slate-interrupt, indexeddb-error-recovery) green; checkFile clean. * refactor(op-log): inject the persistence adapter via DI token Phase B step 1. Both persistence services now obtain their OpLogDbAdapter from OP_LOG_DB_ADAPTER_FACTORY instead of constructing IndexedDbOpLogAdapter directly. The token vends a factory (not a singleton) because each service adopts its own connection into its own adapter instance. Defaults to IndexedDB on all platforms; Phase B step 2 will override it to return a SqliteOpLogAdapter when running native, with the stores untouched. adoptConnection() becomes an optional bridge method on the OpLogDbAdapter interface — documented as IDB-transition-only; a self-managing backend (SQLite) leaves it undefined and callers guard with `?.()`. Verified: 170 store unit + 3 archive unit + 367 op-log integration specs green; tsc clean. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * feat(op-log): add SqliteOpLogAdapter skeleton (Phase B, no native dep) Dependency-free skeleton of the SQLite backend behind the OpLogDbAdapter port. Solves the hard schema-mapping question the reviewers flagged without pulling in a native plugin or anything untestable in CI: - planTables()/buildDdl(): derive the physical SQL layout from the shared OP_LOG_DB_SCHEMA. Each store -> a table with a JSON `value` column plus one extracted column per IDB index. ops gets `seq INTEGER PRIMARY KEY AUTOINCREMENT` (monotonic, never-reused — matches IDB + getLastSeq), `op_id TEXT UNIQUE` (byId), `synced_at` (bySyncedAt) and a composite (source, application_status) index. keyPath stores -> TEXT PK from the keyPath; keyless singletons -> caller-supplied TEXT key. - A minimal SqliteDb port (run/query) the adapter talks to instead of importing @capacitor-community/sqlite, so this file has no native dependency and is unit-testable with a fake. - init() applies the DDL (idempotent); query/tx methods throw a loud not-implemented error (fail loudly rather than silently lose data) with the intended SQL documented per method. adoptConnection is intentionally absent — SQLite self-manages its handle. 12 specs cover the plan/DDL derivation and that init() emits the expected DDL. Doc updated with status + the deferred native-dependency decision. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * feat(op-log): fully implement SqliteOpLogAdapter (still no native dep) Completes the SQLite backend behind the OpLogDbAdapter port. All query, index, range, count, cursor-iterate and transaction methods are now implemented against the minimal SqliteDb port: - value→column extraction: each store row stores the JSON object in a `value` column plus extracted columns for the indexed paths (op_id/synced_at/source/application_status); writes populate them. - transactions map to BEGIN IMMEDIATE/COMMIT/ROLLBACK with rollback-on- throw; readonly iterate/transaction use no write lock. - SQLite errors map to the SAME DOMException names the store's existing catch blocks expect: UNIQUE→ConstraintError (→DUPLICATE_OPERATION_ERROR), disk-full→QuotaExceededError (→StorageQuotaExceededError). - ops uses AUTOINCREMENT so seq is monotonic and never reused across clear() — matching IDB + getLastSeq. Still imports no native plugin: a thin wrapper over @capacitor-community/sqlite's SQLiteDBConnection will satisfy SqliteDb on device. 23 specs validate the translation layer + transaction semantics (commit/rollback/abort-on-unique) against an in-memory SQLite stand-in; a real-engine on-device run is the remaining Phase B step (documented). https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * docs(sync): add SQLite migration follow-up backlog Actionable, ordered backlog companion to sqlite-migration.md: - Track A: ship the #7892 safeguards now (persist() diagnostics + native filesystem auto-backup) — independent of SQLite, recommended near-term fix. - Track B: finish the native SQLite backend (plugin + SqliteDb wrapper, real-engine validation, DI flip behind a flag). - Track C: one-time IDB→SQLite data migration, staged rollout. - Track D: cleanup once SQLite is the native default. https://claude.ai/code/session_011wcqZgubKqoT6wxt1L1KBT * fix(op-log): scan full-state ops read-only to drop the write lock clearFullStateOps / clearFullStateOpsExcept iterate the ops store only to collect ids (the delete runs in a separate transaction), but the migrated iterate() defaulted to 'readwrite' — so these pure-read scans took an exclusive write lock on the hot ops store and serialized against appends. Pre-adapter (master) these scans used a readonly cursor. Pass mode:'readonly' to restore parity. Same regression class the earlier W1 fix addressed for getLastSeq/hasSyncedOps/getLatestFullStateOp(Entry); these two scans were missed because they are no longer delete-walks. Verified: 170 store unit + server-migration/import-sync/remote-apply/ vector-clock-import integration specs green; checkFile + tsc clean. * fix(op-log): correct SQLite seq round-trip + enforce tx scope and readonly Hardens the dormant SQLiteOpLogAdapter against three multi-review findings (translation-layer only; the backend is still wired to nothing): - C1 (data duplication): the autoinc `ops` PK (`seq`) lived only in its own column, never the JSON value, and was never re-injected on read. So reads returned seq===undefined and put() emitted INSERT…ON CONFLICT(seq) with no seq bound — the conflict never fired and every mark*/clearUnsynced re-put inserted a duplicate row. Now buildInsert binds seq when the value carries one (re-put / explicit-seq add) and decodeRow injects the PK back from a `__pk` alias on every read, matching IDB's keyPath+autoIncrement store. ON CONFLICT no longer overwrites the PK column. - W2 (atomicity scope): transaction() discarded its `stores` argument, so the OpLogTx could touch any store — silently passing where IDB throws. The tx now enforces the declared scope (and inherits the tx mode for iterate). - W3 (readonly contract): a delete action under a readonly iterate executed the DELETE outside any transaction; it now rejects with ReadOnlyError, matching IDB. Also makes the in-memory FakeSqliteDb faithfully model AUTOINCREMENT (honor an explicit seq, upsert on PK conflict, advance the high-water mark) so the spec actually catches C1-class bugs — verified: reverting the seq fix makes the new "updates in place" test fail with the real UNIQUE violation. Verified: 27 SQLite adapter specs (4 new) green; checkFile + tsc clean. * refactor(op-log): strip the autoinc keyPath prefix via extractPath idiom Follow-up review nit: decodeRow stripped the `$.` from the autoinc keyPath with slice(2); use the same `.replace(/^\$\./, '')` idiom as extractPath for consistency, and note that the autoinc keyPath is a top-level field. No behavior change (keyJsonPath is always `$.seq` for the only autoinc store). --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| long-term-plans | ||
| plans | ||
| promotion | ||
| research | ||
| screens | ||
| sync-and-op-log | ||
| wiki | ||
| add-new-integration.md | ||
| apple-release-automation.md | ||
| build-and-publish-notes.md | ||
| documentation-guide.md | ||
| ENV_SETUP.md | ||
| github-access-token-instructions.md | ||
| gitlab-access-token-instructions.md | ||
| how-to-rate.md | ||
| howto-refresh-snap-credentials.md | ||
| i18n-script-usage.md | ||
| legacy-webview-analysis.md | ||
| mac-app-store-code-signing-guide.md | ||
| performance-project-tag-report.md | ||
| plugin-development.md | ||
| styling-guide.md | ||
| theming-contract.md | ||
| TRANSLATING.md | ||
| unused-translations-analysis.md | ||
| update-android-app.md | ||
| update-mac-certificates.md | ||