fix(persistence): increase IndexedDB retry window for session-restart locks (#7220)

* fix(persistence): increase IndexedDB retry window for session-restart locks

On Linux desktop environments (especially Flatpak with autostart),
logging out and back in can leave stale LevelDB locks for 5-15+
seconds. The previous retry window (~3.5s) was too short to outlast
this, causing "Database Error - Cannot Load Data" on every
logout/login cycle.

Increase IDB_OPEN_RETRIES from 3 to 5 and IDB_OPEN_RETRY_BASE_DELAY_MS
from 500ms to 1000ms, giving a total retry window of ~31 seconds.

Fixes #7191

* fix: update stale retry timing comments to match new constants

* fix(persistence): gate long IDB retry window on lock-related errors

_ensureInit() is awaited by every op-log read/write, and the hydrator
auto-reloads on IndexedDB open errors. Applying the ~31s retry window
to every error class could create a reload -> wait -> reload loop that
feels like a hang. Classify the error on first failure and fall back
to a short retry budget (IDB_OPEN_RETRIES_NON_LOCK=2, ~3s window) for
errors that don't look lock-related. InvalidStateError and messages
containing "backing store" still get the full window used for stale
LevelDB locks on Linux desktop session restart. See #7191.

The spec is hardened to assert concrete constant floors rather than
recomputing the backoff formula (which only proved the formula matched
itself). Also adds unit tests for the classification helper.

* fix(persistence): address review feedback on IDB retry split

- Correct stale "auto-reload" rationale in op-log-errors.const,
  operation-log.const, operation-log-store.service, and archive-store.service:
  the hydrator shows a blocking alert dialog on IndexedDBOpenError, it does
  not auto-reload. The fail-fast argument now reflects the real reason —
  every op-log read/write awaits _ensureInit(), so a 31s wait on a non-lock
  error blocks the subsystem for 31s before the alert dialog surfaces.
- isLockRelatedIdbOpenError now checks `err instanceof DOMException || err
  instanceof Error` before reading .name/.message, mirroring the pattern in
  the sibling isConnectionClosingError. In some Electron / older runtimes
  DOMException does not satisfy instanceof Error.
- Added behavioral tests for _openDbWithRetry that spy on a new
  _openDbOnce testing seam: generic errors take exactly
  1 + IDB_OPEN_RETRIES_NON_LOCK attempts, InvalidStateError takes up to
  1 + IDB_OPEN_RETRIES, and a lock-error followed by success resolves.
- Bumped IDB_OPEN_RETRIES_NON_LOCK from 2 to 3 to match master's pre-PR
  retry count (1s + 2s + 4s = ~7s ceiling). Framing this as "keep master's
  existing retry count for non-lock errors, add long retry only for lock
  errors" is a lower-risk scope. Adjusted the const spec's upper-bound
  assertion accordingly.
- Replaced the "retries 1-5" hardcoded schedule comment in
  operation-log-store.service and archive-store.service with a formula-based
  comment so the two retry budgets can't drift out of sync with reality.
- Added two DOMException-shaped test cases to
  operation-log.const.spec.ts covering the new predicate branch.

* docs(persistence): clean up rot-prone worked examples in retry docstrings
This commit is contained in:
John Costa 2026-04-19 10:20:23 -07:00 committed by GitHub
parent bbd4b41b12
commit 32dbc95ed9
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 333 additions and 16 deletions

View file

@ -0,0 +1,105 @@
import {
IDB_OPEN_RETRIES,
IDB_OPEN_RETRIES_NON_LOCK,
IDB_OPEN_RETRY_BASE_DELAY_MS,
} from './operation-log.const';
import { isLockRelatedIdbOpenError } from '../persistence/op-log-errors.const';
describe('IndexedDB open retry configuration', () => {
// Minimum 20s window exists as a defense-in-depth retry budget for
// session-restart LOCK contention on Linux desktop environments
// (especially Flatpak), where logout/login with autostart can leave
// the old session's LevelDB lock held for 5-15+ seconds.
// See: https://github.com/super-productivity/super-productivity/issues/7191
const MINIMUM_LOCK_RETRY_WINDOW_MS = 20_000;
it('lock-related retry window is at least 20 seconds', () => {
// Assert against concrete constant values rather than recomputing the
// exponential-backoff formula — otherwise the test just checks that the
// formula matches itself. We want to fail if someone changes
// IDB_OPEN_RETRIES or IDB_OPEN_RETRY_BASE_DELAY_MS below the floor.
expect(IDB_OPEN_RETRIES).toBeGreaterThanOrEqual(5);
expect(IDB_OPEN_RETRY_BASE_DELAY_MS).toBeGreaterThanOrEqual(1000);
// Sanity check: the total window implied by those values clears 20s.
// With IDB_OPEN_RETRIES=5 and base 1000ms: 1+2+4+8+16 = 31s.
const totalDelayMs =
(Math.pow(2, IDB_OPEN_RETRIES) - 1) * IDB_OPEN_RETRY_BASE_DELAY_MS;
expect(totalDelayMs).toBeGreaterThanOrEqual(MINIMUM_LOCK_RETRY_WINDOW_MS);
});
it('non-lock retry budget is shorter than the lock budget', () => {
// Non-lock errors must fail fast: every op-log read/write awaits
// _ensureInit(), so a 31s retry blocks the subsystem for 31s before the
// hydrator's alert dialog reaches the user. See #7191.
expect(IDB_OPEN_RETRIES_NON_LOCK).toBeLessThan(IDB_OPEN_RETRIES);
// Geometric series `(2^n - 1) * base` matches the actual wall-clock
// window: with n retries the delays are base, 2*base, ..., 2^(n-1)*base
// before attempts 2..n+1, with no post-delay on the final attempt.
// Cap well under 10s so non-lock failures surface quickly.
const nonLockWindowMs =
(Math.pow(2, IDB_OPEN_RETRIES_NON_LOCK) - 1) * IDB_OPEN_RETRY_BASE_DELAY_MS;
expect(nonLockWindowMs).toBeLessThan(10000);
});
});
describe('isLockRelatedIdbOpenError', () => {
it('returns true for InvalidStateError (Chrome LevelDB lock signal)', () => {
const err = new Error('Internal error.');
err.name = 'InvalidStateError';
expect(isLockRelatedIdbOpenError(err)).toBe(true);
});
it('returns true when the message mentions "backing store" (any case)', () => {
expect(
isLockRelatedIdbOpenError(new Error('Internal error opening backing store')),
).toBe(true);
expect(isLockRelatedIdbOpenError(new Error('BACKING STORE is locked'))).toBe(true);
expect(isLockRelatedIdbOpenError('backing store failure')).toBe(true);
});
it('returns true for DOMException with name InvalidStateError', () => {
// Some Electron / older runtimes don't make DOMException satisfy
// `instanceof Error`, so the predicate must check DOMException too. This
// test constructs a real DOMException when the runtime supports the
// two-arg constructor, and falls back to a duck-typed stand-in otherwise.
let err: unknown;
try {
err = new DOMException('Internal error.', 'InvalidStateError');
} catch {
// Fallback for runtimes without the DOMException constructor:
// mimic the shape and prototype chain that isConnectionClosingError
// already relies on in the same file.
err = Object.assign(Object.create(DOMException.prototype), {
name: 'InvalidStateError',
message: 'Internal error.',
});
}
expect(isLockRelatedIdbOpenError(err)).toBe(true);
});
it('returns true for DOMException whose message mentions "backing store"', () => {
let err: unknown;
try {
err = new DOMException('Internal error opening backing store', 'UnknownError');
} catch {
err = Object.assign(Object.create(DOMException.prototype), {
name: 'UnknownError',
message: 'Internal error opening backing store',
});
}
expect(isLockRelatedIdbOpenError(err)).toBe(true);
});
it('returns false for generic errors that do not look lock-related', () => {
expect(isLockRelatedIdbOpenError(new Error('AbortError'))).toBe(false);
expect(
isLockRelatedIdbOpenError(
Object.assign(new Error('Quota exceeded'), { name: 'QuotaExceededError' }),
),
).toBe(false);
expect(isLockRelatedIdbOpenError(undefined)).toBe(false);
expect(isLockRelatedIdbOpenError(null)).toBe(false);
expect(isLockRelatedIdbOpenError({ random: 'object' })).toBe(false);
});
});

View file

@ -268,12 +268,38 @@ export const POST_SYNC_COOLDOWN_MS = 2000;
* Number of retry attempts when opening IndexedDB fails.
* Total attempts = 1 initial + IDB_OPEN_RETRIES retries.
* Transient failures (file locks, temporary I/O issues) may resolve on retry.
* On Linux desktop environments, session logout/login with autostart can leave
* stale LevelDB locks for 5-15+ seconds, so the total retry window must be long
* enough to outlast this.
* @see https://github.com/johannesjo/super-productivity/issues/6255
* @see https://github.com/super-productivity/super-productivity/issues/7191
*/
export const IDB_OPEN_RETRIES = 3;
export const IDB_OPEN_RETRIES = 5;
/**
* Number of retry attempts for IndexedDB open errors that do NOT look like
* session-restart file locks (e.g. generic `AbortError`, `UnknownError`).
*
* Matches master's pre-PR retry count (3) so the non-lock path keeps the
* existing behavior; the new long window is only applied when the error
* looks lock-related.
*
* With 3 retries at a 1000ms base, the non-lock wall-clock ceiling is
* 1s + 2s + 4s = 7s (delays before attempts 2, 3, 4; no delay after the
* final attempt). That matters because every op-log read/write awaits
* `_ensureInit()`, so a 31s retry window on a non-lock error would block
* the entire op-log subsystem for 31s before the hydrator's alert dialog
* (`OperationLogHydratorService._showIndexedDBOpenError`) reaches the user.
* For non-lock errors there is no expectation that waiting helps, so fail
* fast and let the error surface sooner.
*
* @see https://github.com/super-productivity/super-productivity/issues/7191
*/
export const IDB_OPEN_RETRIES_NON_LOCK = 3;
/**
* Base delay for IndexedDB open retry exponential backoff (milliseconds).
* With IDB_OPEN_RETRIES=3: delays are 500ms, 1000ms, 2000ms for retries 1, 2, 3.
* Delays follow `BASE * 2^(attempt-1)`; see `IDB_OPEN_RETRIES` and
* `IDB_OPEN_RETRIES_NON_LOCK` for the resulting windows.
*/
export const IDB_OPEN_RETRY_BASE_DELAY_MS = 500;
export const IDB_OPEN_RETRY_BASE_DELAY_MS = 1000;

View file

@ -12,10 +12,12 @@ import { runDbUpgrade } from './db-upgrade';
import {
ARCHIVE_STORE_NOT_INITIALIZED,
isConnectionClosingError,
isLockRelatedIdbOpenError,
} from './op-log-errors.const';
import { Log } from '../../core/log';
import {
IDB_OPEN_RETRIES,
IDB_OPEN_RETRIES_NON_LOCK,
IDB_OPEN_RETRY_BASE_DELAY_MS,
} from '../core/operation-log.const';
import { IndexedDBOpenError } from '../core/errors/indexed-db-open.error';
@ -84,11 +86,18 @@ export class ArchiveStoreService {
this._db = db;
}
/**
* See OperationLogStoreService._openDbWithRetry for the rationale behind the
* lock-vs-non-lock retry budget split.
*
* @see https://github.com/super-productivity/super-productivity/issues/7191
*/
private async _openDbWithRetry(): Promise<IDBPDatabase<ArchiveDBSchema>> {
const totalAttempts = 1 + IDB_OPEN_RETRIES;
let maxRetries = IDB_OPEN_RETRIES;
let attempt = 1;
let lastError: unknown;
for (let attempt = 1; attempt <= totalAttempts; attempt++) {
while (attempt <= 1 + maxRetries) {
try {
return await openDB<ArchiveDBSchema>(DB_NAME, DB_VERSION, {
upgrade: (db, oldVersion, _newVersion, transaction) => {
@ -98,7 +107,18 @@ export class ArchiveStoreService {
} catch (e) {
lastError = e;
// Non-lock errors fall back to a short retry budget so we don't block
// the op-log subsystem for 31s before surfacing the error to the user.
// See OperationLogStoreService._openDbWithRetry for details.
if (attempt === 1 && !isLockRelatedIdbOpenError(e)) {
maxRetries = IDB_OPEN_RETRIES_NON_LOCK;
}
const totalAttempts = 1 + maxRetries;
if (attempt < totalAttempts) {
// Exponential backoff: BASE * 2^(attempt-1). Lock errors retry up to
// IDB_OPEN_RETRIES times (~31s total); non-lock errors truncate at
// IDB_OPEN_RETRIES_NON_LOCK (~7s total).
const delay = IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, attempt - 1);
Log.warn(
`[ArchiveStore] IndexedDB open failed (attempt ${attempt}/${totalAttempts}), retrying in ${delay}ms...`,
@ -106,6 +126,8 @@ export class ArchiveStoreService {
);
await new Promise((resolve) => setTimeout(resolve, delay));
}
attempt++;
}
}

View file

@ -54,6 +54,43 @@ export const IDB_OPEN_ERROR_MSG =
*/
export const IDB_BACKING_STORE_PATTERN = 'backing store';
/**
* Heuristic: is this IndexedDB open error likely caused by a file lock held
* by a previous process/session?
*
* Used to gate the long (~31s) retry window on errors where waiting might
* actually help. Non-lock errors should fail fast: every op-log read/write
* awaits `_ensureInit()`, so a 31s retry on a non-lock error blocks the
* entire op-log subsystem for 31s before the hydrator's alert dialog
* (see `OperationLogHydratorService._showIndexedDBOpenError`) can reach
* the user.
*
* Signals that suggest a lock:
* - `InvalidStateError`: Chrome throws this when LevelDB's LOCK file is held.
* - "backing store" in the message: backing-store errors on Linux / Electron
* are commonly caused by a stale LevelDB lock from a crashed or prior
* session (see issue #7191).
*
* `DOMException` is checked explicitly in addition to `Error` because in some
* Electron / older runtimes a `DOMException` does not satisfy
* `instanceof Error`, and IndexedDB failures surface as `DOMException`s.
* Mirrors the pattern in `isConnectionClosingError` below.
*
* @see https://github.com/super-productivity/super-productivity/issues/7191
*/
export const isLockRelatedIdbOpenError = (err: unknown): boolean => {
if (err instanceof DOMException || err instanceof Error) {
if (err.name === 'InvalidStateError') {
return true;
}
return err.message.toLowerCase().includes(IDB_BACKING_STORE_PATTERN);
}
if (typeof err === 'string') {
return err.toLowerCase().includes(IDB_BACKING_STORE_PATTERN);
}
return false;
};
// ============================================================================
// Connection Closing Errors (Issue #6643)
// ============================================================================

View file

@ -1,4 +1,4 @@
import { TestBed } from '@angular/core/testing';
import { fakeAsync, TestBed, tick } from '@angular/core/testing';
import { OperationLogStoreService } from './operation-log-store.service';
import { VectorClockService } from '../sync/vector-clock.service';
import {
@ -16,6 +16,12 @@ import {
} from '../../core/util/vector-clock';
import { limitVectorClockSize, MAX_VECTOR_CLOCK_SIZE } from '@sp/shared-schema';
import { CLIENT_ID_PROVIDER, ClientIdProvider } from '../util/client-id.provider';
import {
IDB_OPEN_RETRIES,
IDB_OPEN_RETRIES_NON_LOCK,
IDB_OPEN_RETRY_BASE_DELAY_MS,
} from '../core/operation-log.const';
import { IndexedDBOpenError } from '../core/errors/indexed-db-open.error';
describe('OperationLogStoreService', () => {
let service: OperationLogStoreService;
@ -2690,4 +2696,89 @@ describe('OperationLogStoreService', () => {
expect(unsynced.length).toBe(0);
});
});
describe('_openDbWithRetry classification and budget', () => {
// Uses a fresh, un-initialized service and spies on `_openDbOnce` (the
// testing seam around the real `openDB` call) to observe classification
// and retry-budget behavior without mocking the `idb` module.
let retryService: OperationLogStoreService;
let openSpy: jasmine.Spy;
const makeFakeDb = (): any => ({
addEventListener: () => {},
});
beforeEach(() => {
TestBed.resetTestingModule();
TestBed.configureTestingModule({
providers: [
OperationLogStoreService,
{ provide: CLIENT_ID_PROVIDER, useValue: mockClientIdProvider },
],
});
retryService = TestBed.inject(OperationLogStoreService);
openSpy = spyOn<any>(retryService, '_openDbOnce');
});
it('makes 1 + IDB_OPEN_RETRIES_NON_LOCK attempts and throws on a generic error', fakeAsync(() => {
openSpy.and.returnValue(Promise.reject(new Error('generic')));
let caught: unknown;
retryService.init().catch((e) => {
caught = e;
});
// Advance past each backoff window. With base=1000ms and 3 non-lock
// retries, delays are 1s, 2s, 4s before attempts 2, 3, 4.
for (let i = 1; i <= IDB_OPEN_RETRIES_NON_LOCK; i++) {
tick(IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, i - 1));
}
// Final tick to resolve the rejection
tick();
expect(openSpy).toHaveBeenCalledTimes(1 + IDB_OPEN_RETRIES_NON_LOCK);
expect(caught).toBeInstanceOf(IndexedDBOpenError);
}));
it('makes up to 1 + IDB_OPEN_RETRIES attempts on a lock-related InvalidStateError', fakeAsync(() => {
const lockErr = new Error('Internal error.');
lockErr.name = 'InvalidStateError';
openSpy.and.returnValue(Promise.reject(lockErr));
let caught: unknown;
retryService.init().catch((e) => {
caught = e;
});
// Drain all backoff windows for the full lock budget.
for (let i = 1; i <= IDB_OPEN_RETRIES; i++) {
tick(IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, i - 1));
}
tick();
expect(openSpy).toHaveBeenCalledTimes(1 + IDB_OPEN_RETRIES);
// Sanity: lock budget strictly exceeds the non-lock budget.
expect(openSpy.calls.count()).toBeGreaterThan(1 + IDB_OPEN_RETRIES_NON_LOCK);
expect(caught).toBeInstanceOf(IndexedDBOpenError);
}));
it('returns the database when a single lock-related failure is followed by a success', fakeAsync(() => {
const lockErr = new Error('Internal error opening backing store');
const fakeDb = makeFakeDb();
openSpy.and.returnValues(Promise.reject(lockErr), Promise.resolve(fakeDb));
let resolved = false;
retryService.init().then(() => {
resolved = true;
});
// First attempt rejects immediately; backoff before attempt 2 is 1s.
tick(IDB_OPEN_RETRY_BASE_DELAY_MS);
tick();
expect(openSpy).toHaveBeenCalledTimes(2);
expect(resolved).toBe(true);
expect((retryService as any)._db).toBe(fakeDb);
}));
});
});

View file

@ -28,11 +28,13 @@ import {
import {
DUPLICATE_OPERATION_ERROR_MSG,
OPERATION_LOG_STORE_NOT_INITIALIZED,
isLockRelatedIdbOpenError,
} from './op-log-errors.const';
import { runDbUpgrade } from './db-upgrade';
import { Log } from '../../core/log';
import {
IDB_OPEN_RETRIES,
IDB_OPEN_RETRIES_NON_LOCK,
IDB_OPEN_RETRY_BASE_DELAY_MS,
} from '../core/operation-log.const';
import { IndexedDBOpenError } from '../core/errors/indexed-db-open.error';
@ -196,32 +198,64 @@ export class OperationLogStoreService {
this._db = db;
}
/**
* Wraps a single `openDB` call. Exists as a testing seam so specs can
* `spyOn(service as any, '_openDbOnce')` to inject failures without mocking
* the `idb` module import. Not intended to be called directly outside the
* retry loop.
*/
private _openDbOnce(): Promise<IDBPDatabase<OpLogDB>> {
return openDB<OpLogDB>(DB_NAME, DB_VERSION, {
upgrade: (db, oldVersion, _newVersion, transaction) => {
runDbUpgrade(db, oldVersion, transaction);
},
});
}
/**
* Opens IndexedDB with retry logic and exponential backoff.
* Transient failures (file locks, temporary I/O issues) may resolve on retry.
*
* Total attempts = 1 initial + IDB_OPEN_RETRIES retries.
* With IDB_OPEN_RETRIES=3: attempts at 0ms, 500ms, 1500ms, 3500ms (total ~3.5s worst case).
* The retry budget depends on the error:
* - Lock-related errors (InvalidStateError, "backing store"): use the full
* IDB_OPEN_RETRIES window (~31s) to outlast stale LevelDB locks from a
* previous session. See issue #7191.
* - Other errors: fall back to IDB_OPEN_RETRIES_NON_LOCK (~7s). Every op-log
* read/write awaits `_ensureInit()`, so a 31s retry window on a non-lock
* error blocks the entire op-log subsystem for 31s before the hydrator's
* alert dialog reaches the user. There's no expectation that waiting
* helps for non-lock errors, so fail fast.
*
* @throws IndexedDBOpenError if all retry attempts fail
* @see https://github.com/johannesjo/super-productivity/issues/6255
* @see https://github.com/super-productivity/super-productivity/issues/7191
*/
private async _openDbWithRetry(): Promise<IDBPDatabase<OpLogDB>> {
const totalAttempts = 1 + IDB_OPEN_RETRIES;
let maxRetries = IDB_OPEN_RETRIES;
let attempt = 1;
let lastError: unknown;
for (let attempt = 1; attempt <= totalAttempts; attempt++) {
// Loop until either openDB succeeds or we exhaust the retry budget for the
// observed error class. `maxRetries` may shrink after the first failure if
// the error doesn't look lock-related.
while (attempt <= 1 + maxRetries) {
try {
return await openDB<OpLogDB>(DB_NAME, DB_VERSION, {
upgrade: (db, oldVersion, _newVersion, transaction) => {
runDbUpgrade(db, oldVersion, transaction);
},
});
return await this._openDbOnce();
} catch (e) {
lastError = e;
// Classify the error on the first failure. If it doesn't look
// lock-related, shrink the retry budget so we fail fast and let the
// hydrator surface the error instead of hanging for the full window.
if (attempt === 1 && !isLockRelatedIdbOpenError(e)) {
maxRetries = IDB_OPEN_RETRIES_NON_LOCK;
}
const totalAttempts = 1 + maxRetries;
if (attempt < totalAttempts) {
// Exponential backoff: 500ms, 1000ms, 2000ms for retries 1, 2, 3
// Exponential backoff: BASE * 2^(attempt-1). Lock errors retry up to
// IDB_OPEN_RETRIES times (~31s total); non-lock errors truncate at
// IDB_OPEN_RETRIES_NON_LOCK (~7s total).
const delay = IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, attempt - 1);
Log.warn(
`[OpLogStore] IndexedDB open failed (attempt ${attempt}/${totalAttempts}), retrying in ${delay}ms...`,
@ -229,6 +263,8 @@ export class OperationLogStoreService {
);
await new Promise((resolve) => setTimeout(resolve, delay));
}
attempt++;
}
}