mirror of
https://github.com/johannesjo/super-productivity.git
synced 2026-07-18 00:46:45 +00:00
fix(persistence): increase IndexedDB retry window for session-restart locks (#7220)
* fix(persistence): increase IndexedDB retry window for session-restart locks On Linux desktop environments (especially Flatpak with autostart), logging out and back in can leave stale LevelDB locks for 5-15+ seconds. The previous retry window (~3.5s) was too short to outlast this, causing "Database Error - Cannot Load Data" on every logout/login cycle. Increase IDB_OPEN_RETRIES from 3 to 5 and IDB_OPEN_RETRY_BASE_DELAY_MS from 500ms to 1000ms, giving a total retry window of ~31 seconds. Fixes #7191 * fix: update stale retry timing comments to match new constants * fix(persistence): gate long IDB retry window on lock-related errors _ensureInit() is awaited by every op-log read/write, and the hydrator auto-reloads on IndexedDB open errors. Applying the ~31s retry window to every error class could create a reload -> wait -> reload loop that feels like a hang. Classify the error on first failure and fall back to a short retry budget (IDB_OPEN_RETRIES_NON_LOCK=2, ~3s window) for errors that don't look lock-related. InvalidStateError and messages containing "backing store" still get the full window used for stale LevelDB locks on Linux desktop session restart. See #7191. The spec is hardened to assert concrete constant floors rather than recomputing the backoff formula (which only proved the formula matched itself). Also adds unit tests for the classification helper. * fix(persistence): address review feedback on IDB retry split - Correct stale "auto-reload" rationale in op-log-errors.const, operation-log.const, operation-log-store.service, and archive-store.service: the hydrator shows a blocking alert dialog on IndexedDBOpenError, it does not auto-reload. The fail-fast argument now reflects the real reason — every op-log read/write awaits _ensureInit(), so a 31s wait on a non-lock error blocks the subsystem for 31s before the alert dialog surfaces. - isLockRelatedIdbOpenError now checks `err instanceof DOMException || err instanceof Error` before reading .name/.message, mirroring the pattern in the sibling isConnectionClosingError. In some Electron / older runtimes DOMException does not satisfy instanceof Error. - Added behavioral tests for _openDbWithRetry that spy on a new _openDbOnce testing seam: generic errors take exactly 1 + IDB_OPEN_RETRIES_NON_LOCK attempts, InvalidStateError takes up to 1 + IDB_OPEN_RETRIES, and a lock-error followed by success resolves. - Bumped IDB_OPEN_RETRIES_NON_LOCK from 2 to 3 to match master's pre-PR retry count (1s + 2s + 4s = ~7s ceiling). Framing this as "keep master's existing retry count for non-lock errors, add long retry only for lock errors" is a lower-risk scope. Adjusted the const spec's upper-bound assertion accordingly. - Replaced the "retries 1-5" hardcoded schedule comment in operation-log-store.service and archive-store.service with a formula-based comment so the two retry budgets can't drift out of sync with reality. - Added two DOMException-shaped test cases to operation-log.const.spec.ts covering the new predicate branch. * docs(persistence): clean up rot-prone worked examples in retry docstrings
This commit is contained in:
parent
bbd4b41b12
commit
32dbc95ed9
6 changed files with 333 additions and 16 deletions
105
src/app/op-log/core/operation-log.const.spec.ts
Normal file
105
src/app/op-log/core/operation-log.const.spec.ts
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
import {
|
||||
IDB_OPEN_RETRIES,
|
||||
IDB_OPEN_RETRIES_NON_LOCK,
|
||||
IDB_OPEN_RETRY_BASE_DELAY_MS,
|
||||
} from './operation-log.const';
|
||||
import { isLockRelatedIdbOpenError } from '../persistence/op-log-errors.const';
|
||||
|
||||
describe('IndexedDB open retry configuration', () => {
|
||||
// Minimum 20s window exists as a defense-in-depth retry budget for
|
||||
// session-restart LOCK contention on Linux desktop environments
|
||||
// (especially Flatpak), where logout/login with autostart can leave
|
||||
// the old session's LevelDB lock held for 5-15+ seconds.
|
||||
// See: https://github.com/super-productivity/super-productivity/issues/7191
|
||||
const MINIMUM_LOCK_RETRY_WINDOW_MS = 20_000;
|
||||
|
||||
it('lock-related retry window is at least 20 seconds', () => {
|
||||
// Assert against concrete constant values rather than recomputing the
|
||||
// exponential-backoff formula — otherwise the test just checks that the
|
||||
// formula matches itself. We want to fail if someone changes
|
||||
// IDB_OPEN_RETRIES or IDB_OPEN_RETRY_BASE_DELAY_MS below the floor.
|
||||
expect(IDB_OPEN_RETRIES).toBeGreaterThanOrEqual(5);
|
||||
expect(IDB_OPEN_RETRY_BASE_DELAY_MS).toBeGreaterThanOrEqual(1000);
|
||||
|
||||
// Sanity check: the total window implied by those values clears 20s.
|
||||
// With IDB_OPEN_RETRIES=5 and base 1000ms: 1+2+4+8+16 = 31s.
|
||||
const totalDelayMs =
|
||||
(Math.pow(2, IDB_OPEN_RETRIES) - 1) * IDB_OPEN_RETRY_BASE_DELAY_MS;
|
||||
expect(totalDelayMs).toBeGreaterThanOrEqual(MINIMUM_LOCK_RETRY_WINDOW_MS);
|
||||
});
|
||||
|
||||
it('non-lock retry budget is shorter than the lock budget', () => {
|
||||
// Non-lock errors must fail fast: every op-log read/write awaits
|
||||
// _ensureInit(), so a 31s retry blocks the subsystem for 31s before the
|
||||
// hydrator's alert dialog reaches the user. See #7191.
|
||||
expect(IDB_OPEN_RETRIES_NON_LOCK).toBeLessThan(IDB_OPEN_RETRIES);
|
||||
// Geometric series `(2^n - 1) * base` matches the actual wall-clock
|
||||
// window: with n retries the delays are base, 2*base, ..., 2^(n-1)*base
|
||||
// before attempts 2..n+1, with no post-delay on the final attempt.
|
||||
// Cap well under 10s so non-lock failures surface quickly.
|
||||
const nonLockWindowMs =
|
||||
(Math.pow(2, IDB_OPEN_RETRIES_NON_LOCK) - 1) * IDB_OPEN_RETRY_BASE_DELAY_MS;
|
||||
expect(nonLockWindowMs).toBeLessThan(10000);
|
||||
});
|
||||
});
|
||||
|
||||
describe('isLockRelatedIdbOpenError', () => {
|
||||
it('returns true for InvalidStateError (Chrome LevelDB lock signal)', () => {
|
||||
const err = new Error('Internal error.');
|
||||
err.name = 'InvalidStateError';
|
||||
expect(isLockRelatedIdbOpenError(err)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true when the message mentions "backing store" (any case)', () => {
|
||||
expect(
|
||||
isLockRelatedIdbOpenError(new Error('Internal error opening backing store')),
|
||||
).toBe(true);
|
||||
expect(isLockRelatedIdbOpenError(new Error('BACKING STORE is locked'))).toBe(true);
|
||||
expect(isLockRelatedIdbOpenError('backing store failure')).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true for DOMException with name InvalidStateError', () => {
|
||||
// Some Electron / older runtimes don't make DOMException satisfy
|
||||
// `instanceof Error`, so the predicate must check DOMException too. This
|
||||
// test constructs a real DOMException when the runtime supports the
|
||||
// two-arg constructor, and falls back to a duck-typed stand-in otherwise.
|
||||
let err: unknown;
|
||||
try {
|
||||
err = new DOMException('Internal error.', 'InvalidStateError');
|
||||
} catch {
|
||||
// Fallback for runtimes without the DOMException constructor:
|
||||
// mimic the shape and prototype chain that isConnectionClosingError
|
||||
// already relies on in the same file.
|
||||
err = Object.assign(Object.create(DOMException.prototype), {
|
||||
name: 'InvalidStateError',
|
||||
message: 'Internal error.',
|
||||
});
|
||||
}
|
||||
expect(isLockRelatedIdbOpenError(err)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns true for DOMException whose message mentions "backing store"', () => {
|
||||
let err: unknown;
|
||||
try {
|
||||
err = new DOMException('Internal error opening backing store', 'UnknownError');
|
||||
} catch {
|
||||
err = Object.assign(Object.create(DOMException.prototype), {
|
||||
name: 'UnknownError',
|
||||
message: 'Internal error opening backing store',
|
||||
});
|
||||
}
|
||||
expect(isLockRelatedIdbOpenError(err)).toBe(true);
|
||||
});
|
||||
|
||||
it('returns false for generic errors that do not look lock-related', () => {
|
||||
expect(isLockRelatedIdbOpenError(new Error('AbortError'))).toBe(false);
|
||||
expect(
|
||||
isLockRelatedIdbOpenError(
|
||||
Object.assign(new Error('Quota exceeded'), { name: 'QuotaExceededError' }),
|
||||
),
|
||||
).toBe(false);
|
||||
expect(isLockRelatedIdbOpenError(undefined)).toBe(false);
|
||||
expect(isLockRelatedIdbOpenError(null)).toBe(false);
|
||||
expect(isLockRelatedIdbOpenError({ random: 'object' })).toBe(false);
|
||||
});
|
||||
});
|
||||
|
|
@ -268,12 +268,38 @@ export const POST_SYNC_COOLDOWN_MS = 2000;
|
|||
* Number of retry attempts when opening IndexedDB fails.
|
||||
* Total attempts = 1 initial + IDB_OPEN_RETRIES retries.
|
||||
* Transient failures (file locks, temporary I/O issues) may resolve on retry.
|
||||
* On Linux desktop environments, session logout/login with autostart can leave
|
||||
* stale LevelDB locks for 5-15+ seconds, so the total retry window must be long
|
||||
* enough to outlast this.
|
||||
* @see https://github.com/johannesjo/super-productivity/issues/6255
|
||||
* @see https://github.com/super-productivity/super-productivity/issues/7191
|
||||
*/
|
||||
export const IDB_OPEN_RETRIES = 3;
|
||||
export const IDB_OPEN_RETRIES = 5;
|
||||
|
||||
/**
|
||||
* Number of retry attempts for IndexedDB open errors that do NOT look like
|
||||
* session-restart file locks (e.g. generic `AbortError`, `UnknownError`).
|
||||
*
|
||||
* Matches master's pre-PR retry count (3) so the non-lock path keeps the
|
||||
* existing behavior; the new long window is only applied when the error
|
||||
* looks lock-related.
|
||||
*
|
||||
* With 3 retries at a 1000ms base, the non-lock wall-clock ceiling is
|
||||
* 1s + 2s + 4s = 7s (delays before attempts 2, 3, 4; no delay after the
|
||||
* final attempt). That matters because every op-log read/write awaits
|
||||
* `_ensureInit()`, so a 31s retry window on a non-lock error would block
|
||||
* the entire op-log subsystem for 31s before the hydrator's alert dialog
|
||||
* (`OperationLogHydratorService._showIndexedDBOpenError`) reaches the user.
|
||||
* For non-lock errors there is no expectation that waiting helps, so fail
|
||||
* fast and let the error surface sooner.
|
||||
*
|
||||
* @see https://github.com/super-productivity/super-productivity/issues/7191
|
||||
*/
|
||||
export const IDB_OPEN_RETRIES_NON_LOCK = 3;
|
||||
|
||||
/**
|
||||
* Base delay for IndexedDB open retry exponential backoff (milliseconds).
|
||||
* With IDB_OPEN_RETRIES=3: delays are 500ms, 1000ms, 2000ms for retries 1, 2, 3.
|
||||
* Delays follow `BASE * 2^(attempt-1)`; see `IDB_OPEN_RETRIES` and
|
||||
* `IDB_OPEN_RETRIES_NON_LOCK` for the resulting windows.
|
||||
*/
|
||||
export const IDB_OPEN_RETRY_BASE_DELAY_MS = 500;
|
||||
export const IDB_OPEN_RETRY_BASE_DELAY_MS = 1000;
|
||||
|
|
|
|||
|
|
@ -12,10 +12,12 @@ import { runDbUpgrade } from './db-upgrade';
|
|||
import {
|
||||
ARCHIVE_STORE_NOT_INITIALIZED,
|
||||
isConnectionClosingError,
|
||||
isLockRelatedIdbOpenError,
|
||||
} from './op-log-errors.const';
|
||||
import { Log } from '../../core/log';
|
||||
import {
|
||||
IDB_OPEN_RETRIES,
|
||||
IDB_OPEN_RETRIES_NON_LOCK,
|
||||
IDB_OPEN_RETRY_BASE_DELAY_MS,
|
||||
} from '../core/operation-log.const';
|
||||
import { IndexedDBOpenError } from '../core/errors/indexed-db-open.error';
|
||||
|
|
@ -84,11 +86,18 @@ export class ArchiveStoreService {
|
|||
this._db = db;
|
||||
}
|
||||
|
||||
/**
|
||||
* See OperationLogStoreService._openDbWithRetry for the rationale behind the
|
||||
* lock-vs-non-lock retry budget split.
|
||||
*
|
||||
* @see https://github.com/super-productivity/super-productivity/issues/7191
|
||||
*/
|
||||
private async _openDbWithRetry(): Promise<IDBPDatabase<ArchiveDBSchema>> {
|
||||
const totalAttempts = 1 + IDB_OPEN_RETRIES;
|
||||
let maxRetries = IDB_OPEN_RETRIES;
|
||||
let attempt = 1;
|
||||
let lastError: unknown;
|
||||
|
||||
for (let attempt = 1; attempt <= totalAttempts; attempt++) {
|
||||
while (attempt <= 1 + maxRetries) {
|
||||
try {
|
||||
return await openDB<ArchiveDBSchema>(DB_NAME, DB_VERSION, {
|
||||
upgrade: (db, oldVersion, _newVersion, transaction) => {
|
||||
|
|
@ -98,7 +107,18 @@ export class ArchiveStoreService {
|
|||
} catch (e) {
|
||||
lastError = e;
|
||||
|
||||
// Non-lock errors fall back to a short retry budget so we don't block
|
||||
// the op-log subsystem for 31s before surfacing the error to the user.
|
||||
// See OperationLogStoreService._openDbWithRetry for details.
|
||||
if (attempt === 1 && !isLockRelatedIdbOpenError(e)) {
|
||||
maxRetries = IDB_OPEN_RETRIES_NON_LOCK;
|
||||
}
|
||||
|
||||
const totalAttempts = 1 + maxRetries;
|
||||
if (attempt < totalAttempts) {
|
||||
// Exponential backoff: BASE * 2^(attempt-1). Lock errors retry up to
|
||||
// IDB_OPEN_RETRIES times (~31s total); non-lock errors truncate at
|
||||
// IDB_OPEN_RETRIES_NON_LOCK (~7s total).
|
||||
const delay = IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, attempt - 1);
|
||||
Log.warn(
|
||||
`[ArchiveStore] IndexedDB open failed (attempt ${attempt}/${totalAttempts}), retrying in ${delay}ms...`,
|
||||
|
|
@ -106,6 +126,8 @@ export class ArchiveStoreService {
|
|||
);
|
||||
await new Promise((resolve) => setTimeout(resolve, delay));
|
||||
}
|
||||
|
||||
attempt++;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -54,6 +54,43 @@ export const IDB_OPEN_ERROR_MSG =
|
|||
*/
|
||||
export const IDB_BACKING_STORE_PATTERN = 'backing store';
|
||||
|
||||
/**
|
||||
* Heuristic: is this IndexedDB open error likely caused by a file lock held
|
||||
* by a previous process/session?
|
||||
*
|
||||
* Used to gate the long (~31s) retry window on errors where waiting might
|
||||
* actually help. Non-lock errors should fail fast: every op-log read/write
|
||||
* awaits `_ensureInit()`, so a 31s retry on a non-lock error blocks the
|
||||
* entire op-log subsystem for 31s before the hydrator's alert dialog
|
||||
* (see `OperationLogHydratorService._showIndexedDBOpenError`) can reach
|
||||
* the user.
|
||||
*
|
||||
* Signals that suggest a lock:
|
||||
* - `InvalidStateError`: Chrome throws this when LevelDB's LOCK file is held.
|
||||
* - "backing store" in the message: backing-store errors on Linux / Electron
|
||||
* are commonly caused by a stale LevelDB lock from a crashed or prior
|
||||
* session (see issue #7191).
|
||||
*
|
||||
* `DOMException` is checked explicitly in addition to `Error` because in some
|
||||
* Electron / older runtimes a `DOMException` does not satisfy
|
||||
* `instanceof Error`, and IndexedDB failures surface as `DOMException`s.
|
||||
* Mirrors the pattern in `isConnectionClosingError` below.
|
||||
*
|
||||
* @see https://github.com/super-productivity/super-productivity/issues/7191
|
||||
*/
|
||||
export const isLockRelatedIdbOpenError = (err: unknown): boolean => {
|
||||
if (err instanceof DOMException || err instanceof Error) {
|
||||
if (err.name === 'InvalidStateError') {
|
||||
return true;
|
||||
}
|
||||
return err.message.toLowerCase().includes(IDB_BACKING_STORE_PATTERN);
|
||||
}
|
||||
if (typeof err === 'string') {
|
||||
return err.toLowerCase().includes(IDB_BACKING_STORE_PATTERN);
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
// ============================================================================
|
||||
// Connection Closing Errors (Issue #6643)
|
||||
// ============================================================================
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
import { TestBed } from '@angular/core/testing';
|
||||
import { fakeAsync, TestBed, tick } from '@angular/core/testing';
|
||||
import { OperationLogStoreService } from './operation-log-store.service';
|
||||
import { VectorClockService } from '../sync/vector-clock.service';
|
||||
import {
|
||||
|
|
@ -16,6 +16,12 @@ import {
|
|||
} from '../../core/util/vector-clock';
|
||||
import { limitVectorClockSize, MAX_VECTOR_CLOCK_SIZE } from '@sp/shared-schema';
|
||||
import { CLIENT_ID_PROVIDER, ClientIdProvider } from '../util/client-id.provider';
|
||||
import {
|
||||
IDB_OPEN_RETRIES,
|
||||
IDB_OPEN_RETRIES_NON_LOCK,
|
||||
IDB_OPEN_RETRY_BASE_DELAY_MS,
|
||||
} from '../core/operation-log.const';
|
||||
import { IndexedDBOpenError } from '../core/errors/indexed-db-open.error';
|
||||
|
||||
describe('OperationLogStoreService', () => {
|
||||
let service: OperationLogStoreService;
|
||||
|
|
@ -2690,4 +2696,89 @@ describe('OperationLogStoreService', () => {
|
|||
expect(unsynced.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('_openDbWithRetry classification and budget', () => {
|
||||
// Uses a fresh, un-initialized service and spies on `_openDbOnce` (the
|
||||
// testing seam around the real `openDB` call) to observe classification
|
||||
// and retry-budget behavior without mocking the `idb` module.
|
||||
let retryService: OperationLogStoreService;
|
||||
let openSpy: jasmine.Spy;
|
||||
|
||||
const makeFakeDb = (): any => ({
|
||||
addEventListener: () => {},
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
TestBed.resetTestingModule();
|
||||
TestBed.configureTestingModule({
|
||||
providers: [
|
||||
OperationLogStoreService,
|
||||
{ provide: CLIENT_ID_PROVIDER, useValue: mockClientIdProvider },
|
||||
],
|
||||
});
|
||||
retryService = TestBed.inject(OperationLogStoreService);
|
||||
openSpy = spyOn<any>(retryService, '_openDbOnce');
|
||||
});
|
||||
|
||||
it('makes 1 + IDB_OPEN_RETRIES_NON_LOCK attempts and throws on a generic error', fakeAsync(() => {
|
||||
openSpy.and.returnValue(Promise.reject(new Error('generic')));
|
||||
|
||||
let caught: unknown;
|
||||
retryService.init().catch((e) => {
|
||||
caught = e;
|
||||
});
|
||||
|
||||
// Advance past each backoff window. With base=1000ms and 3 non-lock
|
||||
// retries, delays are 1s, 2s, 4s before attempts 2, 3, 4.
|
||||
for (let i = 1; i <= IDB_OPEN_RETRIES_NON_LOCK; i++) {
|
||||
tick(IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, i - 1));
|
||||
}
|
||||
// Final tick to resolve the rejection
|
||||
tick();
|
||||
|
||||
expect(openSpy).toHaveBeenCalledTimes(1 + IDB_OPEN_RETRIES_NON_LOCK);
|
||||
expect(caught).toBeInstanceOf(IndexedDBOpenError);
|
||||
}));
|
||||
|
||||
it('makes up to 1 + IDB_OPEN_RETRIES attempts on a lock-related InvalidStateError', fakeAsync(() => {
|
||||
const lockErr = new Error('Internal error.');
|
||||
lockErr.name = 'InvalidStateError';
|
||||
openSpy.and.returnValue(Promise.reject(lockErr));
|
||||
|
||||
let caught: unknown;
|
||||
retryService.init().catch((e) => {
|
||||
caught = e;
|
||||
});
|
||||
|
||||
// Drain all backoff windows for the full lock budget.
|
||||
for (let i = 1; i <= IDB_OPEN_RETRIES; i++) {
|
||||
tick(IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, i - 1));
|
||||
}
|
||||
tick();
|
||||
|
||||
expect(openSpy).toHaveBeenCalledTimes(1 + IDB_OPEN_RETRIES);
|
||||
// Sanity: lock budget strictly exceeds the non-lock budget.
|
||||
expect(openSpy.calls.count()).toBeGreaterThan(1 + IDB_OPEN_RETRIES_NON_LOCK);
|
||||
expect(caught).toBeInstanceOf(IndexedDBOpenError);
|
||||
}));
|
||||
|
||||
it('returns the database when a single lock-related failure is followed by a success', fakeAsync(() => {
|
||||
const lockErr = new Error('Internal error opening backing store');
|
||||
const fakeDb = makeFakeDb();
|
||||
openSpy.and.returnValues(Promise.reject(lockErr), Promise.resolve(fakeDb));
|
||||
|
||||
let resolved = false;
|
||||
retryService.init().then(() => {
|
||||
resolved = true;
|
||||
});
|
||||
|
||||
// First attempt rejects immediately; backoff before attempt 2 is 1s.
|
||||
tick(IDB_OPEN_RETRY_BASE_DELAY_MS);
|
||||
tick();
|
||||
|
||||
expect(openSpy).toHaveBeenCalledTimes(2);
|
||||
expect(resolved).toBe(true);
|
||||
expect((retryService as any)._db).toBe(fakeDb);
|
||||
}));
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -28,11 +28,13 @@ import {
|
|||
import {
|
||||
DUPLICATE_OPERATION_ERROR_MSG,
|
||||
OPERATION_LOG_STORE_NOT_INITIALIZED,
|
||||
isLockRelatedIdbOpenError,
|
||||
} from './op-log-errors.const';
|
||||
import { runDbUpgrade } from './db-upgrade';
|
||||
import { Log } from '../../core/log';
|
||||
import {
|
||||
IDB_OPEN_RETRIES,
|
||||
IDB_OPEN_RETRIES_NON_LOCK,
|
||||
IDB_OPEN_RETRY_BASE_DELAY_MS,
|
||||
} from '../core/operation-log.const';
|
||||
import { IndexedDBOpenError } from '../core/errors/indexed-db-open.error';
|
||||
|
|
@ -196,32 +198,64 @@ export class OperationLogStoreService {
|
|||
this._db = db;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wraps a single `openDB` call. Exists as a testing seam so specs can
|
||||
* `spyOn(service as any, '_openDbOnce')` to inject failures without mocking
|
||||
* the `idb` module import. Not intended to be called directly outside the
|
||||
* retry loop.
|
||||
*/
|
||||
private _openDbOnce(): Promise<IDBPDatabase<OpLogDB>> {
|
||||
return openDB<OpLogDB>(DB_NAME, DB_VERSION, {
|
||||
upgrade: (db, oldVersion, _newVersion, transaction) => {
|
||||
runDbUpgrade(db, oldVersion, transaction);
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Opens IndexedDB with retry logic and exponential backoff.
|
||||
* Transient failures (file locks, temporary I/O issues) may resolve on retry.
|
||||
*
|
||||
* Total attempts = 1 initial + IDB_OPEN_RETRIES retries.
|
||||
* With IDB_OPEN_RETRIES=3: attempts at 0ms, 500ms, 1500ms, 3500ms (total ~3.5s worst case).
|
||||
* The retry budget depends on the error:
|
||||
* - Lock-related errors (InvalidStateError, "backing store"): use the full
|
||||
* IDB_OPEN_RETRIES window (~31s) to outlast stale LevelDB locks from a
|
||||
* previous session. See issue #7191.
|
||||
* - Other errors: fall back to IDB_OPEN_RETRIES_NON_LOCK (~7s). Every op-log
|
||||
* read/write awaits `_ensureInit()`, so a 31s retry window on a non-lock
|
||||
* error blocks the entire op-log subsystem for 31s before the hydrator's
|
||||
* alert dialog reaches the user. There's no expectation that waiting
|
||||
* helps for non-lock errors, so fail fast.
|
||||
*
|
||||
* @throws IndexedDBOpenError if all retry attempts fail
|
||||
* @see https://github.com/johannesjo/super-productivity/issues/6255
|
||||
* @see https://github.com/super-productivity/super-productivity/issues/7191
|
||||
*/
|
||||
private async _openDbWithRetry(): Promise<IDBPDatabase<OpLogDB>> {
|
||||
const totalAttempts = 1 + IDB_OPEN_RETRIES;
|
||||
let maxRetries = IDB_OPEN_RETRIES;
|
||||
let attempt = 1;
|
||||
let lastError: unknown;
|
||||
|
||||
for (let attempt = 1; attempt <= totalAttempts; attempt++) {
|
||||
// Loop until either openDB succeeds or we exhaust the retry budget for the
|
||||
// observed error class. `maxRetries` may shrink after the first failure if
|
||||
// the error doesn't look lock-related.
|
||||
while (attempt <= 1 + maxRetries) {
|
||||
try {
|
||||
return await openDB<OpLogDB>(DB_NAME, DB_VERSION, {
|
||||
upgrade: (db, oldVersion, _newVersion, transaction) => {
|
||||
runDbUpgrade(db, oldVersion, transaction);
|
||||
},
|
||||
});
|
||||
return await this._openDbOnce();
|
||||
} catch (e) {
|
||||
lastError = e;
|
||||
|
||||
// Classify the error on the first failure. If it doesn't look
|
||||
// lock-related, shrink the retry budget so we fail fast and let the
|
||||
// hydrator surface the error instead of hanging for the full window.
|
||||
if (attempt === 1 && !isLockRelatedIdbOpenError(e)) {
|
||||
maxRetries = IDB_OPEN_RETRIES_NON_LOCK;
|
||||
}
|
||||
|
||||
const totalAttempts = 1 + maxRetries;
|
||||
if (attempt < totalAttempts) {
|
||||
// Exponential backoff: 500ms, 1000ms, 2000ms for retries 1, 2, 3
|
||||
// Exponential backoff: BASE * 2^(attempt-1). Lock errors retry up to
|
||||
// IDB_OPEN_RETRIES times (~31s total); non-lock errors truncate at
|
||||
// IDB_OPEN_RETRIES_NON_LOCK (~7s total).
|
||||
const delay = IDB_OPEN_RETRY_BASE_DELAY_MS * Math.pow(2, attempt - 1);
|
||||
Log.warn(
|
||||
`[OpLogStore] IndexedDB open failed (attempt ${attempt}/${totalAttempts}), retrying in ${delay}ms...`,
|
||||
|
|
@ -229,6 +263,8 @@ export class OperationLogStoreService {
|
|||
);
|
||||
await new Promise((resolve) => setTimeout(resolve, delay));
|
||||
}
|
||||
|
||||
attempt++;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue