etherpad-lite/src/node/db/DB.ts
John McLear d0180033e4
fix: page sessionstorage cleanup to avoid OOM (#7830) (#7831)
* fix: page sessionstorage cleanup to avoid OOM (#7830)

SessionStore._cleanup() previously called `findKeys('sessionstorage:*',
null)`, materialising every session key into a single array. On decade-
old MariaDB installs with millions of sessions this OOMs the node
process within ~15 minutes — see #7830.

Switch to ueberdb2 6.1.0's findKeysPaged with a 500-key page size, and
yield to the event loop between pages so the DB driver can release each
page's buffered rows and request handlers can interleave.

The break is now driven by `page.length === 0` rather than `page.length
< CLEANUP_PAGE_SIZE` so a stubbed/throttled paged source still iterates
the full keyspace.

Adds a regression test that seeds 50 sessionstorage rows, monkey-patches
`DB.findKeysPaged` to use a 4-key page, runs cleanup, and asserts every
expired row is removed plus every valid row preserved across page
boundaries.

Closes #7830

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: address Qodo review on #7831

Four follow-ups raised by Qodo on the session cleanup paging fix:

- DB.ts: fail-fast at init() if any required wrapper method (incl.
  findKeysPaged) is missing, so a stale ueberdb2 pin surfaces at boot
  rather than crashing the first cleanup run an hour later.
- SessionStore: bound a single _cleanup() run to 10 minutes. Under
  sustained session creation the keyspace can grow faster than cleanup
  drains it; without a budget the next scheduled run would never fire.
  When the budget hits, log a warning and let the next run continue.
- SessionStore: log the defensive `page[0] <= after` cursor-stall break.
  Previously the loop exited silently, leaving expired rows behind with
  no operator-visible signal of the backend regression.
- Tests: the paged-cleanup regression test now removes both expiredSids
  AND validSids in finally, so a failed assertion doesn't leak rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: note paged session cleanup in CHANGELOG + settings template

CHANGELOG.md picks up an entry under 3.1.0 Notable fixes describing the
OOM cause, the paged iteration, the 10-minute per-run budget, the
cursor-stall logging, and the fail-fast init guard.

settings.json.template's sessionCleanup comment adds the page-size,
budget, and pointer to #7830 so admins can reason about the new
behaviour from the template alone.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate lockfile against ueberdb2 6.1.2

Now that ether/ueberDB#983 unblocked the publish workflow (OIDC trusted
publishing), ueberdb2 6.1.2 is live on npm and the `^6.1.0` pin in
src/package.json resolves cleanly. Resolves the ERR_PNPM_OUTDATED_LOCKFILE
that was blocking CI on this PR.

29 SessionStore backend tests still green against the published tarball.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-22 10:56:35 +01:00

67 lines
2.3 KiB
TypeScript

'use strict';
/**
* The DB Module provides a database initialized with the settings
* provided by the settings module
*/
/*
* 2011 Peter 'Pita' Martischka (Primary Technology Ltd)
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS-IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import type {DatabaseType} from 'ueberdb2';
import settings from '../utils/Settings';
import log4js from 'log4js';
const stats = require('../stats')
const logger = log4js.getLogger('ueberDB');
/**
* The UeberDB Object that provides the database functions
*/
exports.db = null;
/**
* Initializes the database with the settings provided by the settings module
*/
exports.init = async () => {
// ueberdb2 v6 is ESM-only; load via dynamic import so CJS consumers work.
const {Database} = await import('ueberdb2');
exports.db = new Database(settings.dbType as DatabaseType, settings.dbSettings, null, logger);
await exports.db.init();
if (exports.db.metrics != null) {
for (const [metric, value] of Object.entries(exports.db.metrics)) {
if (typeof value !== 'number') continue;
stats.gauge(`ueberdb_${metric}`, () => exports.db.metrics[metric]);
}
}
for (const fn of ['get', 'set', 'findKeys', 'findKeysPaged', 'getSub', 'setSub', 'remove']) {
const f = exports.db[fn];
if (typeof f !== 'function') {
throw new Error(
`ueberdb2 ${exports.db.constructor.name} is missing required method ${fn}; ` +
'check that ueberdb2 is at the minimum version pinned in package.json');
}
exports[fn] = async (...args:string[]) => await f.call(exports.db, ...args);
Object.setPrototypeOf(exports[fn], Object.getPrototypeOf(f));
Object.defineProperties(exports[fn], Object.getOwnPropertyDescriptors(f));
}
};
exports.shutdown = async (hookName: string, context:any) => {
if (exports.db != null) await exports.db.close();
exports.db = null;
logger.log('Database closed');
};