mirror of
https://github.com/ether/etherpad-lite.git
synced 2026-07-18 00:57:55 +00:00
* docs(updater): PR 2 (Tier 2 manual-click) implementation plan 20-task TDD plan for shipping the manual-click update flow on top of the Tier 1 (notify) work merged in #7601. Covers UpdateExecutor, RollbackHandler, SessionDrainer, lock + trustedKeys, four admin endpoints (apply / cancel / acknowledge / log), admin UI updates, integration tests against a tmp git repo, and a manual smoke runbook for the spec's "before each tier ships" gate. Plan deliberately scopes signature verification to an opt-in stub (updates.requireSignature: false default) to avoid blocking on a separate release-signing project. Plan: docs/superpowers/plans/2026-05-08-auto-update-pr2-manual-click.md Spec: docs/superpowers/specs/2026-04-25-auto-update-design.md Issue: ether/etherpad#7607 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): extend state + settings for Tier 2 manual-click Adds ExecutionStatus discriminated union, bootCount, and lastResult to UpdateState, plus the preApplyGraceMinutes/drainSeconds/diskSpaceMinMB/ requireSignature/trustedKeysPath knobs that Tier 2's executor needs. loadState backfills the new fields on Tier 1 state files so existing installs keep working. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): PID-based update.lock with stale-pid reaping Single-flight guard for Tier 2's UpdateExecutor. Atomic O_CREAT|O_EXCL acquire; on EEXIST, sends signal 0 to the recorded PID and reaps if dead. Unparseable / partially-written lock files are treated as stale rather than fatal so a half-written lock from a SIGKILL'd parent doesn't lock the install out forever. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): verifyReleaseTag — gpg-via-git stub for Tier 2 preflight Default updates.requireSignature=false: log a warning and return ok with reason=signature-not-required. Set true to make preflight refuse a tag whose signature does not verify under the system keyring (or trustedKeysPath via GNUPGHOME). Etherpad's release process does not yet sign tags consistently; turning the check on by default would break Tier 2 for every admin and forcing a release-signing change is out of scope for this PR. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): preflight check pipeline for Tier 2 Pure orchestrator over injected probes for install-method, working tree, disk space, pnpm presence, lock state, remote tag existence and signature verification. Cheap-and-definitive checks run first; first failure short-circuits with a typed reason that the route layer will surface in the preflight-failed admin banner. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): rolling update.log helpers (appendLine + tailLines) Direct file-append + size-based rotation rather than a log4js appender — avoids re-configuring log4js on top of the user's existing logconfig. appendLine creates parents, rotates at 10MB (configurable), keeps 5 backups by default. tailLines reads the last N lines for /admin/update/log. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): SessionDrainer + handshake guard Drainer schedules T-60 / -30 / -10 broadcasts and resolves at T=0; isAcceptingConnections() flips off for the duration. PadMessageHandler consults the flag at the start of CLIENT_READY and disconnects new joiners with reason "updateInProgress" — existing sockets are unaffected. Drains shorter than 30s collapse the early timers to fire ASAP rather than queue past the drain end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): UpdateExecutor — snapshot, fetch/checkout/install/build, exit 75 Pure-DI orchestrator: spawnFn, copyFile, readSha, saveState, exit are all injected so unit tests run the full pipeline without spawning real children or mutating the real install. Streams stdout/stderr to update.log via the now-best-effort appendLine helper (swallows fs errors so the executor itself never breaks on read-only / unwritable log dirs). Failure paths transition to rolling-back and return — the route layer hands off to RollbackHandler which owns the rollback exit, so we don't double-exit and lose tail lines. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): RollbackHandler — health-check timer + crash-loop guard checkPendingVerification arms a 60s timer at boot when state is pending-verification and increments bootCount; bootCount>2 forces an immediate rollback (crash-loop guard). markVerified persists the verified state and stops the timer. performRollback restores the backup lockfile, runs git checkout <fromSha> and pnpm install, lands on rolled-back or rollback-failed (terminal) on sub-step failure, exits 75 either way so the supervisor restart brings the new state up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): wire RollbackHandler into boot + UpdatePolicy honours rollback-failed - expressCreateServer now invokes checkPendingVerification before polling starts so a previous boot's pending-verification either re-arms the health-check timer or, when bootCount has climbed past the crash-loop threshold, forces an immediate rollback. - server.ts calls markBootHealthy after state hits RUNNING so /health-being-up is the implicit happy-path signal that cancels the rollback timer. - /admin/update/status surfaces execution + lastResult + lockHeld so the admin UI can render the right Apply / Cancel / Acknowledge state. - UpdatePolicy gains an `executionStatus` input. While it equals 'rollback-failed', canAuto / canAutonomous are denied (reason: rollback-failed-terminal); manual stays on because clicking Apply IS the intervention the terminal state needs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): apply / cancel / acknowledge / log endpoints Strict admin-only POSTs that drive Tier 2's manual-click flow: - POST /admin/update/apply: acquire lock, persist preflight, run preflight, drain $drainSeconds, executeUpdate (which exits 75 on success), or run performRollback on a failure path (also exits 75). - POST /admin/update/cancel: cancel a pre-execute drain/preflight, write cancelled lastResult, release lock. - POST /admin/update/acknowledge: clear terminal states (preflight-failed, rolled-back, rollback-failed) back to idle. lastResult is preserved so the admin still sees what happened. - GET /admin/update/log: tail var/log/update.log (200 lines) for the in- progress UI. Strict admin auth. Also: - socketio hook exports getIo() so the apply endpoint can broadcast the drain shoutMessage outside the regular hook surface. - ep.json registers updateActions after admin/updateStatus. - 11 mocha integration tests cover auth, policy denial, execution-busy, acknowledge-clears-terminal, log content-type. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): admin UI Apply/Cancel/Acknowledge + live log stream UpdatePage renders the right action set based on execution.status: Apply when idle/verified and policy allows, Cancel during preflight/draining, Acknowledge on terminal preflight-failed / rolled-back / rollback-failed. While the executor is in flight (preflight/draining/executing/rolling-back) the page polls /admin/update/log + /admin/update/status once a second and shows the rolling tail; polling stops automatically when the run terminates. lastResult and policy denial reasons surface localised copy. Buttons disable themselves while a network round-trip is in flight to dodge double-clicks. New i18n keys live under update.page.{apply,cancel, acknowledge,log,execution,policy.*,last_result.*}, update.execution.*, update.banner.terminal.rollback-failed, and update.drain.{t60,t30,t10}. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): pad shoutMessage renders update.drain.* via html10n broadcastShout now sends {messageKey, values, sticky} so the existing pad-side shout pipeline can route through html10n.get(). The renderer gains a values pass-through so update.drain.t60 etc. interpolate {{seconds}}, and gives updater shouts a different gritter title (the banner.title localised string) so users know it's a system event rather than a generic admin message. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): rollback uses git checkout -f + integration suite over tmp git repo RollbackHandler now does git checkout -f <fromSha> BEFORE overlaying the backup lockfile. Without -f, git refuses checkout when there are unstaged modifications to files it would overwrite — exactly the case after a partial executor run that mutated the working tree. With -f the partial mutation is discarded and the working tree returns to fromSha cleanly. The backup-lockfile copy is still done (belt-and-braces) but tolerates ENOENT since checkout already restored the right lockfile. The new integration suite at src/tests/backend/specs/updater-integration.ts exercises the full pipeline against a disposable git repo: happy path, install-fail rollback, build-fail rollback, crash-loop guard, and a target-sha-doesn't-exist rollback-failed terminal case. 5 mocha tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(updater): Playwright admin Apply / Cancel / Acknowledge flow Stubs /admin/update/status (and /admin/update/apply for the apply path) at the route level so we can assert UI transitions without actually running an update. Four scenarios: - Apply button POSTs and re-fetches status (>=2 status fetches total). - install-method-not-writable hides the button and shows localised denial copy. - rollback-failed terminal state shows the Acknowledge button and the "Manual intervention required" lastResult copy. - lockHeld=true hides Apply even when policy.canManual is on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): admin banner shows rollback-failed terminal alert When execution.status === 'rollback-failed' the banner switches to a role=alert with the strong update.banner.terminal.rollback-failed copy and overrides the regular "update available" framing — an admin who left the system in this state needs to fix it before any other admin work matters. Other terminal states (preflight-failed, rolled-back) are informational and surface on the page itself, not the banner. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(updater): Tier 2 admin docs + manual smoke runbook + CHANGELOG doc/admin/updates.md gains a full Tier 2 section: prerequisites (git install + process supervisor with sample systemd unit), Apply flow with timings, every failure mode and the resulting state, the four endpoints, and the signature-verification opt-in. Settings table picks up the new updates.* knobs. docs/superpowers/specs/2026-04-25-auto-update-runbook.md is the manual smoke runbook the design spec calls for: disposable VM, systemd unit, every observable transition (happy path, install/ build-fail rollback, crash-loop guard, rollback-failed terminal, cancel during drain) plus a sign-off checklist for the release cut. CHANGELOG Unreleased section explains the supervisor requirement and points readers at the runbook. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(updater): note docker-friendly update flows as follow-up work Tier 2 refuses Apply on installMethod=docker because in-container mutation doesn't survive a container restart. Adds a future-work note covering the two reasonable paths for an in-product docker Apply button (instructions-only vs deploy-webhook) and explicitly rules out mounting /var/run/docker.sock as a footgun. Watchtower gets a pointer for admins who want fully autonomous docker updates today. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(updater): address Qodo review (1-6) + Playwright strict-mode CI fix 1. Tier 2 endpoints now gate on tier in {manual, auto, autonomous} — notify and off return 404 to match the prior PR-1 behaviour. Gate is evaluated per-request via app.use middleware so a settings.json reload takes effect without a full restart, and so integration tests can flip the tier dynamically. Adds a regression test that exercises 404 at tier=notify across all four endpoints. 2. cancel/apply race fixed: /admin/update/cancel no longer releases the lock — apply's finally block owns it for the request's lifetime. Apply now reloads state after preflight and aborts with 409 cancelled-during- preflight if execution.status is no longer 'preflight' for the same targetTag. Prevents a second apply from sneaking in while the first is still running its slow checks, and prevents the post-cancel apply from continuing into drain/execute. 3. SessionDrainer now restores acceptingConnections=true at drain completion (not just on cancel). The lock + persisted execution.status prevent a fresh apply from racing in — the in-memory flag was redundant safety that turned into a wedge if the executor threw post-drain. Adds a unit test asserting the flag is restored after natural drain end. 4. PadMessageHandler drain guard switched from socket.json.send (a socket.io v2/v3 API that may not exist on v4) to socket.emit('message', ...) for consistency with the other disconnect paths in the file. 5. Spawn 'error' handlers added to runStep helpers in UpdateExecutor and RollbackHandler, plus the gpg verify-tag spawn in trustedKeys. Without them, a missing/unexecutable binary leaves the promise hanging forever and the update flow stuck in-flight. SpawnFn type extended to allow on('error', ...) listeners cleanly. Spawn errors now resolve with code 1 + the error message in stderr, so the existing failure-detection branches fire normally. 6. executeUpdate body wrapped in try/catch. An exception from readSha, saveState, copyFile, or any step now lands in a rolling-back persist + returns failed-checkout, so the route's post-executor rollback path picks it up. State can no longer wedge at 'executing'. The catch's inner saveState is itself try/wrapped so a write-after-write failure doesn't crash the route either. CI: Playwright update-page-actions strict-mode violation fixed. Both the banner and the lastResult <p> contain "Manual intervention required"; selector now scopes to p.last-result-rollback-failed for the lastResult assertion specifically. 129 vitest unit tests + 23 mocha integration tests passing; ts-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(updater): address Qodo #7 (status leak) + #8 (short-drain values) #7. /admin/update/status now redacts diagnostic strings for unauth callers even when requireAdminForStatus is left at its default (false). Status enum + outcome enum are kept (the admin banner / pad-side badge need them to render the right UI) but execution.reason / execution.fromSha / execution.targetTag and the same fields on lastResult are stripped. Authed admin sessions still get the full payload — they're looking at their own server's diagnostics. Two new mocha tests cover both paths: "redacts execution.reason / lastResult.reason for unauth callers" and "returns full diagnostic payload to authed admin sessions". #8. SessionDrainer no longer schedules T-30 / T-10 broadcasts when the configured drainSeconds can't honour them. Previously, with drainSeconds < 30 the T-30 timer fired at zero remaining but the broadcast still claimed "30 seconds" — misleading. Now T-30 only schedules when drainSeconds > 30 and T-10 only when > 10. Admins picking a short drain get fewer announcements but each carries an accurate countdown. The opening announcement now reports the configured drain length rather than a hardcoded 60. Two updated unit tests: drainSeconds=15 (skips T-30, still fires T-10) and drainSeconds=5 (skips both). 131 vitest unit + 26 mocha integration tests passing; ts-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(updater): address Qodo follow-up — tag injection, rollback rejections, state validation Qodo posted three new concerns after the first fix push. 1. Git tag option injection (security). The release tag from GitHub's tag_name flowed into `git checkout` / `git verify-tag` as a positional arg. A tag starting with '-' would be parsed as an option and could bypass signature verification or change checkout semantics. Mitigated in three layers: - New refSafety helper (isValidTag / assertValidTag / refsTagsForm) enforces a strict subset of git's check-ref-format spec: rejects leading '-' or '.', whitespace, control chars, and ~ ^ : ? * [ \\ and the '..' sequence. - VersionChecker validates tag_name before persisting to state, so a malformed value from a misconfigured githubRepo never lands on disk. - UpdateExecutor calls assertValidTag and uses the refs/tags/<tag> form for git checkout. trustedKeys also validates and adds '--' to git verify-tag for an end-of-options marker. updateActions does an up-front isValidTag check on state.latest.tag so a corrupt state file gets a clean 409 instead of a 500. 2. Unhandled rollback rejections. checkPendingVerification was firing `void deps.saveState(...)` and `void performRollback(...)` without .catch(), so an fs error during boot's rollback path would bubble out as an unhandled rejection. Both callsites now go through fireSaveState / fireRollback helpers that catch and log; rollback rejections fall through to a best-effort terminal-state write + exit 75 so the supervisor can re-try the next boot with bootCount++. 3. Execution state under-validated. isValidExecution previously checked only that `status` was a known enum value, so a hand-edited state file with `{execution: {status: 'pending-verification'}}` (missing fromSha / targetTag / deadlineAt) would pass validation and reach RollbackHandler with undefined refs. The validator now consults a per-status required-fields map mirroring the ExecutionStatus union in types.ts and rejects empty strings as well as missing fields. Same tightening applied to lastResult.outcome (must be in the allowed enum, not just any string). Six new unit tests cover hand-edited corruption. 145 vitest + 26 mocha tests green; ts-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
John McLear
GitHub
parent
4f1b524864
commit
efb8328084
45 changed files with 6991 additions and 47 deletions
|
|
@ -37,6 +37,7 @@ import settings, {
|
|||
sofficeAvailable
|
||||
} from '../utils/Settings';
|
||||
import {anonymizeIp} from '../utils/anonymizeIp';
|
||||
import {isAcceptingConnections} from '../updater/SessionDrainer';
|
||||
const logIp = (ip: string | null | undefined) => anonymizeIp(ip, settings.ipLogging);
|
||||
const securityManager = require('../db/SecurityManager');
|
||||
const plugins = require('../../static/js/pluginfw/plugin_defs');
|
||||
|
|
@ -377,6 +378,17 @@ exports.handleMessage = async (socket:any, message: ClientVarMessage) => {
|
|||
if (!thisSession) throw new Error('message from an unknown connection');
|
||||
|
||||
if (message.type === 'CLIENT_READY') {
|
||||
// Refuse new joiners while the updater drainer is running. Existing sockets
|
||||
// are unaffected — only the initial CLIENT_READY handshake is gated. The
|
||||
// pad UI will show the drain announcement separately via shoutMessage.
|
||||
// Use socket.emit('message', ...) for consistency with the other disconnect
|
||||
// paths in this file (see line ~221, 569). socket.json.send is a socket.io
|
||||
// v2/v3-era API that may not exist on v4 Socket objects.
|
||||
if (!isAcceptingConnections()) {
|
||||
socket.emit('message', {disconnect: 'updateInProgress'});
|
||||
socket.disconnect(true);
|
||||
return;
|
||||
}
|
||||
// Prefer the HttpOnly author-token cookie over the in-message token (GDPR
|
||||
// PR3). Legacy clients (pre-PR3 browsers or API consumers) still send
|
||||
// `token` in the CLIENT_READY payload — honour it one more release, warn
|
||||
|
|
|
|||
|
|
@ -14,6 +14,9 @@ const padMessageHandler = require('../../handler/PadMessageHandler');
|
|||
|
||||
let io:any;
|
||||
const logger = log4js.getLogger('socket.io');
|
||||
|
||||
/** Returns the socket.io Server once expressCreateServer has run, or null otherwise. Used by features that need to broadcast outside the regular hook surface. */
|
||||
export const getIo = (): any => io;
|
||||
const sockets = new Set();
|
||||
const socketsEvents = new events.EventEmitter();
|
||||
|
||||
|
|
|
|||
360
src/node/hooks/express/updateActions.ts
Normal file
360
src/node/hooks/express/updateActions.ts
Normal file
|
|
@ -0,0 +1,360 @@
|
|||
'use strict';
|
||||
|
||||
import path from 'node:path';
|
||||
import fs from 'node:fs/promises';
|
||||
import {spawn} from 'node:child_process';
|
||||
import log4js from 'log4js';
|
||||
import {ArgsExpressType} from '../../types/ArgsExpressType';
|
||||
import settings, {getEpVersion} from '../../utils/Settings';
|
||||
import {getDetectedInstallMethod, stateFilePath, getRollbackDeps} from '../../updater';
|
||||
import {evaluatePolicy} from '../../updater/UpdatePolicy';
|
||||
import {loadState, saveState} from '../../updater/state';
|
||||
import {acquireLock, releaseLock} from '../../updater/lock';
|
||||
import {executeUpdate, SpawnFn} from '../../updater/UpdateExecutor';
|
||||
import {createDrainer, DrainBroadcastKey, Drainer} from '../../updater/SessionDrainer';
|
||||
import {runPreflight} from '../../updater/preflight';
|
||||
import {verifyReleaseTag} from '../../updater/trustedKeys';
|
||||
import {tailLines, appendLine} from '../../updater/updateLog';
|
||||
import {performRollback} from '../../updater/RollbackHandler';
|
||||
import {UpdateState} from '../../updater/types';
|
||||
import {isValidTag} from '../../updater/refSafety';
|
||||
import {getIo} from './socketio';
|
||||
|
||||
const logger = log4js.getLogger('updater');
|
||||
|
||||
const lockPath = (): string => path.join(settings.root, 'var', 'update.lock');
|
||||
const logPath = (): string => path.join(settings.root, 'var', 'log', 'update.log');
|
||||
const backupDir = (): string => path.join(settings.root, 'var', 'update-backup');
|
||||
|
||||
let drainer: Drainer | null = null;
|
||||
|
||||
const requireAdmin = (req: any, res: any): boolean => {
|
||||
const u = req.session?.user;
|
||||
if (!u) { res.status(401).send('Authentication required'); return false; }
|
||||
if (!u.is_admin) { res.status(403).send('Forbidden'); return false; }
|
||||
return true;
|
||||
};
|
||||
|
||||
const wrapAsync =
|
||||
(fn: (req: any, res: any, next: Function) => Promise<unknown>) =>
|
||||
(req: any, res: any, next: Function) => Promise.resolve(fn(req, res, next)).catch((err) => next(err));
|
||||
|
||||
const broadcastShout = (key: DrainBroadcastKey, values: Record<string, unknown>): void => {
|
||||
try {
|
||||
const io = getIo();
|
||||
if (!io) return;
|
||||
// The pad-side renderer (src/static/js/pad.ts) already handles `messageKey`
|
||||
// by routing through html10n.get(); we add a `values` field that the
|
||||
// renderer interpolates into the localised string.
|
||||
const message = {
|
||||
type: 'COLLABROOM',
|
||||
data: {
|
||||
type: 'shoutMessage',
|
||||
payload: {
|
||||
message: {messageKey: key, values, sticky: false},
|
||||
timestamp: Date.now(),
|
||||
},
|
||||
},
|
||||
};
|
||||
io.sockets.emit('shout', message);
|
||||
} catch (err) {
|
||||
logger.warn(`broadcastShout: ${(err as Error).message}`);
|
||||
}
|
||||
};
|
||||
|
||||
const buildPreflightDeps = (installMethod: ReturnType<typeof getDetectedInstallMethod>) => ({
|
||||
installMethod,
|
||||
workingTreeClean: () => new Promise<boolean>((resolve) => {
|
||||
const c = spawn('git', ['status', '--porcelain'], {cwd: settings.root});
|
||||
let out = '';
|
||||
c.stdout.on('data', (b) => { out += b.toString(); });
|
||||
c.on('close', () => resolve(out.trim().length === 0));
|
||||
c.on('error', () => resolve(false));
|
||||
}),
|
||||
freeDiskMB: async (): Promise<number> => {
|
||||
try {
|
||||
const s = await (fs as any).statfs?.(settings.root);
|
||||
if (!s) return Number.POSITIVE_INFINITY;
|
||||
return Math.floor((Number(s.bavail) * Number(s.bsize)) / (1024 * 1024));
|
||||
} catch {
|
||||
// statfs unsupported on this platform — treat as "no constraint" rather than block.
|
||||
return Number.POSITIVE_INFINITY;
|
||||
}
|
||||
},
|
||||
pnpmOnPath: () => new Promise<boolean>((resolve) => {
|
||||
const c = spawn('pnpm', ['--version'], {stdio: 'ignore'});
|
||||
c.on('close', (code) => resolve(code === 0));
|
||||
c.on('error', () => resolve(false));
|
||||
}),
|
||||
// We just acquired the lock in the apply endpoint, so don't double-check it here.
|
||||
lockHeld: async () => false,
|
||||
remoteHasTag: (tag: string) => new Promise<boolean>((resolve) => {
|
||||
const c = spawn('git', ['ls-remote', '--tags', 'origin', tag],
|
||||
{cwd: settings.root, stdio: ['ignore', 'pipe', 'ignore']});
|
||||
let out = '';
|
||||
c.stdout.on('data', (b) => { out += b.toString(); });
|
||||
c.on('close', () => resolve(out.trim().length > 0));
|
||||
c.on('error', () => resolve(false));
|
||||
}),
|
||||
verifyTag: () => verifyReleaseTag({
|
||||
tag: '', // overridden below — we close over targetTag
|
||||
repoDir: settings.root,
|
||||
requireSignature: settings.updates.requireSignature,
|
||||
trustedKeysPath: settings.updates.trustedKeysPath,
|
||||
}),
|
||||
});
|
||||
|
||||
/**
|
||||
* The set of update tiers at which the Tier 2 action endpoints serve.
|
||||
* `notify` only ships read-only routes (registered in updateStatus.ts);
|
||||
* `manual` and higher are the supersets that include manual-click. Disabled
|
||||
* paths (off / notify) match prior behaviour: requests 404, no new attack
|
||||
* surface vs PR 1.
|
||||
*
|
||||
* Read at request time (not hook-init time) so that operators flipping
|
||||
* `updates.tier` in settings.json + reloading take effect without a full
|
||||
* restart, and so that integration tests can drive the gate dynamically.
|
||||
*/
|
||||
const TIER2_TIERS: ReadonlySet<string> = new Set(['manual', 'auto', 'autonomous']);
|
||||
const tierAllowsActions = (): boolean => TIER2_TIERS.has(settings.updates.tier);
|
||||
|
||||
export const expressCreateServer = (
|
||||
_hookName: string,
|
||||
{app}: ArgsExpressType,
|
||||
cb: Function,
|
||||
): void => {
|
||||
// Always register the routes; gate at request time so a runtime tier change
|
||||
// takes effect on the next request rather than requiring a restart.
|
||||
// The early 404 below preserves Qodo #1's "disabled path matches prior
|
||||
// behaviour (no Tier 2 endpoints existed before this PR)" requirement.
|
||||
const tierGate = (req: any, res: any, next: Function) => {
|
||||
if (!tierAllowsActions()) return res.status(404).send('Not found');
|
||||
next();
|
||||
};
|
||||
app.use(['/admin/update/apply', '/admin/update/cancel', '/admin/update/acknowledge', '/admin/update/log'], tierGate);
|
||||
|
||||
app.post('/admin/update/apply', wrapAsync(async (req: any, res: any) => {
|
||||
if (!requireAdmin(req, res)) return;
|
||||
|
||||
const state = await loadState(stateFilePath());
|
||||
if (!state.latest) return res.status(409).json({error: 'no-known-latest'});
|
||||
|
||||
// Defence in depth: VersionChecker validates tag_name before persisting,
|
||||
// but a hand-edited update-state.json could still surface an unsafe tag
|
||||
// here. Reject up-front rather than throw later when the executor calls
|
||||
// assertValidTag, so the admin sees a clear 409 instead of a 500.
|
||||
if (!isValidTag(state.latest.tag)) {
|
||||
return res.status(409).json({error: 'invalid-tag-in-state'});
|
||||
}
|
||||
|
||||
// Allowed entry statuses: idle / verified / preflight-failed / rolled-back.
|
||||
// Anything else means an in-flight or terminal-needs-acknowledge state.
|
||||
const allowedEntry = ['idle', 'verified', 'preflight-failed', 'rolled-back'];
|
||||
if (!allowedEntry.includes(state.execution.status)) {
|
||||
return res.status(409).json({error: `execution-busy:${state.execution.status}`});
|
||||
}
|
||||
|
||||
const installMethod = getDetectedInstallMethod();
|
||||
const policy = evaluatePolicy({
|
||||
installMethod,
|
||||
tier: settings.updates.tier,
|
||||
current: getEpVersion(),
|
||||
latest: state.latest.version,
|
||||
executionStatus: state.execution.status,
|
||||
});
|
||||
if (!policy.canManual) {
|
||||
return res.status(409).json({error: 'policy-denied', reason: policy.reason});
|
||||
}
|
||||
|
||||
if (!await acquireLock(lockPath())) {
|
||||
return res.status(409).json({error: 'lock-held'});
|
||||
}
|
||||
|
||||
const targetTag = state.latest.tag;
|
||||
let cleanupLock = true;
|
||||
|
||||
try {
|
||||
// Persist preflight state.
|
||||
const startedAt = new Date().toISOString();
|
||||
const preState: UpdateState = {
|
||||
...state,
|
||||
execution: {status: 'preflight', targetTag, startedAt},
|
||||
};
|
||||
await saveState(stateFilePath(), preState);
|
||||
appendLine(logPath(), `[${startedAt}] PREFLIGHT target=${targetTag}`);
|
||||
|
||||
const baseDeps = buildPreflightDeps(installMethod);
|
||||
const pf = await runPreflight(
|
||||
{
|
||||
targetTag,
|
||||
diskSpaceMinMB: Number(settings.updates.diskSpaceMinMB) || 500,
|
||||
requireSignature: settings.updates.requireSignature,
|
||||
trustedKeysPath: settings.updates.trustedKeysPath,
|
||||
},
|
||||
{
|
||||
...baseDeps,
|
||||
verifyTag: () => verifyReleaseTag({
|
||||
tag: targetTag,
|
||||
repoDir: settings.root,
|
||||
requireSignature: settings.updates.requireSignature,
|
||||
trustedKeysPath: settings.updates.trustedKeysPath,
|
||||
}),
|
||||
},
|
||||
);
|
||||
|
||||
if (!pf.ok) {
|
||||
const at = new Date().toISOString();
|
||||
await saveState(stateFilePath(), {
|
||||
...preState,
|
||||
execution: {status: 'preflight-failed', targetTag, reason: pf.reason, at},
|
||||
lastResult: {
|
||||
targetTag, fromSha: '',
|
||||
outcome: 'preflight-failed', reason: pf.reason, at,
|
||||
},
|
||||
});
|
||||
appendLine(logPath(), `[${at}] PREFLIGHT_FAILED ${pf.reason}`);
|
||||
cleanupLock = true;
|
||||
return res.status(409).json({error: 'preflight-failed', reason: pf.reason});
|
||||
}
|
||||
|
||||
// Re-check state after preflight: /admin/update/cancel may have flipped
|
||||
// execution back to 'idle' while we were running the slow checks. The
|
||||
// cancel handler intentionally leaves the lock alone (we own it) and
|
||||
// signals via state instead, so a stale apply can detect cancellation
|
||||
// here before mutating the filesystem.
|
||||
const afterPreflight = await loadState(stateFilePath());
|
||||
if (afterPreflight.execution.status !== 'preflight'
|
||||
|| (afterPreflight.execution as {targetTag?: string}).targetTag !== targetTag) {
|
||||
appendLine(logPath(),
|
||||
`[${new Date().toISOString()}] APPLY aborted post-preflight (state=${afterPreflight.execution.status})`);
|
||||
return res.status(409).json({error: 'cancelled-during-preflight'});
|
||||
}
|
||||
|
||||
// Drain — respond 202 first so the UI starts polling /log without waiting.
|
||||
const drainSeconds = Number(settings.updates.drainSeconds) || 60;
|
||||
drainer = createDrainer({
|
||||
drainSeconds,
|
||||
broadcast: (key, values) => broadcastShout(key, values),
|
||||
});
|
||||
const drainEndsAt = new Date(Date.now() + drainSeconds * 1000).toISOString();
|
||||
await saveState(stateFilePath(), {
|
||||
...preState,
|
||||
execution: {status: 'draining', targetTag, drainEndsAt, startedAt: new Date().toISOString()},
|
||||
});
|
||||
appendLine(logPath(), `[${new Date().toISOString()}] DRAIN start drainSeconds=${drainSeconds}`);
|
||||
|
||||
res.status(202).json({accepted: true, drainEndsAt});
|
||||
|
||||
const drainResult = await drainer.start();
|
||||
drainer = null;
|
||||
if (drainResult.outcome === 'cancelled') {
|
||||
// /admin/update/cancel already updated state and lastResult; just release the lock.
|
||||
appendLine(logPath(), `[${new Date().toISOString()}] DRAIN cancelled by admin`);
|
||||
return;
|
||||
}
|
||||
|
||||
// Re-load state right before the executor runs so anything the cancel
|
||||
// endpoint or another concurrent handler wrote is honoured.
|
||||
const fresh = await loadState(stateFilePath());
|
||||
|
||||
const r = await executeUpdate({
|
||||
repoDir: settings.root,
|
||||
backupDir: backupDir(),
|
||||
spawnFn: spawn as unknown as SpawnFn,
|
||||
readSha: () => new Promise<string>((resolve, reject) => {
|
||||
const c = spawn('git', ['rev-parse', 'HEAD'],
|
||||
{cwd: settings.root, stdio: ['ignore', 'pipe', 'ignore']});
|
||||
let out = '';
|
||||
c.stdout.on('data', (b) => { out += b.toString(); });
|
||||
c.on('close', (code) => code === 0
|
||||
? resolve(out.trim())
|
||||
: reject(new Error(`git rev-parse exit ${code}`)));
|
||||
c.on('error', reject);
|
||||
}),
|
||||
copyFile: async (src: string, dst: string) => {
|
||||
await fs.mkdir(path.dirname(dst), {recursive: true});
|
||||
await fs.copyFile(src, dst);
|
||||
},
|
||||
saveState: (s: UpdateState) => saveState(stateFilePath(), s),
|
||||
initialState: fresh,
|
||||
targetTag,
|
||||
now: () => new Date(),
|
||||
// executeUpdate calls exit on success (75) — that takes the process down,
|
||||
// so anything after this is the failure path.
|
||||
exit: (code: number) => process.exit(code),
|
||||
});
|
||||
|
||||
// Failure paths: executor returned without exiting, state is rolling-back.
|
||||
if (r.outcome !== 'pending-verification') {
|
||||
const after = await loadState(stateFilePath());
|
||||
if (after.execution.status === 'rolling-back') {
|
||||
// performRollback will exit 75 on either success or terminal failure.
|
||||
// We do not release the lock — exit takes the process down and the
|
||||
// next-boot acquireLock reaps the stale PID.
|
||||
cleanupLock = false;
|
||||
await performRollback(after, getRollbackDeps());
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
logger.error(`apply failed: ${(err as Error).stack || err}`);
|
||||
appendLine(logPath(), `[${new Date().toISOString()}] APPLY_ERROR ${(err as Error).message}`);
|
||||
if (!res.headersSent) res.status(500).json({error: 'internal'});
|
||||
} finally {
|
||||
if (cleanupLock) {
|
||||
try { await releaseLock(lockPath()); }
|
||||
catch (err) { logger.warn(`releaseLock: ${(err as Error).message}`); }
|
||||
}
|
||||
}
|
||||
}));
|
||||
|
||||
app.post('/admin/update/cancel', wrapAsync(async (req: any, res: any) => {
|
||||
if (!requireAdmin(req, res)) return;
|
||||
const state = await loadState(stateFilePath());
|
||||
// Cancel is allowed only during pre-execute states. Once executing begins
|
||||
// (filesystem mutated) we either complete or rollback — see spec section
|
||||
// "Error handling" / state machine.
|
||||
if (state.execution.status !== 'preflight' && state.execution.status !== 'draining') {
|
||||
return res.status(409).json({error: 'not-cancellable', status: state.execution.status});
|
||||
}
|
||||
if (drainer) drainer.cancel();
|
||||
const at = new Date().toISOString();
|
||||
await saveState(stateFilePath(), {
|
||||
...state,
|
||||
execution: {status: 'idle'},
|
||||
lastResult: {
|
||||
targetTag: (state.execution as {targetTag?: string}).targetTag ?? '',
|
||||
fromSha: '',
|
||||
outcome: 'cancelled',
|
||||
reason: 'admin-cancelled',
|
||||
at,
|
||||
},
|
||||
});
|
||||
// Intentionally do NOT release the lock here. The apply handler owns the
|
||||
// lock for its lifetime and releases it in its finally block; releasing
|
||||
// here would let a second apply slip in while the first is still mid-
|
||||
// preflight, racing for the same on-disk state.
|
||||
appendLine(logPath(), `[${at}] CANCEL by admin during status=${state.execution.status}`);
|
||||
res.json({cancelled: true});
|
||||
}));
|
||||
|
||||
app.post('/admin/update/acknowledge', wrapAsync(async (req: any, res: any) => {
|
||||
if (!requireAdmin(req, res)) return;
|
||||
const state = await loadState(stateFilePath());
|
||||
const terminal: ReadonlySet<string> = new Set(['rollback-failed', 'preflight-failed', 'rolled-back']);
|
||||
if (!terminal.has(state.execution.status)) {
|
||||
return res.status(409).json({error: 'not-terminal', status: state.execution.status});
|
||||
}
|
||||
await saveState(stateFilePath(), {...state, execution: {status: 'idle'}, bootCount: 0});
|
||||
appendLine(logPath(), `[${new Date().toISOString()}] ACKNOWLEDGE ${state.execution.status} -> idle`);
|
||||
res.json({acknowledged: true});
|
||||
}));
|
||||
|
||||
app.get('/admin/update/log', wrapAsync(async (req: any, res: any) => {
|
||||
if (!requireAdmin(req, res)) return;
|
||||
const lines = await tailLines(logPath(), 200);
|
||||
res.set('Content-Type', 'text/plain; charset=utf-8');
|
||||
res.send(lines.join('\n'));
|
||||
}));
|
||||
|
||||
cb();
|
||||
};
|
||||
|
|
@ -1,11 +1,13 @@
|
|||
'use strict';
|
||||
|
||||
import path from 'node:path';
|
||||
import {ArgsExpressType} from '../../types/ArgsExpressType';
|
||||
import settings, {getEpVersion} from '../../utils/Settings';
|
||||
import {getDetectedInstallMethod, stateFilePath} from '../../updater';
|
||||
import {evaluatePolicy} from '../../updater/UpdatePolicy';
|
||||
import {compareSemver, isMajorBehind, isVulnerable} from '../../updater/versionCompare';
|
||||
import {loadState} from '../../updater/state';
|
||||
import {isHeld} from '../../updater/lock';
|
||||
|
||||
|
||||
let badgeCache: {value: 'severe' | 'vulnerable' | null; at: number} = {value: null, at: 0};
|
||||
|
|
@ -37,6 +39,23 @@ const wrapAsync = (fn: (req: any, res: any, next: Function) => Promise<unknown>)
|
|||
Promise.resolve(fn(req, res, next)).catch((err) => next(err));
|
||||
};
|
||||
|
||||
/**
|
||||
* Strip diagnostic strings (reason, fromSha, targetTag, build/install paths)
|
||||
* from execution before exposing to unauthenticated callers. Status enum is
|
||||
* preserved so the admin banner / pad-side badge can still render the right UI.
|
||||
*/
|
||||
const sanitizeExecution = (e: any): any => {
|
||||
if (!e || typeof e !== 'object' || typeof e.status !== 'string') return {status: 'idle'};
|
||||
return {status: e.status};
|
||||
};
|
||||
|
||||
const sanitizeLastResult = (r: any): any => {
|
||||
if (r === null) return null;
|
||||
if (!r || typeof r !== 'object' || typeof r.outcome !== 'string') return null;
|
||||
// outcome enum + at timestamp are non-sensitive. reason / fromSha / targetTag are dropped.
|
||||
return {outcome: r.outcome, at: typeof r.at === 'string' ? r.at : null};
|
||||
};
|
||||
|
||||
export const expressCreateServer = (
|
||||
_hookName: string,
|
||||
{app}: ArgsExpressType,
|
||||
|
|
@ -68,6 +87,7 @@ export const expressCreateServer = (
|
|||
// release. Admins who want the endpoint gated to authenticated admin sessions —
|
||||
// without disabling the updater entirely — set updates.requireAdminForStatus=true.
|
||||
app.get('/admin/update/status', wrapAsync(async (req, res) => {
|
||||
const isAdmin = !!req.session?.user?.is_admin;
|
||||
if (settings.updates.requireAdminForStatus) {
|
||||
const user = req.session?.user;
|
||||
if (!user) return res.status(401).send('Authentication required');
|
||||
|
|
@ -77,8 +97,29 @@ export const expressCreateServer = (
|
|||
const current = getEpVersion();
|
||||
const installMethod = getDetectedInstallMethod();
|
||||
const policy = state.latest
|
||||
? evaluatePolicy({installMethod, tier: settings.updates.tier, current, latest: state.latest.version})
|
||||
? evaluatePolicy({
|
||||
installMethod,
|
||||
tier: settings.updates.tier,
|
||||
current,
|
||||
latest: state.latest.version,
|
||||
executionStatus: state.execution.status,
|
||||
})
|
||||
: null;
|
||||
const lockHeld = await isHeld(path.join(settings.root, 'var', 'update.lock'));
|
||||
|
||||
// The Tier 2 fields (execution, lastResult) carry diagnostic strings
|
||||
// built from git/pnpm stderr — environment-specific paths, error
|
||||
// messages, etc. Endpoint defaults to unauthenticated; only authed
|
||||
// admin sessions see the full diagnostic payload. Everyone else sees
|
||||
// just the status enum + outcome enum so the pad-side / public banners
|
||||
// can still render correctly without leaking operational detail.
|
||||
const execution = isAdmin
|
||||
? state.execution
|
||||
: sanitizeExecution(state.execution);
|
||||
const lastResult = isAdmin
|
||||
? state.lastResult
|
||||
: sanitizeLastResult(state.lastResult);
|
||||
|
||||
res.json({
|
||||
currentVersion: current,
|
||||
latest: state.latest,
|
||||
|
|
@ -87,6 +128,10 @@ export const expressCreateServer = (
|
|||
tier: settings.updates.tier,
|
||||
policy,
|
||||
vulnerableBelow: state.vulnerableBelow,
|
||||
// PR 2 additions:
|
||||
execution,
|
||||
lastResult,
|
||||
lockHeld,
|
||||
});
|
||||
}));
|
||||
|
||||
|
|
|
|||
|
|
@ -177,6 +177,17 @@ exports.start = async () => {
|
|||
// @ts-ignore
|
||||
startDoneGate.resolve();
|
||||
|
||||
// Once the server is RUNNING, /health responds 200 — that is the implicit
|
||||
// health signal the updater's pending-verification timer is waiting for.
|
||||
// Wrapped in try/catch because it must never block startup on a bug here.
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-var-requires
|
||||
const updater = require('./updater');
|
||||
if (typeof updater.markBootHealthy === 'function') updater.markBootHealthy();
|
||||
} catch (err) {
|
||||
logger.debug(`markBootHealthy: ${(err as Error).message}`);
|
||||
}
|
||||
|
||||
// Return the HTTP server to make it easier to write tests.
|
||||
return express.server;
|
||||
};
|
||||
|
|
|
|||
246
src/node/updater/RollbackHandler.ts
Normal file
246
src/node/updater/RollbackHandler.ts
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
import path from 'node:path';
|
||||
import log4js from 'log4js';
|
||||
import {UpdateState} from './types';
|
||||
import type {SpawnFn} from './UpdateExecutor';
|
||||
import {appendLine} from './updateLog';
|
||||
|
||||
const logger = log4js.getLogger('updater');
|
||||
|
||||
export interface RollbackDeps {
|
||||
/** Path of the on-disk Etherpad install (the git working tree). */
|
||||
repoDir: string;
|
||||
/** Where pnpm-lock.yaml was backed up by the executor. */
|
||||
backupDir: string;
|
||||
spawnFn: SpawnFn;
|
||||
copyFile: (src: string, dst: string) => Promise<void>;
|
||||
saveState: (s: UpdateState) => Promise<void>;
|
||||
exit: (code: number) => void;
|
||||
now: () => Date;
|
||||
/** Health-check window after a fresh boot. Default 60s; set via updates.rollbackHealthCheckSeconds. */
|
||||
rollbackHealthCheckSeconds: number;
|
||||
}
|
||||
|
||||
const runStep = (
|
||||
spawnFn: SpawnFn,
|
||||
cwd: string,
|
||||
logPath: string,
|
||||
cmd: string,
|
||||
args: string[],
|
||||
): Promise<number | null> => new Promise((resolve) => {
|
||||
let settled = false;
|
||||
const settle = (c: number | null) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
resolve(c);
|
||||
};
|
||||
const child = spawnFn(cmd, args, {cwd, stdio: ['ignore', 'pipe', 'pipe']});
|
||||
const tag = `${cmd} ${args.join(' ')}`;
|
||||
child.stdout.on('data', (b: Buffer) => {
|
||||
const t = b.toString().trimEnd();
|
||||
logger.info(`[rollback ${tag}] ${t}`);
|
||||
appendLine(logPath, `[${new Date().toISOString()}] rollback ${tag} | ${t}`);
|
||||
});
|
||||
child.stderr.on('data', (b: Buffer) => {
|
||||
const t = b.toString().trimEnd();
|
||||
logger.warn(`[rollback ${tag}] ${t}`);
|
||||
appendLine(logPath, `[${new Date().toISOString()}] rollback ${tag} ERR | ${t}`);
|
||||
});
|
||||
// Spawn failures (binary missing, permissions) — without this listener the
|
||||
// promise hangs forever and the rollback path never lands on terminal state.
|
||||
child.on('error', (err: Error) => {
|
||||
logger.error(`[rollback ${tag}] spawn error: ${err.message}`);
|
||||
appendLine(logPath, `[${new Date().toISOString()}] rollback ${tag} SPAWN_ERR | ${err.message}`);
|
||||
settle(1);
|
||||
});
|
||||
child.on('close', (c) => settle(c));
|
||||
});
|
||||
|
||||
/**
|
||||
* Restore the previous SHA + lockfile and exit 75 so the supervisor restarts.
|
||||
*
|
||||
* Lands on `rolled-back` on success, `rollback-failed` on any sub-step error.
|
||||
* Both paths exit 75 — the supervisor restart is what brings the rolled-back
|
||||
* (or terminal) state up where the admin UI can surface it. Rollback-failed
|
||||
* disables auto/autonomous tiers globally (see UpdatePolicy) until an admin
|
||||
* POSTs /admin/update/acknowledge.
|
||||
*/
|
||||
export const performRollback = async (state: UpdateState, deps: RollbackDeps): Promise<void> => {
|
||||
const exec = state.execution;
|
||||
if (exec.status !== 'rolling-back' && exec.status !== 'pending-verification') {
|
||||
throw new Error(`performRollback called from unexpected status: ${exec.status}`);
|
||||
}
|
||||
const fromSha = (exec as {fromSha: string}).fromSha;
|
||||
const targetTag = (exec as {targetTag: string}).targetTag;
|
||||
const reason = exec.status === 'rolling-back'
|
||||
? exec.reason
|
||||
: 'health-check-failed-or-crash-loop';
|
||||
const logPath = path.join(deps.repoDir, 'var', 'log', 'update.log');
|
||||
|
||||
const failTerminal = async (subReason: string): Promise<void> => {
|
||||
const at = deps.now().toISOString();
|
||||
await deps.saveState({
|
||||
...state,
|
||||
execution: {
|
||||
status: 'rollback-failed',
|
||||
reason: `${reason}; rollback also failed: ${subReason}`,
|
||||
targetTag,
|
||||
fromSha,
|
||||
at,
|
||||
},
|
||||
lastResult: {
|
||||
targetTag,
|
||||
fromSha,
|
||||
outcome: 'rollback-failed',
|
||||
reason: `${reason}; rollback failed: ${subReason}`,
|
||||
at,
|
||||
},
|
||||
bootCount: 0,
|
||||
});
|
||||
logger.error(
|
||||
`rollback FAILED: ${subReason}; manual intervention required ` +
|
||||
'(POST /admin/update/acknowledge after fixing)',
|
||||
);
|
||||
appendLine(logPath, `[${at}] ROLLBACK_FAILED ${subReason}`);
|
||||
deps.exit(75);
|
||||
};
|
||||
|
||||
// Force-checkout first so any partial mutation from the failed executor run
|
||||
// (rewritten lockfile, half-installed modules) is discarded. -f overwrites
|
||||
// tracked files from the target tree's index — without it, `git checkout`
|
||||
// refuses when there are unstaged modifications to files it would replace.
|
||||
const checkoutCode = await runStep(
|
||||
deps.spawnFn, deps.repoDir, logPath, 'git', ['checkout', '-f', fromSha]);
|
||||
if (checkoutCode !== 0) return failTerminal(`git checkout -f ${fromSha} exit ${checkoutCode}`);
|
||||
|
||||
// Now overlay the backed-up lockfile on top. Belt-and-braces: a force
|
||||
// checkout already restored the lockfile to the target SHA's version; the
|
||||
// backup wins on the rare case where the running install had a hand-edited
|
||||
// lockfile we want to preserve.
|
||||
try {
|
||||
await deps.copyFile(
|
||||
path.join(deps.backupDir, 'pnpm-lock.yaml'),
|
||||
path.join(deps.repoDir, 'pnpm-lock.yaml'),
|
||||
);
|
||||
} catch (err: any) {
|
||||
// ENOENT on the backup is acceptable — the force checkout already
|
||||
// restored the right lockfile from the index.
|
||||
if (err?.code !== 'ENOENT') {
|
||||
return failTerminal(`copy lockfile: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
|
||||
const installCode = await runStep(deps.spawnFn, deps.repoDir, logPath, 'pnpm', ['install', '--frozen-lockfile']);
|
||||
if (installCode !== 0) return failTerminal(`pnpm install exit ${installCode}`);
|
||||
|
||||
const at = deps.now().toISOString();
|
||||
await deps.saveState({
|
||||
...state,
|
||||
execution: {status: 'rolled-back', reason, targetTag, restoredSha: fromSha, at},
|
||||
lastResult: {targetTag, fromSha, outcome: 'rolled-back', reason, at},
|
||||
bootCount: 0,
|
||||
});
|
||||
logger.warn(`rolled back to ${fromSha} (reason: ${reason})`);
|
||||
appendLine(logPath, `[${at}] ROLLED_BACK to ${fromSha}; reason=${reason}; exiting 75`);
|
||||
deps.exit(75);
|
||||
};
|
||||
|
||||
export interface CheckResult {
|
||||
/** True if a health-check timer was armed and is awaiting markVerified or expiry. */
|
||||
armed: boolean;
|
||||
/** Cancels the timer and transitions to `verified`. No-op when armed is false. */
|
||||
markVerified: () => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Inspect the persisted execution state at boot and react:
|
||||
* - idle / verified / etc.: no-op.
|
||||
* - pending-verification with bootCount > 2: force rollback (crash-loop guard).
|
||||
* - pending-verification otherwise: increment bootCount, persist, arm a timer.
|
||||
*/
|
||||
export const checkPendingVerification = (state: UpdateState, deps: RollbackDeps): CheckResult => {
|
||||
const exec = state.execution;
|
||||
if (exec.status !== 'pending-verification') return {armed: false, markVerified: () => {}};
|
||||
|
||||
// Fire-and-forget helpers that swallow rejections cleanly. We intentionally
|
||||
// don't propagate — the boot sequence must proceed even if the rollback
|
||||
// path can't write its terminal state. Worst case: the supervisor restart
|
||||
// brings the same boot back up and the bootCount-based crash-loop guard
|
||||
// catches it on the next attempt.
|
||||
const fireRollback = (s: UpdateState) => {
|
||||
void performRollback(s, deps).catch((err) => {
|
||||
logger.error(`performRollback unhandled rejection: ${(err as Error).message}`);
|
||||
// Best-effort: try to land on rollback-failed terminal state and exit
|
||||
// 75 anyway. If saveState also rejects, log and exit so the supervisor
|
||||
// restart at least re-runs checkPendingVerification with bootCount++.
|
||||
const fb = {
|
||||
...s,
|
||||
execution: {
|
||||
status: 'rollback-failed' as const,
|
||||
reason: `unhandled rollback rejection: ${(err as Error).message}`,
|
||||
targetTag: (s.execution as {targetTag?: string}).targetTag ?? '',
|
||||
fromSha: (s.execution as {fromSha?: string}).fromSha ?? '',
|
||||
at: deps.now().toISOString(),
|
||||
},
|
||||
bootCount: 0,
|
||||
};
|
||||
void deps.saveState(fb).catch((saveErr) => {
|
||||
logger.error(`fallback saveState rejected: ${(saveErr as Error).message}`);
|
||||
}).finally(() => deps.exit(75));
|
||||
});
|
||||
};
|
||||
|
||||
const fireSaveState = (s: UpdateState, ctx: string) => {
|
||||
void deps.saveState(s).catch((err) => {
|
||||
logger.warn(`saveState (${ctx}) rejected: ${(err as Error).message}`);
|
||||
});
|
||||
};
|
||||
|
||||
if (state.bootCount > 2) {
|
||||
// Don't await — fire and forget so the boot sequence proceeds; the rollback
|
||||
// path will exit 75 asynchronously and the supervisor restarts on the
|
||||
// restored SHA. Rejections caught + best-effort terminal-state write.
|
||||
fireRollback(state);
|
||||
return {armed: false, markVerified: () => {}};
|
||||
}
|
||||
|
||||
const incremented: UpdateState = {...state, bootCount: state.bootCount + 1};
|
||||
fireSaveState(incremented, 'bootCount-increment');
|
||||
|
||||
let cleared = false;
|
||||
const timer = setTimeout(() => {
|
||||
if (cleared) return;
|
||||
fireRollback({
|
||||
...incremented,
|
||||
execution: {
|
||||
status: 'rolling-back',
|
||||
reason: 'health-check-timeout',
|
||||
targetTag: exec.targetTag,
|
||||
fromSha: exec.fromSha,
|
||||
at: deps.now().toISOString(),
|
||||
},
|
||||
});
|
||||
}, deps.rollbackHealthCheckSeconds * 1000);
|
||||
|
||||
return {
|
||||
armed: true,
|
||||
markVerified: () => {
|
||||
if (cleared) return;
|
||||
cleared = true;
|
||||
clearTimeout(timer);
|
||||
const at = deps.now().toISOString();
|
||||
fireSaveState({
|
||||
...incremented,
|
||||
execution: {status: 'verified', targetTag: exec.targetTag, verifiedAt: at},
|
||||
lastResult: {
|
||||
targetTag: exec.targetTag,
|
||||
fromSha: exec.fromSha,
|
||||
outcome: 'verified',
|
||||
reason: null,
|
||||
at,
|
||||
},
|
||||
bootCount: 0,
|
||||
}, 'mark-verified');
|
||||
logger.info(`update verified after restart: ${exec.fromSha} -> ${exec.targetTag}`);
|
||||
},
|
||||
};
|
||||
};
|
||||
91
src/node/updater/SessionDrainer.ts
Normal file
91
src/node/updater/SessionDrainer.ts
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
/**
|
||||
* Coordinates the pre-restart drain: refuses new pad connections, broadcasts
|
||||
* "system message" announcements at T-60 / T-30 / T-10, and resolves at T=0
|
||||
* so the executor can take over.
|
||||
*
|
||||
* Per docs/superpowers/specs/2026-04-25-auto-update-design.md (section
|
||||
* "Active sessions"). 60s default; configurable via `updates.drainSeconds`.
|
||||
*/
|
||||
|
||||
let acceptingConnections = true;
|
||||
|
||||
export const isAcceptingConnections = (): boolean => acceptingConnections;
|
||||
|
||||
/** Test-only: reset the module-level flag between tests. */
|
||||
export const _resetForTests = (): void => { acceptingConnections = true; };
|
||||
|
||||
export type DrainBroadcastKey =
|
||||
| 'update.drain.t60'
|
||||
| 'update.drain.t30'
|
||||
| 'update.drain.t10';
|
||||
|
||||
export interface DrainerOpts {
|
||||
drainSeconds: number;
|
||||
/** Called for every announcement; values carries timing data the i18n string can interpolate. */
|
||||
broadcast: (i18nKey: DrainBroadcastKey, values: Record<string, unknown>) => void;
|
||||
}
|
||||
|
||||
export interface Drainer {
|
||||
start: () => Promise<{outcome: 'completed' | 'cancelled'}>;
|
||||
cancel: () => void;
|
||||
}
|
||||
|
||||
export const createDrainer = ({drainSeconds, broadcast}: DrainerOpts): Drainer => {
|
||||
const timers: NodeJS.Timeout[] = [];
|
||||
let resolveDone: ((r: {outcome: 'completed' | 'cancelled'}) => void) | null = null;
|
||||
let cancelled = false;
|
||||
let started = false;
|
||||
|
||||
const fire = (key: DrainBroadcastKey, secondsRemaining: number) => {
|
||||
if (cancelled) return;
|
||||
broadcast(key, {seconds: secondsRemaining});
|
||||
};
|
||||
|
||||
const start = (): Promise<{outcome: 'completed' | 'cancelled'}> => {
|
||||
if (started) return Promise.reject(new Error('drainer already started'));
|
||||
started = true;
|
||||
acceptingConnections = false;
|
||||
return new Promise((resolve) => {
|
||||
resolveDone = resolve;
|
||||
const ms = drainSeconds * 1000;
|
||||
// The opening announcement reports the actual drain length rather than a
|
||||
// hardcoded 60, so a configured drainSeconds of e.g. 30 says "30 seconds".
|
||||
// i18n key is still update.drain.t60 — that's the "start of drain" key in
|
||||
// the locale file; the {{seconds}} placeholder carries the real value.
|
||||
fire('update.drain.t60', drainSeconds);
|
||||
// Only schedule T-30 / T-10 when the configured window can actually
|
||||
// honour them. Firing a "30 seconds" message at zero remaining (because
|
||||
// ms - 30_000 < 0) is misleading; admins picking a short drainSeconds
|
||||
// get fewer announcements but each carries an accurate countdown.
|
||||
if (drainSeconds > 30) {
|
||||
timers.push(setTimeout(() => fire('update.drain.t30', 30), ms - 30_000));
|
||||
}
|
||||
if (drainSeconds > 10) {
|
||||
timers.push(setTimeout(() => fire('update.drain.t10', 10), ms - 10_000));
|
||||
}
|
||||
timers.push(setTimeout(() => {
|
||||
if (cancelled) return;
|
||||
// Restore the gate as soon as the drain window closes. The executor
|
||||
// takes over from here and the supervisor restart wipes module state
|
||||
// anyway; if the executor throws and the process keeps running, we
|
||||
// want join handshakes to recover rather than stay wedged.
|
||||
// The lock + state.execution.status guarantee no fresh apply can race.
|
||||
acceptingConnections = true;
|
||||
resolveDone?.({outcome: 'completed'});
|
||||
resolveDone = null;
|
||||
}, ms));
|
||||
});
|
||||
};
|
||||
|
||||
const cancel = (): void => {
|
||||
if (cancelled) return;
|
||||
cancelled = true;
|
||||
for (const t of timers) clearTimeout(t);
|
||||
timers.length = 0;
|
||||
acceptingConnections = true;
|
||||
resolveDone?.({outcome: 'cancelled'});
|
||||
resolveDone = null;
|
||||
};
|
||||
|
||||
return {start, cancel};
|
||||
};
|
||||
219
src/node/updater/UpdateExecutor.ts
Normal file
219
src/node/updater/UpdateExecutor.ts
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
import path from 'node:path';
|
||||
import log4js from 'log4js';
|
||||
import {SpawnOptions} from 'node:child_process';
|
||||
import {UpdateState} from './types';
|
||||
import {appendLine} from './updateLog';
|
||||
import {assertValidTag, refsTagsForm} from './refSafety';
|
||||
|
||||
const logger = log4js.getLogger('updater');
|
||||
|
||||
export interface SpawnedChild {
|
||||
stdout: {on: (event: 'data', cb: (chunk: Buffer) => void) => void};
|
||||
stderr: {on: (event: 'data', cb: (chunk: Buffer) => void) => void};
|
||||
on: {
|
||||
(event: 'close', cb: (code: number | null) => void): void;
|
||||
(event: 'error', cb: (err: Error) => void): void;
|
||||
};
|
||||
}
|
||||
|
||||
export type SpawnFn = (cmd: string, args: string[], opts: SpawnOptions) => SpawnedChild;
|
||||
|
||||
export interface ExecutorDeps {
|
||||
/** Path of the on-disk Etherpad install (the git working tree). */
|
||||
repoDir: string;
|
||||
/** Where pnpm-lock.yaml + sha info gets backed up. */
|
||||
backupDir: string;
|
||||
/** Injected child_process.spawn so tests can drive the pipeline deterministically. */
|
||||
spawnFn: SpawnFn;
|
||||
/** Returns the current HEAD SHA. Production callers wrap `git rev-parse HEAD`. */
|
||||
readSha: () => Promise<string>;
|
||||
/** Plain file copy. Production callers use fs.copyFile (with mkdir-p of parent). */
|
||||
copyFile: (src: string, dst: string) => Promise<void>;
|
||||
/** Persist the in-flight UpdateState. Production callers use saveState(stateFilePath()). */
|
||||
saveState: (s: UpdateState) => Promise<void>;
|
||||
/** State as it was when Apply was clicked — preserves Tier 1 fields (latest, email, etc.). */
|
||||
initialState: UpdateState;
|
||||
/** Tag to update to. */
|
||||
targetTag: string;
|
||||
/** Clock injection for deterministic timestamps in tests. */
|
||||
now: () => Date;
|
||||
/** process.exit injection so tests can assert exit code without actually exiting. */
|
||||
exit: (code: number) => void;
|
||||
}
|
||||
|
||||
export type ExecutorResult =
|
||||
| {outcome: 'pending-verification'}
|
||||
| {outcome: 'failed-install'; reason: string}
|
||||
| {outcome: 'failed-build'; reason: string}
|
||||
| {outcome: 'failed-checkout'; reason: string};
|
||||
|
||||
const runStep = (
|
||||
spawnFn: SpawnFn,
|
||||
repoDir: string,
|
||||
logPath: string,
|
||||
cmd: string,
|
||||
args: string[],
|
||||
): Promise<{code: number | null; stderr: string}> => new Promise((resolve) => {
|
||||
let stderr = '';
|
||||
let settled = false;
|
||||
const settle = (v: {code: number | null; stderr: string}) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
resolve(v);
|
||||
};
|
||||
const child = spawnFn(cmd, args, {cwd: repoDir, stdio: ['ignore', 'pipe', 'pipe']});
|
||||
const tag = `${cmd} ${args.join(' ')}`;
|
||||
child.stdout.on('data', (chunk: Buffer) => {
|
||||
const txt = chunk.toString().trimEnd();
|
||||
logger.info(`[${tag}] ${txt}`);
|
||||
appendLine(logPath, `[${new Date().toISOString()}] ${tag} | ${txt}`);
|
||||
});
|
||||
child.stderr.on('data', (chunk: Buffer) => {
|
||||
const txt = chunk.toString();
|
||||
stderr += txt;
|
||||
const trimmed = txt.trimEnd();
|
||||
logger.warn(`[${tag}] ${trimmed}`);
|
||||
appendLine(logPath, `[${new Date().toISOString()}] ${tag} ERR | ${trimmed}`);
|
||||
});
|
||||
// Spawn failures (binary missing, permissions) emit 'error' and never close.
|
||||
// Without this listener the promise hangs forever and leaves state in-flight.
|
||||
// Treat as exit code 1 with the error message in stderr so the caller's
|
||||
// failure-detection branch fires normally.
|
||||
child.on('error', (err: Error) => {
|
||||
logger.error(`[${tag}] spawn error: ${err.message}`);
|
||||
appendLine(logPath, `[${new Date().toISOString()}] ${tag} SPAWN_ERR | ${err.message}`);
|
||||
settle({code: 1, stderr: stderr + err.message});
|
||||
});
|
||||
child.on('close', (code) => settle({code, stderr}));
|
||||
});
|
||||
|
||||
/**
|
||||
* Run the update pipeline. Each transition writes state before/after so a hard
|
||||
* kill mid-step lands the next boot in a known state for RollbackHandler.
|
||||
*
|
||||
* On install/build/checkout failure the executor transitions to `rolling-back`,
|
||||
* persists, and returns. The route layer then runs RollbackHandler.performRollback.
|
||||
* The executor does NOT call `exit` on failure paths — the rollback path owns
|
||||
* that exit so we don't double-exit and lose log lines.
|
||||
*
|
||||
* On a thrown exception (e.g., copyFile EACCES, saveState ENOSPC) the executor
|
||||
* also transitions to rolling-back with `failed-checkout` so the route's post-
|
||||
* executor rollback path picks it up. The state must never get stuck at
|
||||
* `executing` — if it does, no further updates can start until an admin
|
||||
* acknowledges.
|
||||
*/
|
||||
export const executeUpdate = async (deps: ExecutorDeps): Promise<ExecutorResult> => {
|
||||
const logPath = path.join(deps.repoDir, 'var', 'log', 'update.log');
|
||||
let fromSha = '';
|
||||
|
||||
// Wrap the whole body so any throw — readSha, saveState, copyFile, even an
|
||||
// unexpected synchronous error in a step — lands us at rolling-back rather
|
||||
// than leaving execution stuck at 'executing' forever.
|
||||
try {
|
||||
// Reject unsafe release-tag strings (option injection guard).
|
||||
// Tag is sourced from GitHub's tag_name and persisted into update-state.json;
|
||||
// a tag starting with '-' would otherwise be parsed by git as an option flag.
|
||||
const safeTag = assertValidTag(deps.targetTag);
|
||||
fromSha = await deps.readSha();
|
||||
|
||||
let s: UpdateState = {
|
||||
...deps.initialState,
|
||||
execution: {
|
||||
status: 'executing',
|
||||
targetTag: deps.targetTag,
|
||||
fromSha,
|
||||
startedAt: deps.now().toISOString(),
|
||||
},
|
||||
bootCount: 0,
|
||||
};
|
||||
await deps.saveState(s);
|
||||
|
||||
// Snapshot lockfile (SHA already captured above; the rollback handler reads
|
||||
// execution.fromSha rather than a separate file so a successful rollback
|
||||
// doesn't depend on /var staying writable past this point).
|
||||
await deps.copyFile(
|
||||
path.join(deps.repoDir, 'pnpm-lock.yaml'),
|
||||
path.join(deps.backupDir, 'pnpm-lock.yaml'),
|
||||
);
|
||||
|
||||
const fail = async (
|
||||
outcome: 'failed-install' | 'failed-build' | 'failed-checkout',
|
||||
reason: string,
|
||||
): Promise<ExecutorResult> => {
|
||||
s = {
|
||||
...s,
|
||||
execution: {
|
||||
status: 'rolling-back',
|
||||
reason,
|
||||
targetTag: deps.targetTag,
|
||||
fromSha,
|
||||
at: deps.now().toISOString(),
|
||||
},
|
||||
};
|
||||
await deps.saveState(s);
|
||||
logger.error(`update step failed (${outcome}): ${reason}`);
|
||||
appendLine(logPath, `[${deps.now().toISOString()}] FAIL ${outcome}: ${reason}`);
|
||||
return {outcome, reason};
|
||||
};
|
||||
|
||||
let r = await runStep(deps.spawnFn, deps.repoDir, logPath, 'git', ['fetch', '--tags', 'origin']);
|
||||
if (r.code !== 0) return fail('failed-checkout', `git fetch exit ${r.code}: ${r.stderr.trim()}`);
|
||||
|
||||
// Use the refs/tags/<tag> form so even an unforeseen edge-case in the tag
|
||||
// string can't be parsed as a git option. assertValidTag above already
|
||||
// rules out leading '-' / whitespace / shell metacharacters.
|
||||
r = await runStep(
|
||||
deps.spawnFn, deps.repoDir, logPath, 'git', ['checkout', refsTagsForm(safeTag)]);
|
||||
if (r.code !== 0) return fail('failed-checkout', `git checkout exit ${r.code}: ${r.stderr.trim()}`);
|
||||
|
||||
r = await runStep(deps.spawnFn, deps.repoDir, logPath, 'pnpm', ['install', '--frozen-lockfile']);
|
||||
if (r.code !== 0) return fail('failed-install', `pnpm install exit ${r.code}: ${r.stderr.trim()}`);
|
||||
|
||||
r = await runStep(deps.spawnFn, deps.repoDir, logPath, 'pnpm', ['run', 'build:ui']);
|
||||
if (r.code !== 0) return fail('failed-build', `pnpm run build:ui exit ${r.code}: ${r.stderr.trim()}`);
|
||||
|
||||
// pending-verification: the next boot's RollbackHandler arms the health-check timer.
|
||||
s = {
|
||||
...s,
|
||||
execution: {
|
||||
status: 'pending-verification',
|
||||
targetTag: deps.targetTag,
|
||||
fromSha,
|
||||
// Real deadline is computed at next boot using rollbackHealthCheckSeconds.
|
||||
// We persist a placeholder here purely so the field is present.
|
||||
deadlineAt: deps.now().toISOString(),
|
||||
},
|
||||
bootCount: 0,
|
||||
};
|
||||
await deps.saveState(s);
|
||||
logger.info(`update executed: ${fromSha} -> ${deps.targetTag}; exiting 75 for supervisor restart`);
|
||||
void appendLine(logPath, `[${deps.now().toISOString()}] OK pending-verification ${fromSha} -> ${deps.targetTag}; exiting 75`);
|
||||
deps.exit(75);
|
||||
return {outcome: 'pending-verification'};
|
||||
} catch (err) {
|
||||
// Unexpected throw — fs ENOSPC, EACCES on the backup dir, network blip
|
||||
// surfaced through readSha, etc. Persist rolling-back so the route's
|
||||
// post-executor rollback path runs and the state never wedges at 'executing'.
|
||||
const reason = `executor exception: ${(err as Error).message}`;
|
||||
logger.error(reason);
|
||||
void appendLine(logPath, `[${deps.now().toISOString()}] EXECUTOR_THROW ${reason}`);
|
||||
try {
|
||||
await deps.saveState({
|
||||
...deps.initialState,
|
||||
execution: {
|
||||
status: 'rolling-back',
|
||||
reason,
|
||||
targetTag: deps.targetTag,
|
||||
fromSha,
|
||||
at: deps.now().toISOString(),
|
||||
},
|
||||
bootCount: 0,
|
||||
});
|
||||
} catch (saveErr) {
|
||||
// Even saveState threw. Best-effort log, rethrow original — the route's
|
||||
// catch will surface it. State on disk is whatever last successfully wrote.
|
||||
logger.error(`could not persist rolling-back: ${(saveErr as Error).message}`);
|
||||
}
|
||||
return {outcome: 'failed-checkout', reason};
|
||||
}
|
||||
};
|
||||
|
|
@ -10,14 +10,27 @@ export interface PolicyInput {
|
|||
tier: Tier;
|
||||
current: string;
|
||||
latest: string;
|
||||
/**
|
||||
* Optional execution-status hint. Only `rollback-failed` materially changes
|
||||
* policy: while it's set, canAuto / canAutonomous are denied (an admin must
|
||||
* acknowledge first). canManual stays on because clicking Apply *is* the
|
||||
* intervention the terminal state requires.
|
||||
*/
|
||||
executionStatus?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide which update tiers are allowed under the given (installMethod, tier, current, latest).
|
||||
* Pure function — no I/O. The single source of truth for "what's allowed in this environment."
|
||||
* `reason` is one of: 'tier-off' | 'up-to-date' | 'install-method-not-writable' | 'ok'.
|
||||
* Decide which update tiers are allowed under the given (installMethod, tier,
|
||||
* current, latest, executionStatus). Pure function — no I/O. The single source
|
||||
* of truth for "what's allowed in this environment."
|
||||
*
|
||||
* `reason` is one of:
|
||||
* 'tier-off' | 'up-to-date' | 'install-method-not-writable'
|
||||
* | 'rollback-failed-terminal' | 'ok'.
|
||||
*/
|
||||
export const evaluatePolicy = ({installMethod, tier, current, latest}: PolicyInput): PolicyResult => {
|
||||
export const evaluatePolicy = ({
|
||||
installMethod, tier, current, latest, executionStatus,
|
||||
}: PolicyInput): PolicyResult => {
|
||||
if (tier === 'off') {
|
||||
return {canNotify: false, canManual: false, canAuto: false, canAutonomous: false, reason: 'tier-off'};
|
||||
}
|
||||
|
|
@ -32,11 +45,12 @@ export const evaluatePolicy = ({installMethod, tier, current, latest}: PolicyInp
|
|||
return {canNotify, canManual: false, canAuto: false, canAutonomous: false, reason: 'install-method-not-writable'};
|
||||
}
|
||||
|
||||
const terminal = executionStatus === 'rollback-failed';
|
||||
return {
|
||||
canNotify,
|
||||
canManual: tier === 'manual' || tier === 'auto' || tier === 'autonomous',
|
||||
canAuto: tier === 'auto' || tier === 'autonomous',
|
||||
canAutonomous: tier === 'autonomous',
|
||||
reason: 'ok',
|
||||
canAuto: !terminal && (tier === 'auto' || tier === 'autonomous'),
|
||||
canAutonomous: !terminal && tier === 'autonomous',
|
||||
reason: terminal ? 'rollback-failed-terminal' : 'ok',
|
||||
};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import {ReleaseInfo, VulnerableBelowDirective} from './types';
|
||||
import {parseVulnerableBelow} from './versionCompare';
|
||||
import {isValidTag} from './refSafety';
|
||||
|
||||
export interface FetchResult {
|
||||
status: number;
|
||||
|
|
@ -49,6 +50,15 @@ export const checkLatestRelease = async (
|
|||
return {kind: 'error', status: 200};
|
||||
}
|
||||
|
||||
// Reject any tag that would be unsafe to hand to git later. Validating at
|
||||
// the persistence boundary (rather than only at the executor) means a
|
||||
// malformed tag_name from a misconfigured fork-as-github-repo never lands
|
||||
// in update-state.json. Treated as a fetch error so the polling loop will
|
||||
// try again next interval.
|
||||
if (!isValidTag(j.tag_name)) {
|
||||
return {kind: 'error', status: 200};
|
||||
}
|
||||
|
||||
const tag = j.tag_name;
|
||||
const version = tag.replace(/^v/, '');
|
||||
const body: string = typeof j.body === 'string' ? j.body : '';
|
||||
|
|
|
|||
|
|
@ -1,4 +1,6 @@
|
|||
import path from 'node:path';
|
||||
import {spawn} from 'node:child_process';
|
||||
import fs from 'node:fs/promises';
|
||||
import log4js from 'log4js';
|
||||
import settings, {getEpVersion} from '../utils/Settings';
|
||||
import {detectInstallMethod} from './InstallMethodDetector';
|
||||
|
|
@ -7,6 +9,8 @@ import {loadState, saveState} from './state';
|
|||
import {isMajorBehind, isVulnerable} from './versionCompare';
|
||||
import {evaluatePolicy} from './UpdatePolicy';
|
||||
import {decideEmails} from './Notifier';
|
||||
import {checkPendingVerification, CheckResult, RollbackDeps} from './RollbackHandler';
|
||||
import type {SpawnFn} from './UpdateExecutor';
|
||||
import {InstallMethod, UpdateState} from './types';
|
||||
|
||||
const logger = log4js.getLogger('updater');
|
||||
|
|
@ -16,6 +20,7 @@ let timer: NodeJS.Timeout | null = null;
|
|||
let initialTimer: NodeJS.Timeout | null = null;
|
||||
let checkInFlight = false;
|
||||
let inMemoryState: UpdateState | null = null;
|
||||
let pendingVerification: CheckResult | null = null;
|
||||
|
||||
export const stateFilePath = () => path.join(settings.root, 'var', 'update-state.json');
|
||||
|
||||
|
|
@ -126,6 +131,21 @@ const startPolling = (): void => {
|
|||
initialTimer = setTimeout(() => { initialTimer = null; void performCheck(); }, 5000);
|
||||
};
|
||||
|
||||
/** Build the dependency bundle RollbackHandler / UpdateExecutor expect. */
|
||||
export const getRollbackDeps = (): RollbackDeps => ({
|
||||
repoDir: settings.root,
|
||||
backupDir: path.join(settings.root, 'var', 'update-backup'),
|
||||
spawnFn: spawn as unknown as SpawnFn,
|
||||
copyFile: async (src: string, dst: string) => {
|
||||
await fs.mkdir(path.dirname(dst), {recursive: true});
|
||||
await fs.copyFile(src, dst);
|
||||
},
|
||||
saveState: (s: UpdateState) => saveState(stateFilePath(), s),
|
||||
exit: (code: number) => process.exit(code),
|
||||
now: () => new Date(),
|
||||
rollbackHealthCheckSeconds: Number(settings.updates.rollbackHealthCheckSeconds) || 60,
|
||||
});
|
||||
|
||||
/** Hook entry point — called by ep.json on createServer. */
|
||||
export const expressCreateServer = async (): Promise<void> => {
|
||||
detectedMethod = await detectInstallMethod({
|
||||
|
|
@ -133,9 +153,29 @@ export const expressCreateServer = async (): Promise<void> => {
|
|||
rootDir: settings.root,
|
||||
});
|
||||
logger.info(`updater: install method = ${detectedMethod}, tier = ${settings.updates.tier}`);
|
||||
|
||||
// Tier 2: if the previous boot left the state in pending-verification, arm
|
||||
// the health-check timer (or force rollback when bootCount has climbed past
|
||||
// the crash-loop threshold). This must run BEFORE polling starts so the
|
||||
// rollback can fire even if the version checker is misconfigured.
|
||||
const state = await getCurrentState();
|
||||
pendingVerification = checkPendingVerification(state, getRollbackDeps());
|
||||
|
||||
if (settings.updates.tier !== 'off') startPolling();
|
||||
};
|
||||
|
||||
/**
|
||||
* Called by the Etherpad runtime once the express stack is fully wired and
|
||||
* /health responds — that's the implicit health signal the
|
||||
* pending-verification timer is waiting for.
|
||||
*/
|
||||
export const markBootHealthy = (): void => {
|
||||
if (pendingVerification) {
|
||||
pendingVerification.markVerified();
|
||||
pendingVerification = null;
|
||||
}
|
||||
};
|
||||
|
||||
/** Shutdown hook. */
|
||||
export const shutdown = async (): Promise<void> => {
|
||||
if (timer) { clearInterval(timer); timer = null; }
|
||||
|
|
|
|||
78
src/node/updater/lock.ts
Normal file
78
src/node/updater/lock.ts
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
import fs from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
|
||||
interface LockFile {
|
||||
pid: number;
|
||||
at: string;
|
||||
}
|
||||
|
||||
const isPidLive = (pid: number): boolean => {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (err: any) {
|
||||
// ESRCH = no such process (stale).
|
||||
// EPERM = exists but we can't signal — treat as live (some other user owns it).
|
||||
return err.code !== 'ESRCH';
|
||||
}
|
||||
};
|
||||
|
||||
const readIfPresent = async (lockPath: string): Promise<LockFile | null> => {
|
||||
let raw: string;
|
||||
try {
|
||||
raw = await fs.readFile(lockPath, 'utf8');
|
||||
} catch (err: any) {
|
||||
if (err.code === 'ENOENT') return null;
|
||||
return null;
|
||||
}
|
||||
let parsed: unknown;
|
||||
try { parsed = JSON.parse(raw); } catch { return null; }
|
||||
if (!parsed || typeof parsed !== 'object') return null;
|
||||
const p = parsed as Record<string, unknown>;
|
||||
if (typeof p.pid !== 'number' || typeof p.at !== 'string') return null;
|
||||
return {pid: p.pid, at: p.at};
|
||||
};
|
||||
|
||||
/**
|
||||
* Atomic acquire via O_CREAT|O_EXCL. If the file already exists, the holder's
|
||||
* PID is checked; when dead we reap it and retry once. Returns false on a live
|
||||
* conflict — the caller is expected to surface "lock-held" to the admin.
|
||||
*/
|
||||
export const acquireLock = async (lockPath: string): Promise<boolean> => {
|
||||
await fs.mkdir(path.dirname(lockPath), {recursive: true});
|
||||
const payload = JSON.stringify({pid: process.pid, at: new Date().toISOString()});
|
||||
|
||||
const tryCreate = async (): Promise<boolean> => {
|
||||
try {
|
||||
const fh = await fs.open(lockPath, 'wx');
|
||||
try { await fh.writeFile(payload); } finally { await fh.close(); }
|
||||
return true;
|
||||
} catch (err: any) {
|
||||
if (err.code === 'EEXIST') return false;
|
||||
throw err;
|
||||
}
|
||||
};
|
||||
|
||||
if (await tryCreate()) return true;
|
||||
|
||||
const existing = await readIfPresent(lockPath);
|
||||
if (existing && isPidLive(existing.pid)) return false;
|
||||
|
||||
// Stale or unparseable — reap and retry once. A concurrent reaper may beat us,
|
||||
// in which case the second tryCreate also returns false (correctly: someone
|
||||
// else holds it now).
|
||||
try { await fs.unlink(lockPath); }
|
||||
catch (err: any) { if (err.code !== 'ENOENT') throw err; }
|
||||
return tryCreate();
|
||||
};
|
||||
|
||||
export const releaseLock = async (lockPath: string): Promise<void> => {
|
||||
try { await fs.unlink(lockPath); }
|
||||
catch (err: any) { if (err.code !== 'ENOENT') throw err; }
|
||||
};
|
||||
|
||||
/** True iff the lock file exists *and* the recorded PID is live. Stale locks read as not-held. */
|
||||
export const isHeld = async (lockPath: string): Promise<boolean> => {
|
||||
const f = await readIfPresent(lockPath);
|
||||
return !!f && isPidLive(f.pid);
|
||||
};
|
||||
54
src/node/updater/preflight.ts
Normal file
54
src/node/updater/preflight.ts
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
import {InstallMethod} from './types';
|
||||
import type {VerifyResult} from './trustedKeys';
|
||||
|
||||
export type PreflightReason =
|
||||
| 'install-method-not-writable'
|
||||
| 'dirty-working-tree'
|
||||
| 'low-disk-space'
|
||||
| 'pnpm-not-found'
|
||||
| 'lock-held'
|
||||
| 'remote-tag-missing'
|
||||
| 'signature-verification-failed';
|
||||
|
||||
export interface PreflightInput {
|
||||
targetTag: string;
|
||||
diskSpaceMinMB: number;
|
||||
requireSignature: boolean;
|
||||
trustedKeysPath: string | null;
|
||||
}
|
||||
|
||||
export interface PreflightDeps {
|
||||
installMethod: Exclude<InstallMethod, 'auto'>;
|
||||
workingTreeClean: () => Promise<boolean>;
|
||||
freeDiskMB: () => Promise<number>;
|
||||
pnpmOnPath: () => Promise<boolean>;
|
||||
lockHeld: () => Promise<boolean>;
|
||||
remoteHasTag: (tag: string) => Promise<boolean>;
|
||||
verifyTag: () => Promise<VerifyResult>;
|
||||
}
|
||||
|
||||
export type PreflightResult = {ok: true} | {ok: false; reason: PreflightReason};
|
||||
|
||||
const WRITABLE_METHODS: ReadonlySet<Exclude<InstallMethod, 'auto'>> = new Set(['git']);
|
||||
|
||||
/**
|
||||
* Sequenced preflight: each check is fast and reads the world. Order matters —
|
||||
* cheap, definitive failures (install method) run before slow ones (network
|
||||
* tag lookup, gpg). The first failure short-circuits.
|
||||
*/
|
||||
export const runPreflight = async (
|
||||
input: PreflightInput,
|
||||
deps: PreflightDeps,
|
||||
): Promise<PreflightResult> => {
|
||||
if (!WRITABLE_METHODS.has(deps.installMethod)) {
|
||||
return {ok: false, reason: 'install-method-not-writable'};
|
||||
}
|
||||
if (!await deps.workingTreeClean()) return {ok: false, reason: 'dirty-working-tree'};
|
||||
if ((await deps.freeDiskMB()) < input.diskSpaceMinMB) return {ok: false, reason: 'low-disk-space'};
|
||||
if (!await deps.pnpmOnPath()) return {ok: false, reason: 'pnpm-not-found'};
|
||||
if (await deps.lockHeld()) return {ok: false, reason: 'lock-held'};
|
||||
if (!await deps.remoteHasTag(input.targetTag)) return {ok: false, reason: 'remote-tag-missing'};
|
||||
const sig = await deps.verifyTag();
|
||||
if (!sig.ok) return {ok: false, reason: 'signature-verification-failed'};
|
||||
return {ok: true};
|
||||
};
|
||||
43
src/node/updater/refSafety.ts
Normal file
43
src/node/updater/refSafety.ts
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
/**
|
||||
* Safety helpers for any release-tag string Etherpad's updater hands to git.
|
||||
*
|
||||
* The release tag originates from GitHub's `releases/latest` API (`tag_name`)
|
||||
* and is then persisted into `var/update-state.json`. A tag that starts with
|
||||
* `-` would be parsed by git as an option flag rather than a positional ref —
|
||||
* `git checkout -fast-forward` and similar tricks could bypass signature
|
||||
* verification or change checkout semantics. A tag with shell metacharacters
|
||||
* is less of an issue under `child_process.spawn` (no shell), but we reject
|
||||
* those too because git's own ref-name rules forbid them and a malformed tag
|
||||
* has nowhere reasonable to be honoured anyway.
|
||||
*
|
||||
* Rules (a subset of git's check-ref-format spec — strict on purpose):
|
||||
* - Non-empty.
|
||||
* - Length <= 200.
|
||||
* - May not start with `-` (option injection) or `.` (git rejects).
|
||||
* - May not contain whitespace, NUL, or any of: ~ ^ : ? * [ \\
|
||||
* - May not contain `..` (git's own rule).
|
||||
*
|
||||
* Callers should also use the `refs/tags/<tag>` form when invoking git so
|
||||
* that even an unforeseen edge-case can't be parsed as an option, and pass
|
||||
* `--` as an end-of-options marker on commands that accept it.
|
||||
*/
|
||||
|
||||
const FORBIDDEN_CHARS = /[\s\x00~^:?*\[\\]/;
|
||||
|
||||
export const isValidTag = (tag: unknown): tag is string => {
|
||||
if (typeof tag !== 'string') return false;
|
||||
if (tag.length === 0 || tag.length > 200) return false;
|
||||
if (tag.startsWith('-') || tag.startsWith('.')) return false;
|
||||
if (FORBIDDEN_CHARS.test(tag)) return false;
|
||||
if (tag.includes('..')) return false;
|
||||
return true;
|
||||
};
|
||||
|
||||
/** Throwing form for hot paths where invalid input is a programmer/data error. */
|
||||
export const assertValidTag = (tag: unknown): string => {
|
||||
if (!isValidTag(tag)) throw new Error(`unsafe release tag: ${JSON.stringify(tag)}`);
|
||||
return tag as string;
|
||||
};
|
||||
|
||||
/** Wrap a validated tag in the `refs/tags/<tag>` form for git invocations. */
|
||||
export const refsTagsForm = (tag: string): string => `refs/tags/${tag}`;
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
import fs from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
import {EMPTY_STATE, UpdateState} from './types';
|
||||
import {EMPTY_STATE, EXECUTION_STATUSES, UpdateState} from './types';
|
||||
|
||||
const isPlainObject = (v: unknown): v is Record<string, unknown> =>
|
||||
v !== null && typeof v === 'object' && !Array.isArray(v);
|
||||
|
|
@ -8,6 +8,52 @@ const isPlainObject = (v: unknown): v is Record<string, unknown> =>
|
|||
const isStringOrNull = (v: unknown): v is string | null =>
|
||||
v === null || typeof v === 'string';
|
||||
|
||||
// Per-status field requirements that mirror the ExecutionStatus union in types.ts.
|
||||
// Persisted-state corruption (a hand-edited file or a future schema bump that
|
||||
// missed a migration) must never reach RollbackHandler with `undefined` refs —
|
||||
// loadState resets to EMPTY_STATE when any required field is missing.
|
||||
const EXEC_REQUIRED_FIELDS: Record<string, readonly string[]> = {
|
||||
'idle': [],
|
||||
'preflight': ['targetTag', 'startedAt'],
|
||||
'preflight-failed': ['targetTag', 'reason', 'at'],
|
||||
'draining': ['targetTag', 'drainEndsAt', 'startedAt'],
|
||||
'executing': ['targetTag', 'fromSha', 'startedAt'],
|
||||
'pending-verification': ['targetTag', 'fromSha', 'deadlineAt'],
|
||||
'verified': ['targetTag', 'verifiedAt'],
|
||||
'rolling-back': ['reason', 'targetTag', 'fromSha', 'at'],
|
||||
'rolled-back': ['reason', 'targetTag', 'restoredSha', 'at'],
|
||||
'rollback-failed': ['reason', 'targetTag', 'fromSha', 'at'],
|
||||
};
|
||||
|
||||
const isValidExecution = (v: unknown): boolean => {
|
||||
if (!isPlainObject(v)) return false;
|
||||
if (typeof v.status !== 'string') return false;
|
||||
if (!(EXECUTION_STATUSES as readonly string[]).includes(v.status)) return false;
|
||||
const required = EXEC_REQUIRED_FIELDS[v.status];
|
||||
if (!required) return false; // unknown status — fail closed
|
||||
for (const field of required) {
|
||||
if (typeof (v as Record<string, unknown>)[field] !== 'string') return false;
|
||||
if (((v as Record<string, unknown>)[field] as string).length === 0) return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
|
||||
// Outcomes that LastUpdateResult.outcome must match.
|
||||
const VALID_OUTCOMES: ReadonlySet<string> = new Set([
|
||||
'verified', 'rolled-back', 'rollback-failed', 'preflight-failed', 'cancelled',
|
||||
]);
|
||||
|
||||
const isValidLastResult = (v: unknown): boolean => {
|
||||
if (v === null) return true;
|
||||
if (!isPlainObject(v)) return false;
|
||||
return typeof v.targetTag === 'string'
|
||||
&& typeof v.fromSha === 'string'
|
||||
&& typeof v.outcome === 'string'
|
||||
&& VALID_OUTCOMES.has(v.outcome)
|
||||
&& (v.reason === null || typeof v.reason === 'string')
|
||||
&& typeof v.at === 'string';
|
||||
};
|
||||
|
||||
const isValidLatest = (v: unknown): boolean => {
|
||||
if (v === null) return true;
|
||||
if (!isPlainObject(v)) return false;
|
||||
|
|
@ -39,14 +85,23 @@ const isValidEmail = (v: unknown): boolean => {
|
|||
// Validate the full shape so loadState() actually delivers on its "safely
|
||||
// reset on malformed input" contract. Downstream code calls .trim() / semver
|
||||
// parsing on these subfields and would crash on a hand-edited file otherwise.
|
||||
const isValid = (raw: unknown): raw is UpdateState => {
|
||||
//
|
||||
// Tier 2 fields (execution, bootCount, lastResult) MAY be absent on a state
|
||||
// file written by a Tier 1 install — those are backfilled at load time.
|
||||
// Present-but-malformed values still reject so a hand-edited file with
|
||||
// e.g. execution.status="totally-bogus" can't poison RollbackHandler.
|
||||
const isValid = (raw: unknown): raw is Partial<UpdateState> & object => {
|
||||
if (!isPlainObject(raw)) return false;
|
||||
return raw.schemaVersion === 1
|
||||
&& isStringOrNull(raw.lastCheckAt)
|
||||
&& isStringOrNull(raw.lastEtag)
|
||||
&& isValidLatest(raw.latest)
|
||||
&& isValidVulnerableBelow(raw.vulnerableBelow)
|
||||
&& isValidEmail(raw.email);
|
||||
if (raw.schemaVersion !== 1) return false;
|
||||
if (!isStringOrNull(raw.lastCheckAt)) return false;
|
||||
if (!isStringOrNull(raw.lastEtag)) return false;
|
||||
if (!isValidLatest(raw.latest)) return false;
|
||||
if (!isValidVulnerableBelow(raw.vulnerableBelow)) return false;
|
||||
if (!isValidEmail(raw.email)) return false;
|
||||
if (raw.execution !== undefined && !isValidExecution(raw.execution)) return false;
|
||||
if (raw.bootCount !== undefined && typeof raw.bootCount !== 'number') return false;
|
||||
if (raw.lastResult !== undefined && !isValidLastResult(raw.lastResult)) return false;
|
||||
return true;
|
||||
};
|
||||
|
||||
/** Reads the on-disk state. Returns a fresh empty-state clone when the file is missing, malformed, or has an unknown schemaVersion. Never throws on parse errors. */
|
||||
|
|
@ -65,7 +120,17 @@ export const loadState = async (filePath: string): Promise<UpdateState> => {
|
|||
return structuredClone(EMPTY_STATE);
|
||||
}
|
||||
if (!isValid(parsed)) return structuredClone(EMPTY_STATE);
|
||||
return parsed;
|
||||
// Backfill Tier 2 fields on a Tier 1 state file. Spread defaults first,
|
||||
// parsed second so explicit values win, then explicit fallback for the
|
||||
// three fields that might be undefined.
|
||||
const partial = parsed as Partial<UpdateState>;
|
||||
return {
|
||||
...structuredClone(EMPTY_STATE),
|
||||
...partial,
|
||||
execution: partial.execution ?? structuredClone(EMPTY_STATE.execution),
|
||||
bootCount: partial.bootCount ?? 0,
|
||||
lastResult: partial.lastResult ?? null,
|
||||
} as UpdateState;
|
||||
};
|
||||
|
||||
/** Atomic write via tmp-then-rename. Creates parent directories as needed. */
|
||||
|
|
|
|||
75
src/node/updater/trustedKeys.ts
Normal file
75
src/node/updater/trustedKeys.ts
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
import {spawn as realSpawn, SpawnOptions} from 'node:child_process';
|
||||
import log4js from 'log4js';
|
||||
import {isValidTag} from './refSafety';
|
||||
|
||||
const logger = log4js.getLogger('updater');
|
||||
|
||||
export type SpawnFn = (cmd: string, args: string[], opts: SpawnOptions) => {
|
||||
on: {
|
||||
(event: 'close', cb: (code: number | null) => void): void;
|
||||
(event: 'error', cb: (err: Error) => void): void;
|
||||
};
|
||||
};
|
||||
|
||||
export interface VerifyArgs {
|
||||
tag: string;
|
||||
repoDir: string;
|
||||
requireSignature: boolean;
|
||||
trustedKeysPath: string | null;
|
||||
/** Override for tests; production callers use the default `child_process.spawn`. */
|
||||
spawnFn?: SpawnFn;
|
||||
}
|
||||
|
||||
export type VerifyResult =
|
||||
| {ok: true; reason: 'signature-verified' | 'signature-not-required'}
|
||||
| {ok: false; reason: 'signature-verification-failed'};
|
||||
|
||||
/**
|
||||
* Verify a release tag's GPG signature via `git verify-tag <tag>`.
|
||||
*
|
||||
* With `requireSignature: false` (default) this is a documented no-op:
|
||||
* Etherpad's release process does not yet sign tags consistently, and
|
||||
* forcing verification on by default would break Tier 2 for everyone.
|
||||
* Admins who run their own builds or who pin to signed forks set
|
||||
* `updates.requireSignature: true` and import the trusted keys into the
|
||||
* Etherpad user's keyring (or a dedicated keyring at
|
||||
* `updates.trustedKeysPath`, which is passed to git via $GNUPGHOME).
|
||||
*/
|
||||
export const verifyReleaseTag = async (args: VerifyArgs): Promise<VerifyResult> => {
|
||||
if (!args.requireSignature) {
|
||||
logger.warn(
|
||||
`verifyReleaseTag: signature check skipped (updates.requireSignature=false) for ${args.tag}`,
|
||||
);
|
||||
return {ok: true, reason: 'signature-not-required'};
|
||||
}
|
||||
// Reject unsafe tag strings before they ever reach git. A tag starting with
|
||||
// '-' could otherwise be parsed as a git option, bypassing verification.
|
||||
if (!isValidTag(args.tag)) {
|
||||
logger.error(`verifyReleaseTag: refused unsafe tag ${JSON.stringify(args.tag)}`);
|
||||
return {ok: false, reason: 'signature-verification-failed'};
|
||||
}
|
||||
const spawnFn = args.spawnFn ?? (realSpawn as unknown as SpawnFn);
|
||||
const env: NodeJS.ProcessEnv = {...process.env};
|
||||
if (args.trustedKeysPath) env.GNUPGHOME = args.trustedKeysPath;
|
||||
// -- terminates options so even a future tag-validation regression can't
|
||||
// smuggle a flag past git verify-tag.
|
||||
const child = spawnFn('git', ['verify-tag', '--', args.tag], {
|
||||
cwd: args.repoDir,
|
||||
env,
|
||||
stdio: 'ignore',
|
||||
});
|
||||
// Listen for both 'close' and 'error' so a missing/unexecutable git binary
|
||||
// surfaces as verification-failure rather than a hung promise.
|
||||
const code: number | null = await new Promise((resolve) => {
|
||||
let settled = false;
|
||||
const settle = (c: number | null) => { if (settled) return; settled = true; resolve(c); };
|
||||
child.on('close', settle);
|
||||
child.on('error', (err: Error) => {
|
||||
logger.error(`verifyReleaseTag: git verify-tag spawn error: ${err.message}`);
|
||||
settle(1);
|
||||
});
|
||||
});
|
||||
if (code === 0) return {ok: true, reason: 'signature-verified'};
|
||||
logger.error(`verifyReleaseTag: git verify-tag ${args.tag} exited ${code}`);
|
||||
return {ok: false, reason: 'signature-verification-failed'};
|
||||
};
|
||||
|
|
@ -45,6 +45,45 @@ export interface EmailSendLog {
|
|||
vulnerableNewReleaseTag: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Discriminated union mirroring the state machine in
|
||||
* docs/superpowers/specs/2026-04-25-auto-update-design.md (section "State machine").
|
||||
*
|
||||
* `rollback-failed` is the only terminal state that disables auto/autonomous
|
||||
* attempts globally until POST /admin/update/acknowledge clears it. Manual
|
||||
* remains permitted because an admin clicking Apply *is* the intervention.
|
||||
*/
|
||||
export type ExecutionStatus =
|
||||
| {status: 'idle'}
|
||||
| {status: 'preflight'; targetTag: string; startedAt: string}
|
||||
| {status: 'preflight-failed'; targetTag: string; reason: string; at: string}
|
||||
| {status: 'draining'; targetTag: string; drainEndsAt: string; startedAt: string}
|
||||
| {status: 'executing'; targetTag: string; fromSha: string; startedAt: string}
|
||||
| {status: 'pending-verification'; targetTag: string; fromSha: string; deadlineAt: string}
|
||||
| {status: 'verified'; targetTag: string; verifiedAt: string}
|
||||
| {status: 'rolling-back'; reason: string; targetTag: string; fromSha: string; at: string}
|
||||
| {status: 'rolled-back'; reason: string; targetTag: string; restoredSha: string; at: string}
|
||||
| {status: 'rollback-failed'; reason: string; targetTag: string; fromSha: string; at: string};
|
||||
|
||||
/** All recognised execution statuses — used by the state validator. */
|
||||
export const EXECUTION_STATUSES = [
|
||||
'idle', 'preflight', 'preflight-failed', 'draining', 'executing',
|
||||
'pending-verification', 'verified', 'rolling-back', 'rolled-back', 'rollback-failed',
|
||||
] as const;
|
||||
|
||||
export type LastUpdateResult = {
|
||||
/** Tag we were updating to. */
|
||||
targetTag: string;
|
||||
/** SHA we were updating from. Empty string when the run never reached executor (e.g. preflight-failed). */
|
||||
fromSha: string;
|
||||
/** Outcome to surface in admin UI. */
|
||||
outcome: 'verified' | 'rolled-back' | 'rollback-failed' | 'preflight-failed' | 'cancelled';
|
||||
/** Human-readable reason on non-success. */
|
||||
reason: string | null;
|
||||
/** ISO timestamp when this result was finalised. */
|
||||
at: string;
|
||||
} | null;
|
||||
|
||||
export interface UpdateState {
|
||||
/** Schema version of this file. Increment when fields change. */
|
||||
schemaVersion: 1;
|
||||
|
|
@ -58,6 +97,15 @@ export interface UpdateState {
|
|||
vulnerableBelow: VulnerableBelowDirective[];
|
||||
/** Email send dedupe state. */
|
||||
email: EmailSendLog;
|
||||
/** Current in-flight execution state. Persisted so a restart mid-update reaches RollbackHandler. */
|
||||
execution: ExecutionStatus;
|
||||
/**
|
||||
* Boot counter that the RollbackHandler increments while a `pending-verification`
|
||||
* status is live. > 2 means the new version crash-looped; force rollback regardless of timer.
|
||||
*/
|
||||
bootCount: number;
|
||||
/** Most recent terminal outcome, surfaced in admin UI even after `execution` returns to idle. */
|
||||
lastResult: LastUpdateResult;
|
||||
}
|
||||
|
||||
/** Zero-value initial state. Treat as immutable — spread before mutating: `{...EMPTY_STATE, lastCheckAt: x}`. */
|
||||
|
|
@ -72,4 +120,7 @@ export const EMPTY_STATE: UpdateState = {
|
|||
vulnerableAt: null,
|
||||
vulnerableNewReleaseTag: null,
|
||||
},
|
||||
execution: {status: 'idle'},
|
||||
bootCount: 0,
|
||||
lastResult: null,
|
||||
};
|
||||
|
|
|
|||
82
src/node/updater/updateLog.ts
Normal file
82
src/node/updater/updateLog.ts
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
import fs from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
|
||||
const DEFAULT_MAX_BYTES = 10 * 1024 * 1024;
|
||||
const DEFAULT_BACKUPS = 5;
|
||||
|
||||
/**
|
||||
* Rotate `<logPath>` when it exceeds `maxBytes`:
|
||||
* <logPath>.{n-1} -> .n (oldest dropped)
|
||||
* <logPath> -> .1
|
||||
* No-op when the file is missing or under the limit.
|
||||
*/
|
||||
export const rotateIfNeeded = async (
|
||||
logPath: string,
|
||||
maxBytes = DEFAULT_MAX_BYTES,
|
||||
backups = DEFAULT_BACKUPS,
|
||||
): Promise<void> => {
|
||||
let size = 0;
|
||||
try { size = (await fs.stat(logPath)).size; } catch (err: any) {
|
||||
if (err.code === 'ENOENT') return;
|
||||
throw err;
|
||||
}
|
||||
if (size < maxBytes) return;
|
||||
|
||||
// Drop the oldest. Walk from highest index down so the rename chain lands cleanly.
|
||||
for (let i = backups - 1; i >= 1; i--) {
|
||||
const src = `${logPath}.${i}`;
|
||||
const dst = `${logPath}.${i + 1}`;
|
||||
try { await fs.rename(src, dst); }
|
||||
catch (err: any) { if (err.code !== 'ENOENT') throw err; }
|
||||
}
|
||||
// Current file becomes .1.
|
||||
try { await fs.rename(logPath, `${logPath}.1`); }
|
||||
catch (err: any) { if (err.code !== 'ENOENT') throw err; }
|
||||
};
|
||||
|
||||
/**
|
||||
* Append `line` to `<logPath>`, rotating first if the file is over the size cap.
|
||||
* Creates parent directories as needed. The line is newline-terminated; do not
|
||||
* include a trailing newline in `line`.
|
||||
*
|
||||
* Best-effort: swallows fs errors silently. Update logging must never break the
|
||||
* update flow itself, and errors are already surfaced via log4js by callers.
|
||||
*/
|
||||
export const appendLine = async (
|
||||
logPath: string,
|
||||
line: string,
|
||||
maxBytes = DEFAULT_MAX_BYTES,
|
||||
backups = DEFAULT_BACKUPS,
|
||||
): Promise<void> => {
|
||||
try {
|
||||
await fs.mkdir(path.dirname(logPath), {recursive: true});
|
||||
await rotateIfNeeded(logPath, maxBytes, backups);
|
||||
await fs.appendFile(logPath, `${line}\n`);
|
||||
} catch {
|
||||
// ignore — caller is fire-and-forget logging
|
||||
}
|
||||
};
|
||||
|
||||
/** Same as appendLine but throws on error — used by tests that want to assert disk failures surface. */
|
||||
export const appendLineStrict = async (
|
||||
logPath: string,
|
||||
line: string,
|
||||
maxBytes = DEFAULT_MAX_BYTES,
|
||||
backups = DEFAULT_BACKUPS,
|
||||
): Promise<void> => {
|
||||
await fs.mkdir(path.dirname(logPath), {recursive: true});
|
||||
await rotateIfNeeded(logPath, maxBytes, backups);
|
||||
await fs.appendFile(logPath, `${line}\n`);
|
||||
};
|
||||
|
||||
/** Read the last `n` newline-separated lines from the active log file. Empty array if missing. */
|
||||
export const tailLines = async (logPath: string, n: number): Promise<string[]> => {
|
||||
if (n <= 0) return [];
|
||||
let raw: string;
|
||||
try { raw = await fs.readFile(logPath, 'utf8'); }
|
||||
catch (err: any) { if (err.code === 'ENOENT') return []; throw err; }
|
||||
const stripped = raw.endsWith('\n') ? raw.slice(0, -1) : raw;
|
||||
if (stripped.length === 0) return [];
|
||||
const all = stripped.split('\n');
|
||||
return all.slice(Math.max(0, all.length - n));
|
||||
};
|
||||
|
|
@ -331,6 +331,15 @@ export type SettingsType = {
|
|||
checkIntervalHours: number,
|
||||
githubRepo: string,
|
||||
requireAdminForStatus: boolean,
|
||||
/** Tier 2+ knobs. Default 0 in PR 2; tier 3 makes preApplyGraceMinutes meaningful. */
|
||||
preApplyGraceMinutes: number,
|
||||
drainSeconds: number,
|
||||
rollbackHealthCheckSeconds: number,
|
||||
diskSpaceMinMB: number,
|
||||
/** When true, refuse updates whose tag is not signed by a trusted key. */
|
||||
requireSignature: boolean,
|
||||
/** Override the OS keyring location (passed to git verify-tag via $GNUPGHOME). */
|
||||
trustedKeysPath: string | null,
|
||||
},
|
||||
adminOpenAPI: {
|
||||
enabled: boolean,
|
||||
|
|
@ -518,6 +527,13 @@ const settings: SettingsType = {
|
|||
// Set true to require an authenticated admin session for the endpoint without
|
||||
// disabling the updater itself.
|
||||
requireAdminForStatus: false,
|
||||
// Tier 2+ knobs. Only meaningful at tier "manual" or higher.
|
||||
preApplyGraceMinutes: 0,
|
||||
drainSeconds: 60,
|
||||
rollbackHealthCheckSeconds: 60,
|
||||
diskSpaceMinMB: 500,
|
||||
requireSignature: false,
|
||||
trustedKeysPath: null,
|
||||
},
|
||||
/**
|
||||
* Admin OpenAPI document endpoint at /admin/openapi.json.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue