mirror of
https://github.com/ether/etherpad-lite.git
synced 2026-07-25 02:57:09 +00:00
* docs(updater): plan tier 4 — autonomous update in maintenance window (#7607) Maps PR 4 of the auto-update design spec (§"Tier 4 — autonomous") to concrete files, tasks, and verification steps. Subsequent commits scaffold against this plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): MaintenanceWindow module — wall-clock window math for tier 4 Pure module: parseWindow, inWindow, nextWindowStart. Supports tz=local|utc and cross-midnight ranges. Used by upcoming Scheduler + UpdatePolicy changes. 22 vitest unit tests cover format validation, same-day + cross-midnight boundaries, and host-local vs UTC clock comparisons. DST handling is absorbed by JS Date constructor's wall-clock normalization (documented in the file header). Refs #7607 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): tier 4 backend — window-gated UpdatePolicy + Scheduler Wires MaintenanceWindow into the existing tier 3 backend so autonomous updates only fire while `now` is inside `updates.maintenanceWindow`. UpdatePolicy - new optional `maintenanceWindow` input - canAutonomous flips on only for git+tier=autonomous+parse-valid window - new reasons `maintenance-window-missing` / `maintenance-window-invalid` - rollback-failed still wins over window denial Scheduler - decideSchedule snaps scheduledFor forward to nextWindowStart when canAutonomous + grace lands outside the window - decideTriggerApply returns a new `{action: 'defer'}` when canAutonomous + fire-time is outside the window; carries nextStart for the runner - canAutonomous=false preserves Tier 3 behavior unchanged index.ts wires settings.updates.maintenanceWindow through both passes and re-arms the timer on defer. Status endpoint surface (nextWindowOpensAt) + admin UI picker land in a follow-up commit. Settings adds `maintenanceWindow: {start, end, tz} | null`, defaulting to null. settings.json.template / settings.json.docker document the shape. Tests - 22 vitest cases for MaintenanceWindow already cover the math - 4 new UpdatePolicy cases for the window outcomes - 6 new Scheduler cases for tier-4 schedule/trigger paths - Full backend-new suite: 629 passed (35 files) Refs #7607 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): tier 4 admin UI — window status, deferred subtitle, banner GET /admin/update/status now returns: - `maintenanceWindow`: the parsed window object (admin sessions only) - `nextWindowOpensAt`: ISO of the next window opening when tier=autonomous UpdatePage - new "Maintenance window" section when tier=autonomous, shows current window summary + next opens at, or "Not configured" when unset - scheduled panel now appends a "deferred until <iso>" line when the backend has snapped scheduledFor to the next window opening UpdateBanner - new variant when tier=autonomous and policy.reason is `maintenance-window-missing` or `maintenance-window-invalid`, linking to /admin/update i18n - 8 new keys under `update.banner.*`, `update.page.policy.*`, `update.page.scheduled.*`, `update.window.*` (en.json only; translations follow via the usual locale workflow) Interactive picker is intentionally deferred — admins edit `updates.maintenanceWindow` via the parsed JSONC settings editor (#7709). A follow-up commit may add a thin write-through component if the JSONC round-trip turns out to be too rough for typical operators. Refs #7607 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(updater): tier 4 — window gate, DST notes, runbook §12 (#7607) CHANGELOG: flip Tier 4 from "designed, not yet implemented" to current. Document maintenanceWindow shape, snap-forward, defer-at-fire, and the two missing/invalid policy reasons. doc/admin/updates.md: new "Tier 4 — autonomous in a maintenance window" section with config example, policy gating, DST/timezone notes, admin UI behavior. runbook: §12 walks a disposable VM through missing-window, malformed, outside-window deferral, fire-at-opening, and window-closes-mid-grace. Adds five sign-off checklist items. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(updater): tier 4 window-boundary integration (#7607) Mocha integration covering the four scenarios called out in the spec §"Tier 4 — autonomous": - outside-window: decideSchedule snaps scheduledFor forward to the next opening and the snapped value round-trips through saveState - inside-window at fire-time: decideTriggerApply returns fire - window-closes-mid-grace: decideTriggerApply returns defer with nextStart at the next opening; persisted state moves forward - cancel during deferred-grace: state returns to idle, and the next decideSchedule pass re-emits a schedule snapped to the next opening All 4 cases passing locally under tsx mocha. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): real SMTP via nodemailer (mail.* settings) (#7607) Replaces the (would send email) stub introduced in PR #7601 with a nodemailer-backed transport. The dependency is lazy-imported so installs that don't set mail.host pay no runtime cost. Settings additions - new top-level mail block: host, port, secure, from, auth (user/pass) - mail.host=null keeps the legacy log-only behaviour; the Notifier still updates dedupe state so we don't re-evaluate every tick - settings.json.template documents the shape inline - settings.json.docker reads MAIL_HOST / MAIL_FROM / MAIL_PORT / MAIL_SECURE from env so operators can configure via container env Transport - lazy import('nodemailer') on first send - transport cached by host; settings reload picks up new host without needing a restart - send errors are swallowed (logged warn) so a transient SMTP failure can never poison the surrounding updater state machine - successful sends log at info; legacy "(would send email)" path remains the visible signal when mail is disabled Refs #7607 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): preflight checks target tag's engines.node (#7607) Before mutating the working tree, runPreflight now reads the target tag's package.json via `git show <tag>:package.json` and verifies that process.versions.node satisfies its engines.node range. Failures land at preflight-failed cleanly (no rollback needed — nothing has changed yet). Motivation: a release that bumps the Node floor used to either fail mid-`pnpm install` (which then rolls back successfully) or restart on the new build and crash in the boot path (which then rolls back via the health-check timer). Both paths recover, but they burn a drain + restart cycle on a condition we can reject upfront. Implementation - new PreflightReason `node-engine-mismatch` - new dep `readTargetEnginesNode(tag)` — runs the git-show as a child process with stdio captured to a string; missing tag / missing file / malformed JSON / missing engines.node all resolve to null (treated as "no constraint, pass") - uses existing semver dep with includePrerelease: true - new PreflightInput field `currentNodeVersion`; threaded from process.versions.node in both wirings (scheduler + manual apply) - check runs *after* signature verification so we trust the package.json - PreflightResult carries an optional `detail` string; applyPipeline appends it to the lastResult.reason so the admin UI shows e.g. "node-engine-mismatch: target requires Node >=26.0.0, running 25.0.0" Tests: 6 new vitest cases (no engines.node, satisfies, fails below floor, caret range, loose-spaced range, ordering after signature). Full backend-new: 635 passed (was 629). Refs #7607 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(updater): email admin on auto-rollback / preflight-failed (#7607) Before this commit, only the terminal rollback-failed state emailed the admin. Auto-recovered failures (rolled-back-install-failed, rolled-back- build-failed, rolled-back-health-check, rolled-back-crash-loop) and pre- flight-failed surfaced only via the /admin/update banner — so a 3am autonomous update that failed because of, say, a Node engine bump would roll back silently and stay invisible until the admin next logged in. Notifier - new EmailKinds: 'update-preflight-failed', 'update-rolled-back', 'update-rollback-failed' - new pure decideOutcomeEmail(input) → {toSend, newState} - dedupe key `<outcome>:<targetTag>` in EmailSendLog.lastFailureKey: same outcome on same tag emits one email per cycle (kills retry-loop spam); a different outcome or different tag resets the key - rollback-failed always fires (terminal — overrides dedupe) - state.ts validator + loadState backfill the new field for legacy state files (Tier 1/2/3 installs upgrading in place) Wiring - new index.ts helper notifyApplyFailure() loads state, runs the pure notifier, sends (via the nodemailer-backed sendEmailViaSmtp from the previous commit), persists the new dedupe key — all best-effort - schedulerTriggerApply: fires on applyUpdate returning preflight-failed or rolled-back - /admin/update/apply HTTP handler: same - boot path in expressCreateServer: if state.lastResult is a failure outcome we haven't already emailed about, fire then. Covers: - health-check timeout rollback (timer expired between boots) - crash-loop forced rollback caught on a later boot - preflight-failed where the process didn't get to email before exit - unacknowledged rollback-failed terminal Tests - 8 new vitest cases for decideOutcomeEmail (adminEmail=null, each outcome's content, dedupe by tag, dedupe by outcome, rollback-failed bypass) - Full backend-new suite: 643 passed (was 635) Refs #7607 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(updater): address Qodo review on tier 4 - UpdatePage: only show "deferred until" subtitle when scheduledFor actually matches nextWindowOpensAt. The previous `scheduledFor > now + 60s` heuristic misfired during a normal in-window 15-min grace period. - applyPipeline: return the enriched preflight reason (`reason: detail`) instead of only `pf.reason`, so /admin/update/apply 409 bodies and failure-notify emails preserve diagnostics like the Node engine mismatch detail. - updater/index: key the cached nodemailer transport on the full set of SMTP options (host + port + secure + auth) so runtime changes to port/credentials via reloadSettings() invalidate the cache. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
John McLear
GitHub
parent
dbd4662d2b
commit
962bfe8649
32 changed files with 1678 additions and 50 deletions
|
|
@ -6,7 +6,7 @@ import {spawn} from 'node:child_process';
|
|||
import log4js from 'log4js';
|
||||
import {ArgsExpressType} from '../../types/ArgsExpressType';
|
||||
import settings, {getEpVersion} from '../../utils/Settings';
|
||||
import {getDetectedInstallMethod, stateFilePath, getRollbackDeps} from '../../updater';
|
||||
import {getDetectedInstallMethod, stateFilePath, getRollbackDeps, notifyApplyFailure} from '../../updater';
|
||||
import {evaluatePolicy} from '../../updater/UpdatePolicy';
|
||||
import {loadState, saveState} from '../../updater/state';
|
||||
import {acquireLock, releaseLock} from '../../updater/lock';
|
||||
|
|
@ -104,6 +104,20 @@ const buildPreflightDeps = (installMethod: ReturnType<typeof getDetectedInstallM
|
|||
requireSignature: settings.updates.requireSignature,
|
||||
trustedKeysPath: settings.updates.trustedKeysPath,
|
||||
}),
|
||||
readTargetEnginesNode: (tag: string) => new Promise<string | null>((resolve) => {
|
||||
const c = spawn('git', ['show', `${tag}:package.json`],
|
||||
{cwd: settings.root, stdio: ['ignore', 'pipe', 'ignore']});
|
||||
let out = '';
|
||||
c.stdout.on('data', (b) => { out += b.toString(); });
|
||||
c.on('close', () => {
|
||||
try {
|
||||
const pkg = JSON.parse(out);
|
||||
const range = pkg?.engines?.node;
|
||||
resolve(typeof range === 'string' && range.trim().length > 0 ? range : null);
|
||||
} catch { resolve(null); }
|
||||
});
|
||||
c.on('error', () => resolve(null));
|
||||
}),
|
||||
});
|
||||
|
||||
/**
|
||||
|
|
@ -193,6 +207,7 @@ export const expressCreateServer = (
|
|||
diskSpaceMinMB: Number(settings.updates.diskSpaceMinMB) || 500,
|
||||
requireSignature: settings.updates.requireSignature,
|
||||
trustedKeysPath: settings.updates.trustedKeysPath,
|
||||
currentNodeVersion: process.versions.node,
|
||||
},
|
||||
{
|
||||
...baseDeps,
|
||||
|
|
@ -256,6 +271,20 @@ export const expressCreateServer = (
|
|||
});
|
||||
drainer = null;
|
||||
|
||||
// Fire the failure-notification email path for outcomes the admin needs
|
||||
// to know about even on manual apply (an admin might click Apply and
|
||||
// walk away; rolling back silently isn't enough). Errors here are
|
||||
// swallowed by notifyApplyFailure — they must not block the response.
|
||||
if (result.outcome === 'preflight-failed') {
|
||||
void notifyApplyFailure({
|
||||
outcome: 'preflight-failed', targetTag, reason: result.reason,
|
||||
});
|
||||
} else if (result.outcome === 'rolled-back') {
|
||||
void notifyApplyFailure({
|
||||
outcome: 'rolled-back', targetTag, reason: 'rolled-back',
|
||||
});
|
||||
}
|
||||
|
||||
if (responded) return; // already 202'd in onAccepted; nothing more to send.
|
||||
|
||||
switch (result.outcome) {
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import {evaluatePolicy} from '../../updater/UpdatePolicy';
|
|||
import {compareSemver, isMajorBehind, isVulnerable} from '../../updater/versionCompare';
|
||||
import {loadState} from '../../updater/state';
|
||||
import {isHeld} from '../../updater/lock';
|
||||
import {nextWindowStart, parseWindow} from '../../updater/MaintenanceWindow';
|
||||
|
||||
|
||||
let badgeCache: {value: 'severe' | 'vulnerable' | null; at: number} = {value: null, at: 0};
|
||||
|
|
@ -103,9 +104,19 @@ export const expressCreateServer = (
|
|||
current,
|
||||
latest: state.latest.version,
|
||||
executionStatus: state.execution.status,
|
||||
maintenanceWindow: settings.updates.maintenanceWindow,
|
||||
})
|
||||
: null;
|
||||
const lockHeld = await isHeld(path.join(settings.root, 'var', 'update.lock'));
|
||||
// Tier 4: surface the configured window + the next opening so the admin UI
|
||||
// can render the picker and the "deferred until..." subtitle on the
|
||||
// scheduled panel. Non-admin requests get null for both fields (the parsed
|
||||
// window is operational config, not a public datum).
|
||||
const parsedWindow = parseWindow(settings.updates.maintenanceWindow);
|
||||
const maintenanceWindow = isAdmin ? parsedWindow : null;
|
||||
const nextWindowOpensAt = isAdmin && parsedWindow && settings.updates.tier === 'autonomous'
|
||||
? nextWindowStart(new Date(), parsedWindow).toISOString()
|
||||
: null;
|
||||
|
||||
// The Tier 2 fields (execution, lastResult) carry diagnostic strings
|
||||
// built from git/pnpm stderr — environment-specific paths, error
|
||||
|
|
@ -132,6 +143,9 @@ export const expressCreateServer = (
|
|||
execution,
|
||||
lastResult,
|
||||
lockHeld,
|
||||
// PR 4 additions:
|
||||
maintenanceWindow,
|
||||
nextWindowOpensAt,
|
||||
});
|
||||
}));
|
||||
|
||||
|
|
|
|||
105
src/node/updater/MaintenanceWindow.ts
Normal file
105
src/node/updater/MaintenanceWindow.ts
Normal file
|
|
@ -0,0 +1,105 @@
|
|||
/**
|
||||
* Maintenance-window math for Tier 4 (autonomous updates).
|
||||
*
|
||||
* Pure — no I/O, no log4js, no globals beyond `Date`. Imported by:
|
||||
* - `UpdatePolicy.ts` (canAutonomous gate)
|
||||
* - `Scheduler.ts` (snap scheduledFor to the next window opening, defer fires)
|
||||
* - `index.ts` (compute nextWindowOpensAt for /admin/update/status)
|
||||
* - admin UI picker (validation)
|
||||
*
|
||||
* Time semantics
|
||||
* --------------
|
||||
* A window is a pair of HH:MM wall-clock times plus a `tz` selector. For
|
||||
* `tz: 'utc'`, comparisons use `getUTCHours/Minutes` and `Date.UTC(...)`. For
|
||||
* `tz: 'local'`, they use the host's local wall clock via the standard `Date`
|
||||
* constructor. `nextWindowStart` therefore returns a `Date` whose wall-clock
|
||||
* components in the configured tz equal `window.start` — DST transitions are
|
||||
* absorbed by the JS Date constructor's normalization (a 02:30 window-start on
|
||||
* a spring-forward day silently lands at 03:30 local because 02:30 does not
|
||||
* exist; documented behavior, not a bug).
|
||||
*
|
||||
* Cross-midnight windows are supported (`end < start` means "wraps past
|
||||
* 00:00"). The `end` minute is exclusive in both same-day and cross-midnight
|
||||
* cases — a `22:00-02:00` window matches `[22:00, 24:00) ∪ [00:00, 02:00)`.
|
||||
*/
|
||||
|
||||
export interface MaintenanceWindow {
|
||||
/** Wall-clock start in `HH:MM` (24h). */
|
||||
start: string;
|
||||
/** Wall-clock end in `HH:MM` (24h). Exclusive. */
|
||||
end: string;
|
||||
/** Whether `start`/`end` are read against UTC or the host's local clock. */
|
||||
tz: 'local' | 'utc';
|
||||
}
|
||||
|
||||
const HHMM = /^([01]\d|2[0-3]):([0-5]\d)$/;
|
||||
|
||||
const toMinutes = (hhmm: string): number | null => {
|
||||
const m = HHMM.exec(hhmm);
|
||||
if (!m) return null;
|
||||
return Number(m[1]) * 60 + Number(m[2]);
|
||||
};
|
||||
|
||||
/**
|
||||
* Parse and validate a raw value (typically from `settings.json`) into a
|
||||
* `MaintenanceWindow`. Returns `null` for any structural or format failure —
|
||||
* callers should treat that as "tier 4 disabled, fall back to tier 3".
|
||||
*/
|
||||
export const parseWindow = (raw: unknown): MaintenanceWindow | null => {
|
||||
if (!raw || typeof raw !== 'object') return null;
|
||||
const r = raw as Record<string, unknown>;
|
||||
if (typeof r.start !== 'string' || typeof r.end !== 'string') return null;
|
||||
if (r.tz !== 'local' && r.tz !== 'utc') return null;
|
||||
const s = toMinutes(r.start);
|
||||
const e = toMinutes(r.end);
|
||||
if (s == null || e == null) return null;
|
||||
if (s === e) return null;
|
||||
return {start: r.start, end: r.end, tz: r.tz};
|
||||
};
|
||||
|
||||
const wallMinutes = (now: Date, tz: MaintenanceWindow['tz']): number => (
|
||||
tz === 'utc'
|
||||
? now.getUTCHours() * 60 + now.getUTCMinutes()
|
||||
: now.getHours() * 60 + now.getMinutes()
|
||||
);
|
||||
|
||||
/**
|
||||
* `true` iff `now`'s wall-clock minute is within `[start, end)` in the window's
|
||||
* tz. Cross-midnight windows wrap at 24:00 — see file header for the exact set.
|
||||
*/
|
||||
export const inWindow = (now: Date, window: MaintenanceWindow): boolean => {
|
||||
const s = toMinutes(window.start);
|
||||
const e = toMinutes(window.end);
|
||||
if (s == null || e == null || s === e) return false;
|
||||
const m = wallMinutes(now, window.tz);
|
||||
return s < e ? (m >= s && m < e) : (m >= s || m < e);
|
||||
};
|
||||
|
||||
const buildAt = (year: number, month: number, day: number, mins: number,
|
||||
tz: MaintenanceWindow['tz']): Date => {
|
||||
const h = Math.floor(mins / 60);
|
||||
const mm = mins % 60;
|
||||
return tz === 'utc'
|
||||
? new Date(Date.UTC(year, month, day, h, mm, 0, 0))
|
||||
: new Date(year, month, day, h, mm, 0, 0);
|
||||
};
|
||||
|
||||
/**
|
||||
* Smallest `Date` `t` such that `t >= now` and `t`'s wall-clock equals
|
||||
* `window.start` in the window's tz. Used by Scheduler to snap a scheduledFor
|
||||
* that lands outside the window forward to the next opening.
|
||||
*
|
||||
* If `now` is *inside* the window, the next opening is tomorrow — we don't
|
||||
* collapse to `now`. Fire-now is gated by `inWindow`, not this function.
|
||||
*/
|
||||
export const nextWindowStart = (now: Date, window: MaintenanceWindow): Date => {
|
||||
const s = toMinutes(window.start);
|
||||
if (s == null) return now;
|
||||
const isUtc = window.tz === 'utc';
|
||||
const year = isUtc ? now.getUTCFullYear() : now.getFullYear();
|
||||
const month = isUtc ? now.getUTCMonth() : now.getMonth();
|
||||
const day = isUtc ? now.getUTCDate() : now.getDate();
|
||||
const todayStart = buildAt(year, month, day, s, window.tz);
|
||||
if (todayStart.getTime() > now.getTime()) return todayStart;
|
||||
return buildAt(year, month, day + 1, s, window.tz);
|
||||
};
|
||||
|
|
@ -13,7 +13,14 @@ export interface NotifierInput {
|
|||
now: Date;
|
||||
}
|
||||
|
||||
export type EmailKind = 'severe' | 'vulnerable' | 'vulnerable-new-release' | 'grace-start';
|
||||
export type EmailKind =
|
||||
| 'severe'
|
||||
| 'vulnerable'
|
||||
| 'vulnerable-new-release'
|
||||
| 'grace-start'
|
||||
| 'update-preflight-failed'
|
||||
| 'update-rolled-back'
|
||||
| 'update-rollback-failed';
|
||||
|
||||
export interface PlannedEmail {
|
||||
kind: EmailKind;
|
||||
|
|
@ -86,3 +93,72 @@ export const decideEmails = (input: NotifierInput): NotifierResult => {
|
|||
|
||||
return {toSend, newState};
|
||||
};
|
||||
|
||||
export type FailureOutcome =
|
||||
| 'preflight-failed'
|
||||
| 'rolled-back'
|
||||
| 'rollback-failed';
|
||||
|
||||
export interface OutcomeEmailInput {
|
||||
adminEmail: string | null;
|
||||
outcome: FailureOutcome;
|
||||
/** Free-text reason string from `ApplyResult.reason` (or RollbackHandler). */
|
||||
reason: string;
|
||||
/** Tag the failed apply was targeting. */
|
||||
targetTag: string;
|
||||
/** Currently-running Etherpad version (so the admin sees what's live now). */
|
||||
currentVersion: string;
|
||||
/** Email-state slice from UpdateState. */
|
||||
state: EmailSendLog;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether to email about a non-success apply outcome. Pure — returns
|
||||
* the planned email + new dedupe state; does not send.
|
||||
*
|
||||
* Dedupe key: `<outcome>:<targetTag>`. Same outcome on the same tag (e.g.
|
||||
* a retry loop that keeps failing `pnpm install` for v2.7.6) emits one
|
||||
* email. A different outcome OR a different tag resets the dedupe key and
|
||||
* fires a new email.
|
||||
*
|
||||
* `rollback-failed` always fires (overrides dedupe) — it's the terminal
|
||||
* state that needs human intervention and the admin must learn about it
|
||||
* even if a previous transient failure happened to share its key.
|
||||
*/
|
||||
export const decideOutcomeEmail = (input: OutcomeEmailInput): NotifierResult => {
|
||||
const {adminEmail, outcome, reason, targetTag, currentVersion, state} = input;
|
||||
if (!adminEmail) return {toSend: [], newState: state};
|
||||
|
||||
const key = `${outcome}:${targetTag}`;
|
||||
const isTerminal = outcome === 'rollback-failed';
|
||||
if (!isTerminal && state.lastFailureKey === key) {
|
||||
return {toSend: [], newState: state};
|
||||
}
|
||||
|
||||
const kind: EmailKind =
|
||||
outcome === 'preflight-failed' ? 'update-preflight-failed'
|
||||
: outcome === 'rolled-back' ? 'update-rolled-back'
|
||||
: 'update-rollback-failed';
|
||||
|
||||
const titleByKind: Record<typeof kind, string> = {
|
||||
'update-preflight-failed':
|
||||
`[Etherpad] Auto-update to ${targetTag} blocked at preflight`,
|
||||
'update-rolled-back':
|
||||
`[Etherpad] Auto-update to ${targetTag} rolled back`,
|
||||
'update-rollback-failed':
|
||||
`[Etherpad] Auto-update FAILED and could not be rolled back — manual intervention required`,
|
||||
};
|
||||
|
||||
const bodyTail = isTerminal
|
||||
? ' Visit /admin/update and POST /admin/update/acknowledge after restoring a working install.'
|
||||
: ' Visit /admin/update for details.';
|
||||
|
||||
const body =
|
||||
`Etherpad attempted to auto-update to ${targetTag} but failed: ${reason}.\n` +
|
||||
`The running version is ${currentVersion}.${bodyTail}`;
|
||||
|
||||
return {
|
||||
toSend: [{kind, subject: titleByKind[kind], body}],
|
||||
newState: {...state, lastFailureKey: key},
|
||||
};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import {EmailSendLog, ExecutionStatus, PolicyResult, ReleaseInfo, UpdateState} from './types';
|
||||
import {EmailSendLog, ExecutionStatus, MaintenanceWindow, PolicyResult, ReleaseInfo, UpdateState} from './types';
|
||||
import {PlannedEmail} from './Notifier';
|
||||
import {inWindow, nextWindowStart} from './MaintenanceWindow';
|
||||
|
||||
export interface DecideScheduleInput {
|
||||
state: UpdateState;
|
||||
|
|
@ -9,6 +10,13 @@ export interface DecideScheduleInput {
|
|||
current: string;
|
||||
preApplyGraceMinutes: number;
|
||||
adminEmail: string | null;
|
||||
/**
|
||||
* Tier 4 only — when `policy.canAutonomous` is true, the scheduler snaps
|
||||
* `scheduledFor` forward to the next window opening (if it would otherwise
|
||||
* land outside the window) and `decideTriggerApply` defers fires that the
|
||||
* window has closed for. Ignored when `canAutonomous === false`.
|
||||
*/
|
||||
maintenanceWindow?: MaintenanceWindow | null;
|
||||
}
|
||||
|
||||
export type SchedulerDecision =
|
||||
|
|
@ -48,7 +56,10 @@ const clampGrace = (m: number): number => {
|
|||
* email when `adminEmail` is set and `email.graceStartTag !== latest.tag`.
|
||||
*/
|
||||
export const decideSchedule = (input: DecideScheduleInput): SchedulerDecision => {
|
||||
const {state, now, policy, latest, current, preApplyGraceMinutes, adminEmail} = input;
|
||||
const {
|
||||
state, now, policy, latest, current, preApplyGraceMinutes, adminEmail,
|
||||
maintenanceWindow,
|
||||
} = input;
|
||||
const status = state.execution.status;
|
||||
|
||||
if (!latest) return {action: 'nothing'};
|
||||
|
|
@ -66,7 +77,14 @@ export const decideSchedule = (input: DecideScheduleInput): SchedulerDecision =>
|
|||
}
|
||||
|
||||
const graceMs = clampGrace(preApplyGraceMinutes) * 60 * 1000;
|
||||
const scheduledFor = new Date(now.getTime() + graceMs).toISOString();
|
||||
let scheduledForDate = new Date(now.getTime() + graceMs);
|
||||
// Tier 4: snap forward to the next opening if grace lands outside the window.
|
||||
if (policy.canAutonomous && maintenanceWindow) {
|
||||
if (!inWindow(scheduledForDate, maintenanceWindow)) {
|
||||
scheduledForDate = nextWindowStart(scheduledForDate, maintenanceWindow);
|
||||
}
|
||||
}
|
||||
const scheduledFor = scheduledForDate.toISOString();
|
||||
const newExecution = {
|
||||
status: 'scheduled' as const,
|
||||
targetTag: latest.tag,
|
||||
|
|
@ -92,7 +110,8 @@ export const decideSchedule = (input: DecideScheduleInput): SchedulerDecision =>
|
|||
export type TriggerApplyDecision =
|
||||
| {action: 'fire'}
|
||||
| {action: 'abort'; reason: string}
|
||||
| {action: 'clear-schedule'; reason: string};
|
||||
| {action: 'clear-schedule'; reason: string}
|
||||
| {action: 'defer'; nextStart: string; reason: 'outside-maintenance-window'};
|
||||
|
||||
/**
|
||||
* Decide whether the scheduler's timer-fire callback should actually run the
|
||||
|
|
@ -100,10 +119,20 @@ export type TriggerApplyDecision =
|
|||
* arming-to-firing has a long delay (the grace window) during which the
|
||||
* admin can cancel, click Apply now, or flip the tier. SchedulerRunnerDeps
|
||||
* documents this contract; this helper is the canonical implementation.
|
||||
*
|
||||
* Tier 4: when `policy.canAutonomous` is true and `now` is outside the
|
||||
* configured `maintenanceWindow`, returns `{action: 'defer'}` so the runner
|
||||
* persists a new `scheduledFor = nextStart` and re-arms.
|
||||
*/
|
||||
export const decideTriggerApply = ({
|
||||
state, targetTag, policy,
|
||||
}: {state: UpdateState; targetTag: string; policy: PolicyResult}): TriggerApplyDecision => {
|
||||
state, targetTag, policy, now, maintenanceWindow,
|
||||
}: {
|
||||
state: UpdateState;
|
||||
targetTag: string;
|
||||
policy: PolicyResult;
|
||||
now?: Date;
|
||||
maintenanceWindow?: MaintenanceWindow | null;
|
||||
}): TriggerApplyDecision => {
|
||||
if (state.execution.status !== 'scheduled') {
|
||||
return {action: 'abort', reason: `state=${state.execution.status}`};
|
||||
}
|
||||
|
|
@ -112,6 +141,13 @@ export const decideTriggerApply = ({
|
|||
}
|
||||
if (!state.latest) return {action: 'abort', reason: 'no-latest'};
|
||||
if (!policy.canAuto) return {action: 'clear-schedule', reason: policy.reason || 'policy-denied'};
|
||||
if (policy.canAutonomous && maintenanceWindow && now && !inWindow(now, maintenanceWindow)) {
|
||||
return {
|
||||
action: 'defer',
|
||||
nextStart: nextWindowStart(now, maintenanceWindow).toISOString(),
|
||||
reason: 'outside-maintenance-window',
|
||||
};
|
||||
}
|
||||
return {action: 'fire'};
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import {compareSemver} from './versionCompare';
|
||||
import {InstallMethod, PolicyResult, Tier} from './types';
|
||||
import {parseWindow} from './MaintenanceWindow';
|
||||
import {InstallMethod, MaintenanceWindow, PolicyResult, Tier} from './types';
|
||||
|
||||
// For PR 1 (notify only) the writable list contains only 'git'.
|
||||
// PR 2+ may add 'npm' here as the executor learns to handle that path.
|
||||
|
|
@ -17,19 +18,29 @@ export interface PolicyInput {
|
|||
* intervention the terminal state requires.
|
||||
*/
|
||||
executionStatus?: string;
|
||||
/**
|
||||
* Configured maintenance window from `updates.maintenanceWindow`. Tier 4
|
||||
* requires a non-null, parse-valid window. When null or malformed,
|
||||
* canAutonomous degrades to false with a reason of
|
||||
* `maintenance-window-missing` / `maintenance-window-invalid`; the other
|
||||
* permissions still resolve as if tier were `auto`.
|
||||
*/
|
||||
maintenanceWindow?: MaintenanceWindow | unknown | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide which update tiers are allowed under the given (installMethod, tier,
|
||||
* current, latest, executionStatus). Pure function — no I/O. The single source
|
||||
* of truth for "what's allowed in this environment."
|
||||
* current, latest, executionStatus, maintenanceWindow). Pure function — no I/O.
|
||||
* The single source of truth for "what's allowed in this environment."
|
||||
*
|
||||
* `reason` is one of:
|
||||
* 'tier-off' | 'up-to-date' | 'install-method-not-writable'
|
||||
* | 'rollback-failed-terminal' | 'ok'.
|
||||
* | 'rollback-failed-terminal'
|
||||
* | 'maintenance-window-missing' | 'maintenance-window-invalid'
|
||||
* | 'ok'.
|
||||
*/
|
||||
export const evaluatePolicy = ({
|
||||
installMethod, tier, current, latest, executionStatus,
|
||||
installMethod, tier, current, latest, executionStatus, maintenanceWindow,
|
||||
}: PolicyInput): PolicyResult => {
|
||||
if (tier === 'off') {
|
||||
return {canNotify: false, canManual: false, canAuto: false, canAutonomous: false, reason: 'tier-off'};
|
||||
|
|
@ -46,11 +57,24 @@ export const evaluatePolicy = ({
|
|||
}
|
||||
|
||||
const terminal = executionStatus === 'rollback-failed';
|
||||
return {
|
||||
canNotify,
|
||||
canManual: tier === 'manual' || tier === 'auto' || tier === 'autonomous',
|
||||
canAuto: !terminal && (tier === 'auto' || tier === 'autonomous'),
|
||||
canAutonomous: !terminal && tier === 'autonomous',
|
||||
reason: terminal ? 'rollback-failed-terminal' : 'ok',
|
||||
};
|
||||
const canManual = tier === 'manual' || tier === 'auto' || tier === 'autonomous';
|
||||
const canAuto = !terminal && (tier === 'auto' || tier === 'autonomous');
|
||||
|
||||
let canAutonomous = false;
|
||||
let windowReason: string | null = null;
|
||||
if (!terminal && tier === 'autonomous') {
|
||||
if (maintenanceWindow == null) {
|
||||
windowReason = 'maintenance-window-missing';
|
||||
} else if (parseWindow(maintenanceWindow) == null) {
|
||||
windowReason = 'maintenance-window-invalid';
|
||||
} else {
|
||||
canAutonomous = true;
|
||||
}
|
||||
}
|
||||
|
||||
const reason = terminal
|
||||
? 'rollback-failed-terminal'
|
||||
: (windowReason ?? 'ok');
|
||||
|
||||
return {canNotify, canManual, canAuto, canAutonomous, reason};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -1,11 +1,11 @@
|
|||
import {UpdateState} from './types';
|
||||
import {PreflightResult, PreflightReason} from './preflight';
|
||||
import {PreflightResult} from './preflight';
|
||||
import {ExecutorResult} from './UpdateExecutor';
|
||||
import {Drainer, DrainBroadcastKey} from './SessionDrainer';
|
||||
|
||||
export type ApplyOutcome =
|
||||
| {outcome: 'pending-verification'}
|
||||
| {outcome: 'preflight-failed'; reason: PreflightReason}
|
||||
| {outcome: 'preflight-failed'; reason: string}
|
||||
| {outcome: 'cancelled'}
|
||||
| {outcome: 'lock-held'}
|
||||
| {outcome: 'busy'; status: string}
|
||||
|
|
@ -89,13 +89,17 @@ export const applyUpdate = async (
|
|||
const pf = await deps.runPreflight(targetTag);
|
||||
if (!pf.ok) {
|
||||
const at = deps.now().toISOString();
|
||||
// Append the optional `detail` (e.g. "target requires Node >=26.0.0,
|
||||
// running 25.0.0" for node-engine-mismatch) so the admin UI shows a
|
||||
// version-specific message without requiring a separate API field.
|
||||
const reasonStr = pf.detail ? `${pf.reason}: ${pf.detail}` : pf.reason;
|
||||
await deps.saveState({
|
||||
...preState,
|
||||
execution: {status: 'preflight-failed', targetTag, reason: pf.reason, at},
|
||||
lastResult: {targetTag, fromSha: '', outcome: 'preflight-failed', reason: pf.reason, at},
|
||||
execution: {status: 'preflight-failed', targetTag, reason: reasonStr, at},
|
||||
lastResult: {targetTag, fromSha: '', outcome: 'preflight-failed', reason: reasonStr, at},
|
||||
});
|
||||
deps.appendLog(`[${at}] PREFLIGHT_FAILED ${pf.reason}`);
|
||||
return {outcome: 'preflight-failed', reason: pf.reason};
|
||||
deps.appendLog(`[${at}] PREFLIGHT_FAILED ${reasonStr}`);
|
||||
return {outcome: 'preflight-failed', reason: reasonStr};
|
||||
}
|
||||
|
||||
// Re-load state after preflight: the cancel endpoint can flip execution
|
||||
|
|
|
|||
|
|
@ -8,10 +8,11 @@ import {checkLatestRelease, realFetcher} from './VersionChecker';
|
|||
import {loadState, saveState} from './state';
|
||||
import {isMajorBehind, isVulnerable} from './versionCompare';
|
||||
import {evaluatePolicy} from './UpdatePolicy';
|
||||
import {decideEmails} from './Notifier';
|
||||
import {decideEmails, decideOutcomeEmail, FailureOutcome} from './Notifier';
|
||||
import {checkPendingVerification, CheckResult, RollbackDeps, performRollback} from './RollbackHandler';
|
||||
import {executeUpdate, SpawnFn} from './UpdateExecutor';
|
||||
import {createSchedulerRunner, decideSchedule, decideTriggerApply, SchedulerRunner} from './Scheduler';
|
||||
import {parseWindow} from './MaintenanceWindow';
|
||||
import {applyUpdate, ApplyPipelineDeps} from './applyPipeline';
|
||||
import {acquireLock, releaseLock} from './lock';
|
||||
import {runPreflight} from './preflight';
|
||||
|
|
@ -42,11 +43,71 @@ export const getCurrentState = async (): Promise<UpdateState> => {
|
|||
|
||||
export const getDetectedInstallMethod = () => detectedMethod;
|
||||
|
||||
/**
|
||||
* Cached nodemailer transport. Built on first use when `settings.mail.host` is
|
||||
* set; never imported when mail is disabled (keeps boot costs predictable for
|
||||
* installs that don't care about outbound mail).
|
||||
*
|
||||
* The cache is keyed on the full set of SMTP options that `buildTransport`
|
||||
* consumes (host, port, secure, auth). `reloadSettings()` can mutate any of
|
||||
* these at runtime, so a host-only key would silently keep using a stale
|
||||
* transport when an operator rotates credentials or moves to a different
|
||||
* port without changing host.
|
||||
*/
|
||||
let transportCache: {key: string; transporter: {sendMail: (m: any) => Promise<any>}} | null = null;
|
||||
|
||||
/**
|
||||
* Stable string key derived from the SMTP options `buildTransport` consumes.
|
||||
* Exported as `_internal` so tests can verify that runtime mutations to
|
||||
* `port`/`secure`/`auth` (without a host change) actually invalidate the
|
||||
* cache — a regression caught by Qodo on PR #7753.
|
||||
*/
|
||||
export const smtpTransportKey = (mail: {
|
||||
host?: string | null;
|
||||
port?: number | string | null;
|
||||
secure?: boolean | null;
|
||||
auth?: unknown;
|
||||
}): string => JSON.stringify({
|
||||
host: mail.host ?? null,
|
||||
port: Number(mail.port) || 587,
|
||||
secure: !!mail.secure,
|
||||
auth: mail.auth ?? null,
|
||||
});
|
||||
|
||||
const buildTransport = async (host: string) => {
|
||||
const {default: nodemailer} = await import('nodemailer');
|
||||
return nodemailer.createTransport({
|
||||
host,
|
||||
port: Number(settings.mail.port) || 587,
|
||||
secure: !!settings.mail.secure,
|
||||
auth: settings.mail.auth ?? undefined,
|
||||
});
|
||||
};
|
||||
|
||||
const sendEmailViaSmtp = async (to: string, subject: string, body: string): Promise<void> => {
|
||||
// Etherpad core has no built-in SMTP. PR 1 ships the dedupe machinery without an actual sender;
|
||||
// subsequent PRs can wire nodemailer or rely on a notification plugin.
|
||||
logger.info(`(would send email) to=${to} subject="${subject}"`);
|
||||
void body;
|
||||
const host = settings.mail.host;
|
||||
if (!host || !settings.mail.from) {
|
||||
// Mail not configured. Log so operators running the runbook can confirm
|
||||
// the Notifier fired even without delivery, and the dedupe state still
|
||||
// advances so we don't re-evaluate the same trigger every tick.
|
||||
logger.info(`(would send email) to=${to} subject="${subject}"`);
|
||||
return;
|
||||
}
|
||||
const key = smtpTransportKey(settings.mail);
|
||||
if (!transportCache || transportCache.key !== key) {
|
||||
transportCache = {key, transporter: await buildTransport(host)};
|
||||
}
|
||||
try {
|
||||
await transportCache.transporter.sendMail({
|
||||
from: settings.mail.from, to, subject, text: body,
|
||||
});
|
||||
logger.info(`email sent to=${to} subject="${subject}"`);
|
||||
} catch (err) {
|
||||
// Never throw out of the sender — a transient SMTP failure must not
|
||||
// poison the surrounding updater state machine. The admin UI banner
|
||||
// is still the source of truth for the underlying condition.
|
||||
logger.warn(`email send failed: ${(err as Error).message}`);
|
||||
}
|
||||
};
|
||||
|
||||
const performCheck = async (): Promise<void> => {
|
||||
|
|
@ -95,6 +156,7 @@ const performCheck = async (): Promise<void> => {
|
|||
tier: settings.updates.tier,
|
||||
current,
|
||||
latest: state.latest.version,
|
||||
maintenanceWindow: settings.updates.maintenanceWindow,
|
||||
});
|
||||
if (policy.canNotify) {
|
||||
const decision = decideEmails({
|
||||
|
|
@ -115,6 +177,7 @@ const performCheck = async (): Promise<void> => {
|
|||
}
|
||||
|
||||
// Tier 3 scheduler pass: decide whether to schedule, reschedule, or cancel.
|
||||
// Tier 4 snap-forward to next maintenance window is layered in here too.
|
||||
if (state.latest && scheduler) {
|
||||
const current = getEpVersion();
|
||||
const policy = evaluatePolicy({
|
||||
|
|
@ -123,12 +186,14 @@ const performCheck = async (): Promise<void> => {
|
|||
current,
|
||||
latest: state.latest.version,
|
||||
executionStatus: state.execution.status,
|
||||
maintenanceWindow: settings.updates.maintenanceWindow,
|
||||
});
|
||||
const decision = decideSchedule({
|
||||
state, now, policy,
|
||||
latest: state.latest, current,
|
||||
preApplyGraceMinutes: Number(settings.updates.preApplyGraceMinutes) || 0,
|
||||
adminEmail: settings.adminEmail,
|
||||
maintenanceWindow: policy.canAutonomous ? parseWindow(settings.updates.maintenanceWindow) : null,
|
||||
});
|
||||
if (decision.action === 'schedule') {
|
||||
state.execution = decision.newExecution;
|
||||
|
|
@ -217,6 +282,7 @@ const buildSchedulerApplyDeps = (): ApplyPipelineDeps => ({
|
|||
diskSpaceMinMB: Number(settings.updates.diskSpaceMinMB) || 500,
|
||||
requireSignature: settings.updates.requireSignature,
|
||||
trustedKeysPath: settings.updates.trustedKeysPath,
|
||||
currentNodeVersion: process.versions.node,
|
||||
},
|
||||
{
|
||||
installMethod: detectedMethod,
|
||||
|
|
@ -256,6 +322,26 @@ const buildSchedulerApplyDeps = (): ApplyPipelineDeps => ({
|
|||
requireSignature: settings.updates.requireSignature,
|
||||
trustedKeysPath: settings.updates.trustedKeysPath,
|
||||
}),
|
||||
readTargetEnginesNode: (tagName: string) => new Promise<string | null>((resolve) => {
|
||||
// Read the target tag's package.json *without* mutating the working
|
||||
// tree: `git show <tag>:package.json` writes to stdout only. Treat
|
||||
// any failure (missing tag, missing file, malformed JSON, missing
|
||||
// engines.node) as "no constraint" — preflight already covers
|
||||
// missing-tag separately; we don't want to gate updates on a
|
||||
// package.json shape that older releases predate.
|
||||
const c = spawn('git', ['show', `${tagName}:package.json`],
|
||||
{cwd: settings.root, stdio: ['ignore', 'pipe', 'ignore']});
|
||||
let out = '';
|
||||
c.stdout.on('data', (b) => { out += b.toString(); });
|
||||
c.on('close', () => {
|
||||
try {
|
||||
const pkg = JSON.parse(out);
|
||||
const range = pkg?.engines?.node;
|
||||
resolve(typeof range === 'string' && range.trim().length > 0 ? range : null);
|
||||
} catch { resolve(null); }
|
||||
});
|
||||
c.on('error', () => resolve(null));
|
||||
}),
|
||||
},
|
||||
),
|
||||
createDrainer: (opts) => createDrainer(opts),
|
||||
|
|
@ -299,6 +385,57 @@ const buildSchedulerApplyDeps = (): ApplyPipelineDeps => ({
|
|||
/** Allow the cancel handler to drop the pending scheduler timer. */
|
||||
export const cancelScheduler = (): void => { scheduler?.cancel(); };
|
||||
|
||||
/**
|
||||
* Map an `applyUpdate` outcome to a `FailureOutcome` for the Notifier, or
|
||||
* `null` when the outcome doesn't warrant an admin email. We deliberately
|
||||
* do NOT email on `cancelled` (the admin did it themselves), `busy` (UI
|
||||
* already surfaced the in-flight state), `lock-held`, `invalid-tag`, or
|
||||
* `no-known-latest` (all transient operational conditions surfaced via the
|
||||
* banner already). The terminal `rollback-failed` is emitted separately
|
||||
* from RollbackHandler's own path — applyUpdate's `rolled-back` covers the
|
||||
* auto-recovered case.
|
||||
*/
|
||||
const failureOutcomeFromApplyResult = (
|
||||
outcome: string,
|
||||
): FailureOutcome | null => {
|
||||
if (outcome === 'preflight-failed') return 'preflight-failed';
|
||||
if (outcome === 'rolled-back') return 'rolled-back';
|
||||
return null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Load state, run Notifier.decideOutcomeEmail for the given failure, send
|
||||
* the planned mail (best-effort), and persist the updated dedupe key. Never
|
||||
* throws — a transient SMTP issue must not poison the surrounding apply
|
||||
* flow's bookkeeping.
|
||||
*/
|
||||
export const notifyApplyFailure = async (params: {
|
||||
outcome: FailureOutcome;
|
||||
reason: string;
|
||||
targetTag: string;
|
||||
}): Promise<void> => {
|
||||
try {
|
||||
const state = await loadState(stateFilePath());
|
||||
const decision = decideOutcomeEmail({
|
||||
adminEmail: settings.adminEmail,
|
||||
outcome: params.outcome,
|
||||
reason: params.reason,
|
||||
targetTag: params.targetTag,
|
||||
currentVersion: getEpVersion(),
|
||||
state: state.email,
|
||||
});
|
||||
if (decision.toSend.length === 0) return;
|
||||
for (const e of decision.toSend) {
|
||||
if (settings.adminEmail) {
|
||||
await sendEmailViaSmtp(settings.adminEmail, e.subject, e.body);
|
||||
}
|
||||
}
|
||||
await saveState(stateFilePath(), {...state, email: decision.newState});
|
||||
} catch (err) {
|
||||
logger.warn(`notifyApplyFailure: ${(err as Error).message}`);
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* Timer-fire callback. Re-reads persisted state and re-evaluates policy
|
||||
* *before* invoking applyUpdate so a last-moment cancel, a manual Apply now
|
||||
|
|
@ -318,9 +455,14 @@ const schedulerTriggerApply = async (targetTag: string): Promise<void> => {
|
|||
current: getEpVersion(),
|
||||
latest: state.latest.version,
|
||||
executionStatus: state.execution.status,
|
||||
maintenanceWindow: settings.updates.maintenanceWindow,
|
||||
})
|
||||
: {canNotify: false, canManual: false, canAuto: false, canAutonomous: false, reason: 'no-latest'};
|
||||
const decision = decideTriggerApply({state, targetTag, policy});
|
||||
const window = policy.canAutonomous ? parseWindow(settings.updates.maintenanceWindow) : null;
|
||||
const decision = decideTriggerApply({
|
||||
state, targetTag, policy,
|
||||
now: new Date(), maintenanceWindow: window,
|
||||
});
|
||||
if (decision.action === 'abort') {
|
||||
logger.info(`scheduler fired for ${targetTag} but aborting (${decision.reason})`);
|
||||
return;
|
||||
|
|
@ -332,8 +474,27 @@ const schedulerTriggerApply = async (targetTag: string): Promise<void> => {
|
|||
await saveState(stateFilePath(), {...state, execution: {status: 'idle'}});
|
||||
return;
|
||||
}
|
||||
if (decision.action === 'defer') {
|
||||
// Tier 4: fire-time was outside the window. Re-arm for the next opening
|
||||
// and persist the new scheduledFor so a restart in the gap rehydrates.
|
||||
logger.info(`scheduler deferred ${targetTag} to next maintenance window at ${decision.nextStart}`);
|
||||
const sched = state.execution.status === 'scheduled' ? state.execution : null;
|
||||
if (sched) {
|
||||
await saveState(stateFilePath(), {
|
||||
...state,
|
||||
execution: {...sched, scheduledFor: decision.nextStart},
|
||||
});
|
||||
scheduler?.arm({targetTag, scheduledFor: decision.nextStart});
|
||||
}
|
||||
return;
|
||||
}
|
||||
const result = await applyUpdate({targetTag, deps: buildSchedulerApplyDeps()});
|
||||
logger.info(`scheduler apply finished: ${result.outcome}`);
|
||||
const failureKind = failureOutcomeFromApplyResult(result.outcome);
|
||||
if (failureKind) {
|
||||
const reason = (result as {reason?: string}).reason ?? failureKind;
|
||||
await notifyApplyFailure({outcome: failureKind, reason, targetTag});
|
||||
}
|
||||
} catch (err) {
|
||||
logger.warn(`scheduler apply failed: ${(err as Error).message}`);
|
||||
}
|
||||
|
|
@ -354,6 +515,26 @@ export const expressCreateServer = async (): Promise<void> => {
|
|||
const state = await getCurrentState();
|
||||
pendingVerification = checkPendingVerification(state, getRollbackDeps());
|
||||
|
||||
// Boot-time failure notification. If a previous run produced a failure
|
||||
// outcome whose admin email we haven't already sent (lastFailureKey
|
||||
// dedupe), fire it now. Covers:
|
||||
// - health-check timeout rollback on the previous boot
|
||||
// - crash-loop forced rollback (detected on a later boot)
|
||||
// - preflight-failed where we never got to send (e.g. process kill)
|
||||
// - rollback-failed terminal that the operator hasn't acknowledged
|
||||
// Fire-and-forget — the rest of boot must proceed regardless.
|
||||
const failureOutcome = state.lastResult?.outcome === 'rolled-back' ? 'rolled-back'
|
||||
: state.lastResult?.outcome === 'rollback-failed' ? 'rollback-failed'
|
||||
: state.lastResult?.outcome === 'preflight-failed' ? 'preflight-failed'
|
||||
: null;
|
||||
if (failureOutcome && state.lastResult) {
|
||||
void notifyApplyFailure({
|
||||
outcome: failureOutcome,
|
||||
targetTag: state.lastResult.targetTag,
|
||||
reason: state.lastResult.reason ?? failureOutcome,
|
||||
});
|
||||
}
|
||||
|
||||
// Tier 3: instantiate the scheduler unless updates are entirely disabled.
|
||||
// The runner is purely in-memory — the persisted state file is the source
|
||||
// of truth for "is something scheduled." On `tier: "off"` we explicitly
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import semver from 'semver';
|
||||
import {InstallMethod} from './types';
|
||||
import type {VerifyResult} from './trustedKeys';
|
||||
|
||||
|
|
@ -8,13 +9,20 @@ export type PreflightReason =
|
|||
| 'pnpm-not-found'
|
||||
| 'lock-held'
|
||||
| 'remote-tag-missing'
|
||||
| 'signature-verification-failed';
|
||||
| 'signature-verification-failed'
|
||||
| 'node-engine-mismatch';
|
||||
|
||||
export interface PreflightInput {
|
||||
targetTag: string;
|
||||
diskSpaceMinMB: number;
|
||||
requireSignature: boolean;
|
||||
trustedKeysPath: string | null;
|
||||
/**
|
||||
* Running Node version (typically `process.versions.node`). Threaded
|
||||
* through `input` rather than read from globals so the function stays
|
||||
* fully testable without process mocking.
|
||||
*/
|
||||
currentNodeVersion: string;
|
||||
}
|
||||
|
||||
export interface PreflightDeps {
|
||||
|
|
@ -25,9 +33,16 @@ export interface PreflightDeps {
|
|||
lockHeld: () => Promise<boolean>;
|
||||
remoteHasTag: (tag: string) => Promise<boolean>;
|
||||
verifyTag: () => Promise<VerifyResult>;
|
||||
/**
|
||||
* Returns the `engines.node` field from the target tag's `package.json`
|
||||
* without mutating the working tree. The implementation typically runs
|
||||
* `git show <tag>:package.json` and parses the JSON. Returns `null` if
|
||||
* the field is absent — that's treated as "no constraint, pass".
|
||||
*/
|
||||
readTargetEnginesNode: (tag: string) => Promise<string | null>;
|
||||
}
|
||||
|
||||
export type PreflightResult = {ok: true} | {ok: false; reason: PreflightReason};
|
||||
export type PreflightResult = {ok: true} | {ok: false; reason: PreflightReason; detail?: string};
|
||||
|
||||
const WRITABLE_METHODS: ReadonlySet<Exclude<InstallMethod, 'auto'>> = new Set(['git']);
|
||||
|
||||
|
|
@ -35,6 +50,11 @@ const WRITABLE_METHODS: ReadonlySet<Exclude<InstallMethod, 'auto'>> = new Set(['
|
|||
* Sequenced preflight: each check is fast and reads the world. Order matters —
|
||||
* cheap, definitive failures (install method) run before slow ones (network
|
||||
* tag lookup, gpg). The first failure short-circuits.
|
||||
*
|
||||
* The Node-engine check runs *after* signature verification: we want the
|
||||
* range to come from a trusted tag. It runs *before* anything mutates the
|
||||
* working tree (the executor does the first `git checkout` after we return
|
||||
* ok), so a failure leaves the system exactly as it was — no rollback needed.
|
||||
*/
|
||||
export const runPreflight = async (
|
||||
input: PreflightInput,
|
||||
|
|
@ -50,5 +70,15 @@ export const runPreflight = async (
|
|||
if (!await deps.remoteHasTag(input.targetTag)) return {ok: false, reason: 'remote-tag-missing'};
|
||||
const sig = await deps.verifyTag();
|
||||
if (!sig.ok) return {ok: false, reason: 'signature-verification-failed'};
|
||||
|
||||
const range = await deps.readTargetEnginesNode(input.targetTag);
|
||||
if (range && !semver.satisfies(input.currentNodeVersion, range, {includePrerelease: true})) {
|
||||
return {
|
||||
ok: false,
|
||||
reason: 'node-engine-mismatch',
|
||||
detail: `target requires Node ${range}, running ${input.currentNodeVersion}`,
|
||||
};
|
||||
}
|
||||
|
||||
return {ok: true};
|
||||
};
|
||||
|
|
|
|||
|
|
@ -88,14 +88,16 @@ const isValidVulnerableBelow = (v: unknown): boolean => {
|
|||
|
||||
const isValidEmail = (v: unknown): boolean => {
|
||||
if (!isPlainObject(v)) return false;
|
||||
// graceStartTag was added in Tier 3. Treat as optional for backwards
|
||||
// compatibility with state files written by Tier 1/2 installs; loadState
|
||||
// backfills the missing field to null. If present, must be string|null.
|
||||
// graceStartTag (Tier 3) and lastFailureKey (Tier 4) are both optional for
|
||||
// backwards compatibility with state files written by earlier installs;
|
||||
// loadState backfills missing fields to null. If present, must be string|null.
|
||||
const graceOk = v.graceStartTag === undefined || isStringOrNull(v.graceStartTag);
|
||||
const failOk = v.lastFailureKey === undefined || isStringOrNull(v.lastFailureKey);
|
||||
return isStringOrNull(v.severeAt)
|
||||
&& isStringOrNull(v.vulnerableAt)
|
||||
&& isStringOrNull(v.vulnerableNewReleaseTag)
|
||||
&& graceOk;
|
||||
&& graceOk
|
||||
&& failOk;
|
||||
};
|
||||
|
||||
// Validate the full shape so loadState() actually delivers on its "safely
|
||||
|
|
@ -145,7 +147,11 @@ export const loadState = async (filePath: string): Promise<UpdateState> => {
|
|||
return {
|
||||
...structuredClone(EMPTY_STATE),
|
||||
...partial,
|
||||
email: {...email, graceStartTag: email.graceStartTag ?? null},
|
||||
email: {
|
||||
...email,
|
||||
graceStartTag: email.graceStartTag ?? null,
|
||||
lastFailureKey: email.lastFailureKey ?? null,
|
||||
},
|
||||
execution: partial.execution ?? structuredClone(EMPTY_STATE.execution),
|
||||
bootCount: partial.bootCount ?? 0,
|
||||
lastResult: partial.lastResult ?? null,
|
||||
|
|
|
|||
|
|
@ -2,6 +2,17 @@ export type InstallMethod = 'auto' | 'git' | 'docker' | 'npm' | 'managed';
|
|||
|
||||
export type Tier = 'off' | 'notify' | 'manual' | 'auto' | 'autonomous';
|
||||
|
||||
/**
|
||||
* Tier 4 (autonomous) maintenance window. `start`/`end` are HH:MM (24h) in the
|
||||
* configured `tz`. `end` is exclusive; `end < start` denotes a cross-midnight
|
||||
* window. See `MaintenanceWindow.ts` for the parser/predicate implementation.
|
||||
*/
|
||||
export interface MaintenanceWindow {
|
||||
start: string;
|
||||
end: string;
|
||||
tz: 'local' | 'utc';
|
||||
}
|
||||
|
||||
/** null = up-to-date (or not yet checked); 'severe' = at least one major version behind; 'vulnerable' = matched a vulnerable-below directive. */
|
||||
export type OutdatedLevel = null | 'severe' | 'vulnerable';
|
||||
|
||||
|
|
@ -45,6 +56,13 @@ export interface EmailSendLog {
|
|||
vulnerableNewReleaseTag: string | null;
|
||||
/** Tag of the most recent release for which we sent a Tier 3 `grace-start` email. */
|
||||
graceStartTag: string | null;
|
||||
/**
|
||||
* Dedupe key for `update-rolled-back` / `update-preflight-failed` emails.
|
||||
* Stores the `<tag>:<outcome>` of the last failure we emailed about so a
|
||||
* retry-loop (e.g. repeated `pnpm install` failures on the same release)
|
||||
* doesn't fire one email per attempt. Cleared when the next outcome differs.
|
||||
*/
|
||||
lastFailureKey: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -123,6 +141,7 @@ export const EMPTY_STATE: UpdateState = {
|
|||
vulnerableAt: null,
|
||||
vulnerableNewReleaseTag: null,
|
||||
graceStartTag: null,
|
||||
lastFailureKey: null,
|
||||
},
|
||||
execution: {status: 'idle'},
|
||||
bootCount: 0,
|
||||
|
|
|
|||
|
|
@ -345,11 +345,30 @@ export type SettingsType = {
|
|||
requireSignature: boolean,
|
||||
/** Override the OS keyring location (passed to git verify-tag via $GNUPGHOME). */
|
||||
trustedKeysPath: string | null,
|
||||
/**
|
||||
* Tier 4: nightly window during which the scheduler is allowed to fire.
|
||||
* Null = tier 4 disabled (canAutonomous is denied with reason
|
||||
* `maintenance-window-missing`). Shape validated at boot by `parseWindow`.
|
||||
*/
|
||||
maintenanceWindow: {start: string; end: string; tz: 'local' | 'utc'} | null,
|
||||
},
|
||||
adminOpenAPI: {
|
||||
enabled: boolean,
|
||||
},
|
||||
adminEmail: string | null,
|
||||
/**
|
||||
* SMTP transport for outbound admin notifications (updater + future
|
||||
* features). Null `host` disables outbound mail — the Notifier still runs
|
||||
* and dedupe state is updated, but messages only log `(would send email)`.
|
||||
* `auth` is optional; omit for unauthenticated relays.
|
||||
*/
|
||||
mail: {
|
||||
host: string | null;
|
||||
port: number;
|
||||
secure: boolean;
|
||||
from: string | null;
|
||||
auth: {user: string; pass: string} | null;
|
||||
},
|
||||
getPublicSettings: () => Pick<SettingsType, "title" | "skinVariants"|"randomVersionString"|"skinName"|"toolbar"| "exposeVersion"| "gitVersion" | "enablePadWideSettings" | "enablePluginPadOptions" | "privacyBanner">,
|
||||
}
|
||||
|
||||
|
|
@ -547,6 +566,9 @@ const settings: SettingsType = {
|
|||
diskSpaceMinMB: 500,
|
||||
requireSignature: false,
|
||||
trustedKeysPath: null,
|
||||
// Tier 4: night-window during which the scheduler may fire. Null disables tier 4 only.
|
||||
// Example: { start: "03:00", end: "05:00", tz: "local" } or tz: "utc".
|
||||
maintenanceWindow: null,
|
||||
},
|
||||
/**
|
||||
* Admin OpenAPI document endpoint at /admin/openapi.json.
|
||||
|
|
@ -565,6 +587,19 @@ const settings: SettingsType = {
|
|||
* Null disables outbound mail from the updater.
|
||||
*/
|
||||
adminEmail: null,
|
||||
/**
|
||||
* SMTP transport for outbound admin notifications. Null `host` keeps the
|
||||
* legacy log-only behaviour. Set `host`+`from` (and optionally `auth`) to
|
||||
* deliver via nodemailer. The dependency is lazy-loaded — installs without
|
||||
* a mail.host pay no runtime cost.
|
||||
*/
|
||||
mail: {
|
||||
host: null,
|
||||
port: 587,
|
||||
secure: false,
|
||||
from: null,
|
||||
auth: null,
|
||||
},
|
||||
/**
|
||||
* Whether certain shortcut keys are enabled for a user in the pad
|
||||
*/
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue