etherpad-lite/doc/admin/updates.md
John McLear cbe551b432
feat(updater): tier 3 — auto update with grace window (#7607) (#7720)
* feat(updater): scheduled execution state + graceStartTag dedupe field (#7607)

Preparation for Tier 3 of the auto-update subsystem:
- ExecutionStatus gains `scheduled` (targetTag, scheduledFor, startedAt).
- EmailSendLog gains `graceStartTag` for one-shot grace-start email dedupe.
- state validator accepts the new shape, requires per-status fields,
  and backfills graceStartTag=null on a Tier 1/2 state file.

Plus the implementation plan at
docs/superpowers/plans/2026-05-11-auto-update-pr3-tier3-auto.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(updater): decideSchedule pure decision function (#7607)

Adds src/node/updater/Scheduler.ts with the Tier 3 pure decision logic:
- schedules when canAuto + idle/verified/terminal-cleared
- reschedules when a newer tag appears mid-grace
- emits a grace-start email (once per tag) when adminEmail is set
- cancels a stale schedule when policy flips canAuto off
- no-ops during in-flight / terminal states
- clamps preApplyGraceMinutes to [0, 7 days]

Also extends Notifier's EmailKind union with 'grace-start' so the
decision result types correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(updater): scheduler timer runner with arm/cancel (#7607)

Adds createSchedulerRunner to Scheduler.ts:
- arm(): clears any prior timer, sets a fresh one for scheduledFor
- cancel(): clears the pending timer, idempotent
- past scheduledFor → fires with delay=0 (rehydrate after restart-in-grace)
- single-fire-per-arm semantics; armedFor cleared on fire

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(updater): extract apply pipeline shared by HTTP + scheduler (#7607)

Lifts the preflight → drain → execute orchestration out of the
/admin/update/apply HTTP handler into src/node/updater/applyPipeline.ts.
The HTTP handler keeps its 4xx status mapping; the pipeline owns the
state transitions, lock release, drain coordination, and rollback hand-
off. The new ApplyPipelineDeps interface accepts an onAccepted callback
so the HTTP path can still 202 mid-flow while the Tier 3 scheduler path
(next commit) can no-op.

Adds `scheduled` to the apply allowed-entry list so an admin can "Apply
now" during the Tier 3 grace window.

13 vitest cases cover happy / preflight-failed / cancelled / busy /
lock-held / scheduled-entry / rollback / lock-release. Existing 12
mocha integration tests still pass without change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(updater): wire Tier 3 scheduler into boot + performCheck (#7607)

- expressCreateServer instantiates the scheduler runner and rehydrates
  the timer when a prior boot left state.execution = scheduled
- performCheck evaluates decideSchedule after the notifier pass:
  schedule transitions state + sends grace-start email + arms timer;
  cancel-schedule resets to idle + cancels timer
- shutdown cancels the timer
- exposes cancelScheduler() so the cancel endpoint (next commit) can
  drop the pending schedule
- buildSchedulerApplyDeps() supplies the full production-wired pipeline
  deps (preflight, executor, rollback) for the scheduler-triggered apply

Adds tests/backend/specs/updater-scheduler-integration.ts covering
boot-rehydrate fire-on-past and the decision-to-state round-trip.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(updater): cancel handler supports Tier 3 scheduled state (#7607)

POST /admin/update/cancel now accepts execution.status === 'scheduled'
in addition to preflight/draining. The handler calls cancelScheduler()
to drop the pending in-process timer, then transitions state to idle
with lastResult.outcome = 'cancelled' (mirroring the existing pattern).

Adds a Tier 3 integration test that seeds a scheduled state, calls
/admin/update/cancel, and asserts the state machine landed correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(admin): countdown + cancel UI for Tier 3 scheduled updates (#7607)

- store.ts: extend Execution union with the scheduled variant
- UpdatePage.tsx: render countdown panel during scheduled; Apply button
  is relabelled "Apply now" so the admin can skip the remaining grace;
  Cancel button accepts scheduled state
- UpdateBanner.tsx: dedicated scheduled banner with live remaining time
- en.json: new i18n keys (execution.scheduled, banner.scheduled,
  page.scheduled.{title,countdown,apply_now})

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(updater): playwright spec for Tier 3 scheduled UI (#7607)

Three cases against a mocked /admin/update/status:
- countdown panel + Apply now + Cancel render when execution is scheduled
- Cancel button posts /admin/update/cancel and triggers re-fetch
- /admin (banner) shows "Auto-update to <tag> scheduled" copy

Mirrors the existing update-page-actions.spec.ts mock pattern (page.route).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(updater): document Tier 3 auto with grace window (#7607)

- doc/admin/updates.md: flip Tier 3 from "designed, not yet implemented"
  to current; expand preApplyGraceMinutes table row; add a Tier 3
  section explaining schedule / cancel / Apply now / restart-in-grace
  and the grace-start email
- settings.json.template: clarify the preApplyGraceMinutes comment
- CHANGELOG.md: Unreleased entry for Tier 3
- runbook §11: full Tier 3 smoke (happy, cancel, apply-now, restart-in-
  grace, email) plus the additional sign-off checkboxes

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(admin): UpdatePage handles missing execution field; scope spec locator (#7607)

Two CI fixes for PR #7720:

1. UpdatePage.tsx — optional-chain us.execution.status. Integration test
   stubs (update-banner.spec.ts) ship payloads without the Tier 2/3
   execution / lastResult / lockHeld fields; without optional chaining
   on the new scheduled-derivation line the whole page crashed before
   the h1 rendered, breaking the unrelated "renders current version"
   test.

2. update-scheduled.spec.ts — scope the v2.7.2 assertion to the
   .update-scheduled section. The regex was matching three elements
   (banner, countdown panel, changelog link) and tripped Playwright's
   strict-mode locator check.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(updater): address Qodo review (Tier 3 race conditions + tier-off bypass) (#7607)

Four fixes for bugs flagged by Qodo's review of PR #7720:

1. **Tier=off bypasses scheduler** (correctness). expressCreateServer
   used to instantiate the scheduler and rehydrate any persisted
   `scheduled` state regardless of `updates.tier`. A user who set
   `tier: "off"` after a schedule had been persisted would still see
   the timer fire after restart. The boot path now skips scheduler
   creation when tier is off and explicitly clears a stale scheduled
   state to idle (logged so the admin sees what happened).

2. **Timer fire skips state recheck** (reliability). The scheduler's
   timer callback called applyUpdate() directly. Race: admin clicks
   Cancel at the same instant the timer fires, or the tier flips
   during the grace window. Now schedulerTriggerApply re-loads state
   and re-evaluates policy via a new pure decideTriggerApply() helper
   in Scheduler.ts. If state is no longer scheduled (or scheduled for a
   different tag), aborts. If policy now denies auto, persists state
   back to idle and aborts.

3. **Apply-now leaves scheduler timer armed** (correctness). The apply
   endpoint accepts `scheduled` as an entry status but didn't cancel
   the in-process scheduler timer. After the admin clicks Apply now,
   the still-armed timer could later fire and attempt another apply
   (especially if the manual one finishes in preflight-failed, which
   is also an allowed-entry status). Apply handler now calls
   cancelScheduler() when entering from `scheduled`.

4. **scheduledFor not validated as timestamp** (reliability). State
   validator only required scheduledFor / startedAt etc. to be
   non-empty strings; a hand-edited "scheduledFor": "garbage" would
   pass validation and yield NaN delay → immediate fire. The
   validator now requires known timestamp fields to be parseable
   via Date.parse().

Tests: 6 new decideTriggerApply cases + 3 new state.ts validation
cases. 189 vitest pass / 29 mocha integration pass / ts-check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 20:50:06 +01:00

15 KiB
Raw Blame History

Etherpad updates

Etherpad ships with a built-in update subsystem.

  • Tier 1 (notify) — default. A banner appears in the admin UI when a new release is available, and pad users see a discreet badge if the running version is severely outdated or flagged as vulnerable. No execution.
  • Tier 2 (manual click) — admins on a git install can click "Apply update" at /admin/update. Etherpad drains active sessions, runs git fetch / checkout / pnpm install / pnpm run build:ui, and exits with code 75 so a process supervisor restarts it on the new version. Auto-rolls back on failure.
  • Tier 3 (auto with grace window) — opt-in. On a git install, a newly detected release transitions execution state to scheduled and is applied after preApplyGraceMinutes. During the grace window, /admin/update shows a live countdown plus Cancel and Apply now buttons; an admin email (if adminEmail is set) fires once per scheduled tag.
  • Tier 4 (autonomous in maintenance window) — designed, not yet implemented.

Settings

In settings.json:

{
  "updates": {
    "tier": "notify",
    "source": "github",
    "channel": "stable",
    "installMethod": "auto",
    "checkIntervalHours": 6,
    "githubRepo": "ether/etherpad",
    "requireAdminForStatus": false,
    // Tier 2+ knobs (only meaningful at tier "manual" or higher):
    "preApplyGraceMinutes": 0,
    "drainSeconds": 60,
    "rollbackHealthCheckSeconds": 60,
    "diskSpaceMinMB": 500,
    "requireSignature": false,
    "trustedKeysPath": null
  },
  "adminEmail": null
}
Setting Default Notes
updates.tier "notify" One of "off", "notify", "manual", "auto", "autonomous". Higher tiers are silently downgraded if the install method does not allow them. PR 1 only honors "notify" and "off".
updates.source "github" Reserved for future alternative sources. Only "github" is implemented.
updates.channel "stable" Reserved. Stable releases only.
updates.installMethod "auto" One of "auto", "git", "docker", "npm", "managed". Auto-detects via filesystem heuristics. Set explicitly to override.
updates.checkIntervalHours 6 How often to poll GitHub Releases.
updates.githubRepo "ether/etherpad" Override for forks.
updates.requireAdminForStatus false Lock the /admin/update/status endpoint to authenticated admin sessions. Default false matches existing Etherpad behavior — /health already exposes releaseId publicly, and changelog data comes from a public GitHub release. Set true to hide the full update payload from non-admins without disabling the updater (tier: "off" is the heavier opt-out that removes the endpoints entirely).
updates.preApplyGraceMinutes 0 Tier 3 only. Wait this many minutes between detecting a new release and starting the drain so the admin can cancel via /admin/update. 0 applies immediately when allowed. Clamped to [0, 7*24*60] (one week). Has no effect at tier "manual".
updates.drainSeconds 60 How long to broadcast "restart imminent" announcements to active pads before exiting. T-60 / T-30 / T-10 broadcasts fire automatically at the matching offsets within this window.
updates.rollbackHealthCheckSeconds 60 After a fresh boot post-update, give /health this long to come up. If it doesn't, RollbackHandler restores the previous SHA.
updates.diskSpaceMinMB 500 Pre-flight refuses to start an update unless the install volume has at least this many MB free.
updates.requireSignature false When true, refuse updates whose tag is not signed by a trusted key. Verification is done via git verify-tag <tag> against the user's GPG keyring. Default false because Etherpad's release process does not yet sign tags consistently — turning the check on by default would block every Tier 2 update. Set true if you run your own builds or have imported a fork's keys.
updates.trustedKeysPath null Override the keyring location passed to git verify-tag via the $GNUPGHOME env var. Useful when the trusted keys live in a dedicated keyring outside the Etherpad user's home. Only meaningful when requireSignature: true.
adminEmail null Top-level. Contact for admin notifications. Setting it enables the email nudges below.

What "outdated" means

  • severe — running at least one major version behind the latest release.
  • vulnerable — the running version is below a vulnerable-below threshold announced in a recent release. Releases declare these via a <!-- updater: vulnerable-below X.Y.Z --> HTML comment in their body. The newest such directive wins.

Email cadence (when adminEmail is set)

Trigger First send Repeat
Vulnerable status detected Immediate Weekly while still vulnerable
New release announced while still vulnerable Immediate n/a (one event per tag change)
Severely outdated detected Immediate Monthly while still severely outdated
Up to date No email

If adminEmail is unset, the updater never sends mail. The admin UI banner and the pad-side badge still work without it.

PR 1 ships the cadence machinery but does not yet wire a real SMTP transport — emails are logged with (would send email) until a future PR adds the transport. The dedupe state still advances correctly so admins are not bombarded once SMTP is wired.

Pad-side badge

Pad users see no version information by default. A small badge appears in the bottom-right corner only when:

  • The instance is severe (one or more major versions behind), or
  • The instance is vulnerable (running below an announced threshold).

The public endpoint /api/version-status returns only {outdated: null|"severe"|"vulnerable"} — it never leaks the running version, so attackers do not gain a fingerprint vector.

Disabling everything

Set updates.tier to "off". No HTTP request will leave the instance and no banner or badge will render.

Privacy

The version check sends no telemetry. Etherpad fetches the public GitHub Releases API (api.github.com/repos/<repo>/releases/latest) with If-None-Match to be cache-friendly. The only metadata GitHub sees is the same as any other GitHub API client — your IP and a User-Agent: etherpad-self-update header. No instance ID, no version, no identifiers travel upstream.

How install method is detected

updates.installMethod defaults to "auto", which uses these heuristics in order:

  1. /.dockerenv exists → "docker".
  2. .git/ directory present and the install root is writable → "git".
  3. package-lock.json present and writable → "npm".
  4. Otherwise → "managed".

Set the value explicitly if the heuristics get it wrong (e.g., a docker container that bind-mounts a writable git checkout).

In PR 1 (notify only) the install method does not change behavior — every install method gets the banner. From PR 2 onward the install method gates whether the manual-click and automatic tiers can run; only "git" is initially supported for write tiers.

Tier 2 — manual click

Tier 2 is opt-in. To enable: set updates.tier: "manual" and ensure your install was deployed via git (not docker / npm / managed package).

Process supervisor is required

Etherpad applies an update by exiting with code 75 so a process supervisor restarts it. Without a supervisor the instance simply exits and stays down. Common supervisor setups:

  • systemd: add Restart=on-failure + RestartSec=5 to your unit file.
  • pm2: the default behaviour restarts on exit.
  • docker: add --restart=unless-stopped (Tier 2 itself is not supported on docker installs anyway, but if you wrap your own image around a git checkout this applies).

What clicking "Apply update" does

  1. Lock acquirevar/update.lock (PID-based, stale locks reaped automatically).
  2. Pre-flight checks — install method writable, working tree clean, free disk ≥ diskSpaceMinMB, pnpm on PATH, target tag exists at the configured remote, signature verifies (if requireSignature: true). On failure, state goes to preflight-failed with a typed reason; the admin sees a banner and clicks Acknowledge to clear it. No filesystem mutation has happened — nothing to roll back.
  3. DraindrainSeconds window during which T-60 / T-30 / T-10 announcements broadcast to every connected pad and new socket connections are refused. Click Cancel during this window to abort cleanly.
  4. Executegit fetch --tags origin, git checkout <tag>, pnpm install --frozen-lockfile, pnpm run build:ui. Output streams to var/log/update.log (rotated 10 MB × 5).
  5. Exit 75 — the supervisor restarts on the new version.
  6. Health check — RollbackHandler arms a rollbackHealthCheckSeconds timer at boot. When /health responds 200 (i.e., Etherpad reaches the RUNNING state) the timer cancels and the state lands on verified.

Failure modes

What went wrong Resulting state Admin action
Pre-flight check fails preflight-failed Click Acknowledge after fixing the underlying issue (free up disk, clean working tree, etc.).
git fetch / git checkout fails mid-flow rolled-back Informational. The working tree is back where it started; click Acknowledge to clear.
pnpm install or pnpm run build:ui fails rolled-back Same as above. The lockfile and SHA are restored.
/health doesn't come up within rollbackHealthCheckSeconds rolled-back Same — RollbackHandler restores the previous SHA + lockfile and exits 75 again.
The new version crashes at boot more than twice (bootCount > 2) rolled-back Crash-loop guard kicks in regardless of the health-check timer.
Rollback itself fails (e.g., pnpm install errors restoring old lockfile) rollback-failed Manual intervention required. The admin banner switches to a strong red alert. Restore the install by hand, then click Acknowledge to clear the lock and re-allow Tier 2 attempts.

Endpoints

All Tier 2 endpoints require an authenticated admin session (is_admin: true) regardless of requireAdminForStatus.

  • POST /admin/update/apply — start an apply. Returns 202 {accepted, drainEndsAt} once the drain begins. Body unused.
  • POST /admin/update/cancel — cancel during pre-flight or drain. Returns 409 once the executor has begun mutating the filesystem (state machine guarantees we either complete or roll back from there).
  • POST /admin/update/acknowledge — clear a terminal preflight-failed / rolled-back / rollback-failed state back to idle.
  • GET /admin/update/log — tail the last 200 lines of var/log/update.log. Plain text. Used by the in-progress UI.

Signature verification

Default off. Etherpad releases are not yet consistently signed; turning verification on by default would block every Tier 2 update. To enable:

"updates": {
  "requireSignature": true,
  "trustedKeysPath": "/srv/etherpad/keys"   // optional — defaults to the OS user keyring
}

The check shells out to git verify-tag <tag>. The keyring at trustedKeysPath is passed to git via GNUPGHOME. If trustedKeysPath is null (default), the OS user's default keyring is used.

Docker-friendly update flows (future work)

Tier 2 deliberately refuses to apply on installMethod: "docker" because in-container git fetch / pnpm install / build:ui doesn't survive a container restart — the orchestrator brings the container back up on the same image tag and the work is lost. Docker installs stay on Tier 1 (banner + version status) for now.

Tier 3 — auto with grace window

Tier 3 builds on Tier 2 by scheduling the apply automatically when a new release is detected. The same git fetch / checkout / pnpm install / build:ui / exit 75 pipeline runs — only the trigger changes.

To enable, on a git install: set updates.tier: "auto" and (optionally) updates.preApplyGraceMinutes to the grace duration you want.

What happens when a new release lands

  1. The periodic version checker (updates.checkIntervalHours) hits GitHub Releases.
  2. If policy.canAuto is true (install is git, no terminal rollback-failed state, tier is "auto" or "autonomous"), the scheduler transitions execution.status to scheduled with scheduledFor = now + preApplyGraceMinutes.
  3. The schedule is persisted to var/update-state.json, so an Etherpad restart inside the grace window rehydrates the timer rather than losing the schedule.
  4. /admin/update shows a live countdown panel plus two buttons:
    • CancelPOST /admin/update/cancel returns the state to idle and drops the in-process timer.
    • Apply nowPOST /admin/update/apply skips the remaining grace; the regular Tier 2 pipeline runs immediately.
  5. When the timer fires, the scheduler runs the exact same pipeline as a manual Tier 2 click: pre-flight → drain → execute → exit 75.

Re-scheduling and stale state

  • If a newer release tag appears while a schedule is pending, the scheduler re-arms the timer for the new tag. The email.graceStartTag dedupe field guards against duplicate grace-start notifications.
  • If updates.tier is flipped back to "manual" or "notify" while a schedule is pending, the next periodic check cancels the schedule (state back to idle).
  • rollback-failed disables Tier 3 globally. The admin must POST /admin/update/acknowledge (or visit /admin/update and click Acknowledge) before any further auto-schedules are armed. Tier 2 manual click stays available because the admin click is the intervention the terminal state requires.

Email (adminEmail set)

A single grace-start notification fires per scheduled tag:

[Etherpad] Auto-update scheduled for 2.7.2

with the scheduledFor timestamp. Etherpad core does not yet wire SMTP; the message logs as (would send email) until a future PR adds a transport. Cadence and dedupe still update correctly.

The right way to give docker admins an in-product Apply button is to delegate to the orchestrator rather than mutate the container. Two patterns to consider in a follow-up PR:

  • Instructions-only. When the page detects installMethod: docker and a newer release exists, swap the policy-denial copy for actionable instructions (docker pull etherpad/etherpad:<tag> for plain docker; docker compose pull && docker compose up -d for compose). Cheap, no new attack surface.
  • Deploy webhook. New setting updates.dockerWebhook. When set, the Apply button on a docker install POSTs to the configured URL and trusts the orchestrator (Render / Railway / Fly / Portainer / Coolify / GitHub Actions — they all expose redeploy webhooks) to do the actual pull-and-recreate.

Direct Docker-socket access (mount /var/run/docker.sock into the container) is out of scope — anyone who escapes the Etherpad process via that socket gets root on the host. Admins who want fully autonomous docker updates should run Watchtower alongside Etherpad rather than bake equivalent privilege into Etherpad itself.