mirror of https://github.com/ether/etherpad-lite.git synced 2026-07-17 16:47:05 +00:00

feat(test): feature tags + declared-disables contract for opt-out plugins (#7648 )

Plugins that intentionally remove a baseline Etherpad feature
(ep_disable_chat, ep_disable_change_author_name, ep_disable_error_messages,
ep_disable_reset_authorship_colours) currently break core tests for the
removed feature. Their main branches are red, their auto-publish gates
never fire, and Dependabot PRs pile up.

The temptation is to give these plugins an "opt-out of these tests"
flag — but that's a self-serving attestation: a plugin can claim "I
just disable chat, ignore those tests" and quietly break unrelated
functionality on the user's install. etherpad.org/plugins would still
show it green.

This commit introduces a small declared-disables contract that closes
that gap:

  1. Core specs grow @feature:* Playwright tags. Initial set:
     @feature:chat, @feature:username, @feature:clear-authorship,
     @feature:error-gritter. Tags are added test-by-test where the
     test exercises a single feature, so the contract stays precise.

  2. Plugins declare which feature tags they disable in their ep.json:
       { "name": "ep_disable_chat", "disables": ["@feature:chat"], ... }

  3. bin/run-frontend-tests-with-disables.sh enforces the contract via
     two passes:
       - Pass 1 (regression): every test NOT in the disabled list must
         pass. Catches plugins that break things they don't claim to.
       - Pass 2 (honesty): every test that IS in the disabled list
         must FAIL. Catches plugins that lie about disabling features
         they don't actually disable, and stops them from grep-inverting
         arbitrary unrelated tests.

  4. doc/PLUGIN_FEATURE_DISABLES.md walks the design and migration.

The disables list is in ep.json (publicly visible), so etherpad.org/plugins
can surface "this plugin disables: chat" alongside the green CI badge —
users see what they're losing before they install.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-02 10:00:28 +01:00

4.4 KiB

Raw Blame History

Feature-disabling plugins

Some Etherpad plugins exist specifically to remove a baseline feature — ep_disable_chat, ep_disable_change_author_name, ep_disable_error_messages, and so on. When the plugin is installed, the feature it disables is intentionally absent.

This is awkward for CI: the core test suite asserts the disabled feature works. Without coordination, every disable plugin's CI is permanently red, every dependency bump is permanently stuck, and the green/red signal on etherpad.org/plugins becomes meaningless.

To fix that — without giving plugins a free pass to opt out of arbitrary tests — Etherpad uses a small declared-disables contract.

How it works

1. Core specs are tagged by feature

Tests that exercise a single feature carry a Playwright tag like @feature:chat:

test('opens chat, sends a message, makes sure it exists on the page and hides chat', {
  tag: '@feature:chat',
}, async ({page}) => { ... });

test.describe('error sanitization', { tag: '@feature:error-gritter' }, () => { ... });

Tags currently in use:

@feature:chat
@feature:username
@feature:clear-authorship
@feature:error-gritter

2. A plugin declares the features it disables in its `ep.json`

{
  "name": "ep_disable_chat",
  "description": "Disable chat",
  "disables": ["@feature:chat"],
  "parts": [...]
}

The disables field is publicly visible in the plugin's metadata and surfaces on etherpad.org/plugins, so users see the contract before installing.

3. The plugin's CI runs the two-pass test script

bin/run-frontend-tests-with-disables.sh enforces the contract:

# .github/workflows/frontend-tests.yml
- name: Run the frontend tests
  working-directory: ./etherpad-lite/src
  run: ../bin/run-frontend-tests-with-disables.sh -- --project=chromium

The script reads disables (from EP_PLUGIN_DISABLES, an explicit --plugin-ep-json PATH, or auto-detection in plugin_packages/) and runs two passes:

Pass	What it runs	Must
1. Regression	Every spec not tagged with a disabled feature	Pass — proves the plugin doesn't break anything beyond what it claims to disable.
2. Honesty	Every spec that is tagged with a disabled feature	Fail — proves the plugin is genuinely disabling the feature it declares. If those tests pass, the plugin's `disables` claim is wrong.

Both passes have to succeed for CI to be green.

What this catches

Failure mode	Caught by
Plugin breaks an unrelated feature	Pass 1 — its tests aren't excluded, they fail, CI red.
Plugin claims to disable a feature but the feature still works	Pass 2 — tagged tests pass when they should fail, script exits non-zero.
Plugin breaks a feature without declaring it (so etherpad.org/plugins shows it as harmless)	Pass 1 — the feature's tests aren't excluded, they fail, CI red.
Plugin lists a feature in `disables` it doesn't actually disable	Pass 2.

A plugin cannot ship green with broken functionality the user can't see ahead of time.

Adding a new feature tag

When a core spec needs a new feature tag (because a new disable plugin needs to opt out of it):

Pick a stable name: @feature:<area> — short, lowercase, kebab-case, plural where appropriate.
Tag the relevant test() or test.describe() blocks. Multiple tags are fine: tag: ['@feature:chat', '@feature:username'].
Update the list above.
Submit the tag PR before the plugin's PR — the plugin can then declare disables and pass CI.

Adding a new disable plugin

Confirm the feature you're disabling is tagged in core. If not, propose a tag upstream first.
Add "disables": ["@feature:thing"] to your ep.json.
Replace the test invocation in .github/workflows/frontend-tests.yml with a call to bin/run-frontend-tests-with-disables.sh (see ep_disable_chat for a worked example).
Confirm both passes go green locally before opening the PR.

Why not just `--grep-invert`?

The earlier draft of this design just told plugin maintainers to add --grep-invert "<pattern>" in CI. That works for the regression case (pass 1 above), but it lets a careless or malicious plugin silently skip arbitrary tests and present green CI on etherpad.org/plugins despite breaking unrelated functionality. Pass 2 — and the requirement that disables be declared in ep.json rather than inferred from a CI argument — closes that gap.

4.4 KiB Raw Blame History