feat(metrics): 3 Prometheus counters for scaling dive (#7756) (#7762)

* feat(metrics): expose 3 Prometheus counters for the scaling dive

Per the spec section 6 of #7756: enables the load-test harness to
attribute *where* time goes on the server, not just the gauge headline
(CPU / event-loop / memory) the dive doc starts from.

New /stats/prometheus rows:

- etherpad_pad_users{padId} — gauge, derived from sessioninfos on
  each scrape. Lets the harness confirm the pad it points at actually
  has the expected concurrency.

- etherpad_changeset_apply_duration_seconds — histogram observed
  inside handleUserChanges. Separates "apply path is slow" from
  "fan-out is slow" when latency rises.

- etherpad_socket_emits_total{type} — counter at the broadcast
  emit sites (handleCustomObjectMessage, handleCustomMessage,
  sendChatMessageToPadClients) and inside the NEW_CHANGES per-socket
  loop in updatePadClients. Bucketed by message type so the harness
  can measure the amplification factor of each lever (especially the
  fan-out batching lever).

Metric handles live in a new prom-instruments.ts module rather than
in prometheus.ts itself, so PadMessageHandler can import the
recording helpers without creating a circular dependency
(prometheus.ts already requires PadMessageHandler).

Tests: smoke test verifies recordSocketEmit + recordChangesetApply
move the underlying counters/histogram.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(metrics): address Qodo review — flag-gate, scope histogram, bound label cardinality

Three issues raised on the initial PR:

1. **Feature flag.** Per project compliance rule, new features must
   be behind a flag and disabled by default. Adds
   `settings.scalingDiveMetrics` (default `false`). When off,
   recordSocketEmit() / recordChangesetApply() short-circuit to
   no-ops and the metrics are never even registered with the
   Prometheus register. Enable only when running the
   ether/etherpad-load-test scaling-dive harness.

2. **Histogram scope.** Previously the
   etherpad_changeset_apply_duration_seconds timer wrapped the
   whole handleUserChanges() body — including
   `await exports.updatePadClients(pad)` — so the histogram
   measured apply+fan-out, defeating its stated purpose. Now
   stopped immediately after the apply work (`assert.equal(...rev,
   r)`), before the ACCEPT_COMMIT socket emit and the
   updatePadClients call. Failed applies deliberately don't observe
   so the success-path distribution stays clean.

3. **Label cardinality.** handleCustomMessage was passing the
   user-supplied msgString (an HTTP-API param) directly as the
   `type` label value. A misbehaving API caller could grow
   prom-client's internal label map until OOM. Now bucketed against
   a known-types allowlist; anything outside it lands in `other`.

Tests updated: 5/5 — covers happy path, "other" bucketing of
unknown/unsafe labels, and that the flag-disabled state is a true
no-op.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

src/node/handler/PadMessageHandler.ts

@@ -47,6 +47,7 @@ const accessLogger = log4js.getLogger('access');
 const hooks = require('../../static/js/pluginfw/hooks');
 const stats = require('../stats')
 const assert = require('assert').strict;
+import {recordChangesetApply, recordSocketEmit} from '../prom-instruments';
 import {RateLimiterMemory} from 'rate-limiter-flexible';
 import {ChangesetRequest, PadUserInfo, SocketClientRequest} from "../types/SocketClientRequest";
 import {APool, AText, PadAuthor, PadType} from "../types/PadType";
@@ -117,6 +118,20 @@ function getActivePadCountFromSessionInfos() {
 }
 exports.getActivePadCountFromSessionInfos = getActivePadCountFromSessionInfos;
 
+// Per-pad user counts derived on demand from sessioninfos. Used by
+// prometheus.ts to populate `etherpad_pad_users{padId}` so the #7756
+// scaling-dive harness can confirm the pad it's pointing at actually
+// has the expected concurrency.
+function getPadUsersMap(): Map<string, number> {
+  const out = new Map<string, number>();
+  for (const {padId} of Object.values(sessioninfos)) {
+    if (!padId) continue;
+    out.set(padId, (out.get(padId) ?? 0) + 1);
+  }
+  return out;
+}
+exports.getPadUsersMap = getPadUsersMap;
+
 /**
  * Build a sanitized copy of the plugins registry suitable for sending to the
  * client as part of clientVars. The shape is preserved but each plugin's
@@ -625,6 +640,7 @@ exports.handleCustomObjectMessage = (msg: CustomMessage, sessionID: string) => {
     } else {
       // broadcast to all clients on this pad
       socketio.sockets.in(msg.data.payload.padId).emit('message', msg);
+      recordSocketEmit(msg.data.type);
     }
   }
 };
@@ -645,6 +661,7 @@ exports.handleCustomMessage = (padID: string, msgString:string) => {
     },
   };
   socketio.sockets.in(padID).emit('message', msg);
+  recordSocketEmit(msg.data.type);
 };
 
 /**
@@ -685,6 +702,7 @@ exports.sendChatMessageToPadClients = async (mt: ChatMessage|number, puId: strin
     type: 'COLLABROOM',
     data: {type: 'CHAT_MESSAGE', message},
   });
+  recordSocketEmit('CHAT_MESSAGE');
   await promise;
 };
 
@@ -802,8 +820,14 @@ const handleUserChanges = async (socket:any, message: {
   // if the session was valid when the message arrived in the first place
   if (!thisSession) throw new Error('client disconnected');
 
-  // Measure time to process edit
+  // Measure time to process edit. stats.timer('edits') spans the full handler
+  // (apply + fan-out) for backwards-compat; the new Prometheus histogram below
+  // wraps only the apply path so the scaling-dive harness can distinguish
+  // "apply is slow" from "fan-out is slow". Failed applies do not call the
+  // stopper — leaving the timer un-observed keeps the success-path
+  // distribution clean.
   const stopWatch = stats.timer('edits').start();
+  const stopApplyHistogram = recordChangesetApply();
   try {
     const {data: {baseRev, apool, changeset}} = message;
     if (baseRev == null) throw new Error('missing baseRev');
@@ -905,6 +929,10 @@ const handleUserChanges = async (socket:any, message: {
     // The client assumes that ACCEPT_COMMIT and NEW_CHANGES messages arrive in order. Make sure we
     // have already sent any previous ACCEPT_COMMIT and NEW_CHANGES messages.
     assert.equal(thisSession.rev, r);
+    // End of the apply path. The Prometheus histogram observes here so that
+    // fan-out (socket emit + updatePadClients) does NOT inflate the apply
+    // duration. Failed applies are deliberately not recorded.
+    stopApplyHistogram();
     socket.emit('message', {type: 'COLLABROOM', data: {type: 'ACCEPT_COMMIT', newRev}});
     thisSession.rev = newRev;
     if (newRev !== r) thisSession.time = await pad.getRevisionDate(newRev);
@@ -968,6 +996,7 @@ exports.updatePadClients = async (pad: PadType) => {
       };
       try {
         socket.emit('message', msg);
+        recordSocketEmit('NEW_CHANGES');
       } catch (err:any) {
         messageLogger.error(`Failed to notify user of new revision: ${err.stack || err}`);
         return;