mirror of
https://github.com/ether/etherpad-lite.git
synced 2026-07-18 00:57:55 +00:00
test(ci): OS-level sidecar watcher for the Windows silent ELIFECYCLE (#7846)
In-process diagnostics (diagnostics.ts heartbeat at 5 Hz + node-report snapshots on every beforeEach and heartbeat tick) merged in #7838 and #7842 reach a hard ceiling: during every captured death window the V8 main isolate is event-loop-starved for 200-400 ms before the process is externally terminated, so any timer-driven probe (heartbeat, setTimeout, --report-on-signal handler) never gets serviced and we have zero JS-visible state from the actual moment of death. To capture state during the starvation window we need a probe whose own scheduling does not depend on the dying process's libuv event loop. This commit adds a tiny bash background loop to the Windows backend-test steps (both with- and without-plugins). Every 500 ms it appends: - netstat.log: localhost TCP socket state — surfaces TIME_WAIT / CLOSE_WAIT accumulation or ephemeral-port exhaustion that the in-process libuv handle list can't see (libuv only shows handles Node currently knows about; the kernel may hold many more sockets in disposal states). - tasklist.log: node.exe process state from the Windows OS view (handle count, working set, CPU time), independent of whether V8 is responsive. Both files land in $GITHUB_WORKSPACE/node-report/ which is already the artifact-upload target on failure, so they ride for free on existing infrastructure. The watcher is killed cleanly after `pnpm test` returns so it never holds the runner open. On the next captured silent ELIFECYCLE we'll have, for the first time, a 500 ms-resolution external observation of TCP and process state across the death window. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
118d48ca9e
commit
c4d2467cb3
1 changed files with 74 additions and 0 deletions
74
.github/workflows/backend-tests.yml
vendored
74
.github/workflows/backend-tests.yml
vendored
|
|
@ -222,11 +222,48 @@ jobs:
|
|||
NODE_OPTIONS: "--report-on-fatalerror --report-uncaught-exception --report-on-signal --report-compact --report-directory=${{ github.workspace }}/node-report"
|
||||
run: |
|
||||
mkdir -p "${{ github.workspace }}/node-report"
|
||||
OUT="${{ github.workspace }}/node-report"
|
||||
# Out-of-process OS-level watcher for the silent-ELIFECYCLE flake.
|
||||
# In-process diagnostics (diagnostics.ts heartbeat + node-report
|
||||
# snapshots) showed that during the death window the V8 main
|
||||
# isolate is starved — heartbeat stops firing entirely, then the
|
||||
# process is externally terminated, bypassing all JS handlers and
|
||||
# Node's --report-on-fatalerror. To capture state during that
|
||||
# starvation we need a process that doesn't depend on the dying
|
||||
# process's event loop. A bash background loop polling Windows
|
||||
# OS state every 500 ms gives us that:
|
||||
# - netstat.log: localhost TCP socket states over time
|
||||
# (TIME_WAIT/CLOSE_WAIT accumulation, handle exhaustion)
|
||||
# - tasklist.log: node.exe process handle count, working set,
|
||||
# CPU time — captured by the OS independent of V8.
|
||||
# Both logs are appended to node-report/ which already gets
|
||||
# uploaded as an artifact on failure.
|
||||
(
|
||||
while true; do
|
||||
ts=$(date '+%H:%M:%S.%3N')
|
||||
{
|
||||
echo "=== $ts ==="
|
||||
netstat -an 2>/dev/null | grep -E "TCP\s+(127\.0\.0\.1|\[::1\])" || true
|
||||
} >> "$OUT/netstat.log"
|
||||
{
|
||||
echo "=== $ts ==="
|
||||
tasklist /v /fi "imagename eq node.exe" /fo csv 2>/dev/null || true
|
||||
} >> "$OUT/tasklist.log"
|
||||
sleep 0.5
|
||||
done
|
||||
) &
|
||||
WATCHER_PID=$!
|
||||
# --exit forces process.exit(failures) after the suite completes,
|
||||
# closing the post-suite event-loop drain window where Windows +
|
||||
# Node 24 hard-kills the process. Scoped to Windows so Linux/local
|
||||
# runs still surface real handle leaks via natural drain.
|
||||
set +e
|
||||
pnpm test -- --exit
|
||||
EXIT=$?
|
||||
set -e
|
||||
kill "$WATCHER_PID" 2>/dev/null || true
|
||||
wait "$WATCHER_PID" 2>/dev/null || true
|
||||
exit $EXIT
|
||||
- name: Upload Node diagnostic reports on failure
|
||||
if: ${{ failure() }}
|
||||
uses: actions/upload-artifact@v7
|
||||
|
|
@ -319,11 +356,48 @@ jobs:
|
|||
NODE_OPTIONS: "--report-on-fatalerror --report-uncaught-exception --report-on-signal --report-compact --report-directory=${{ github.workspace }}/node-report"
|
||||
run: |
|
||||
mkdir -p "${{ github.workspace }}/node-report"
|
||||
OUT="${{ github.workspace }}/node-report"
|
||||
# Out-of-process OS-level watcher for the silent-ELIFECYCLE flake.
|
||||
# In-process diagnostics (diagnostics.ts heartbeat + node-report
|
||||
# snapshots) showed that during the death window the V8 main
|
||||
# isolate is starved — heartbeat stops firing entirely, then the
|
||||
# process is externally terminated, bypassing all JS handlers and
|
||||
# Node's --report-on-fatalerror. To capture state during that
|
||||
# starvation we need a process that doesn't depend on the dying
|
||||
# process's event loop. A bash background loop polling Windows
|
||||
# OS state every 500 ms gives us that:
|
||||
# - netstat.log: localhost TCP socket states over time
|
||||
# (TIME_WAIT/CLOSE_WAIT accumulation, handle exhaustion)
|
||||
# - tasklist.log: node.exe process handle count, working set,
|
||||
# CPU time — captured by the OS independent of V8.
|
||||
# Both logs are appended to node-report/ which already gets
|
||||
# uploaded as an artifact on failure.
|
||||
(
|
||||
while true; do
|
||||
ts=$(date '+%H:%M:%S.%3N')
|
||||
{
|
||||
echo "=== $ts ==="
|
||||
netstat -an 2>/dev/null | grep -E "TCP\s+(127\.0\.0\.1|\[::1\])" || true
|
||||
} >> "$OUT/netstat.log"
|
||||
{
|
||||
echo "=== $ts ==="
|
||||
tasklist /v /fi "imagename eq node.exe" /fo csv 2>/dev/null || true
|
||||
} >> "$OUT/tasklist.log"
|
||||
sleep 0.5
|
||||
done
|
||||
) &
|
||||
WATCHER_PID=$!
|
||||
# --exit forces process.exit(failures) after the suite completes,
|
||||
# closing the post-suite event-loop drain window where Windows +
|
||||
# Node 24 hard-kills the process. Scoped to Windows so Linux/local
|
||||
# runs still surface real handle leaks via natural drain.
|
||||
set +e
|
||||
pnpm test -- --exit
|
||||
EXIT=$?
|
||||
set -e
|
||||
kill "$WATCHER_PID" 2>/dev/null || true
|
||||
wait "$WATCHER_PID" 2>/dev/null || true
|
||||
exit $EXIT
|
||||
- name: Upload Node diagnostic reports on failure
|
||||
if: ${{ failure() }}
|
||||
uses: actions/upload-artifact@v7
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue