test(ci): OS-level sidecar watcher for the Windows silent ELIFECYCLE (#7846)

In-process diagnostics (diagnostics.ts heartbeat at 5 Hz + node-report
snapshots on every beforeEach and heartbeat tick) merged in #7838 and
#7842 reach a hard ceiling: during every captured death window the V8
main isolate is event-loop-starved for 200-400 ms before the process
is externally terminated, so any timer-driven probe (heartbeat,
setTimeout, --report-on-signal handler) never gets serviced and we
have zero JS-visible state from the actual moment of death.

To capture state during the starvation window we need a probe whose
own scheduling does not depend on the dying process's libuv event
loop. This commit adds a tiny bash background loop to the Windows
backend-test steps (both with- and without-plugins). Every 500 ms it
appends:
  - netstat.log: localhost TCP socket state — surfaces TIME_WAIT /
    CLOSE_WAIT accumulation or ephemeral-port exhaustion that the
    in-process libuv handle list can't see (libuv only shows handles
    Node currently knows about; the kernel may hold many more sockets
    in disposal states).
  - tasklist.log: node.exe process state from the Windows OS view
    (handle count, working set, CPU time), independent of whether V8
    is responsive.

Both files land in $GITHUB_WORKSPACE/node-report/ which is already
the artifact-upload target on failure, so they ride for free on
existing infrastructure. The watcher is killed cleanly after `pnpm
test` returns so it never holds the runner open.

On the next captured silent ELIFECYCLE we'll have, for the first
time, a 500 ms-resolution external observation of TCP and process
state across the death window.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
John McLear 2026-05-26 12:06:29 +01:00 committed by GitHub
parent 118d48ca9e
commit c4d2467cb3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194

View file

@ -222,11 +222,48 @@ jobs:
NODE_OPTIONS: "--report-on-fatalerror --report-uncaught-exception --report-on-signal --report-compact --report-directory=${{ github.workspace }}/node-report"
run: |
mkdir -p "${{ github.workspace }}/node-report"
OUT="${{ github.workspace }}/node-report"
# Out-of-process OS-level watcher for the silent-ELIFECYCLE flake.
# In-process diagnostics (diagnostics.ts heartbeat + node-report
# snapshots) showed that during the death window the V8 main
# isolate is starved — heartbeat stops firing entirely, then the
# process is externally terminated, bypassing all JS handlers and
# Node's --report-on-fatalerror. To capture state during that
# starvation we need a process that doesn't depend on the dying
# process's event loop. A bash background loop polling Windows
# OS state every 500 ms gives us that:
# - netstat.log: localhost TCP socket states over time
# (TIME_WAIT/CLOSE_WAIT accumulation, handle exhaustion)
# - tasklist.log: node.exe process handle count, working set,
# CPU time — captured by the OS independent of V8.
# Both logs are appended to node-report/ which already gets
# uploaded as an artifact on failure.
(
while true; do
ts=$(date '+%H:%M:%S.%3N')
{
echo "=== $ts ==="
netstat -an 2>/dev/null | grep -E "TCP\s+(127\.0\.0\.1|\[::1\])" || true
} >> "$OUT/netstat.log"
{
echo "=== $ts ==="
tasklist /v /fi "imagename eq node.exe" /fo csv 2>/dev/null || true
} >> "$OUT/tasklist.log"
sleep 0.5
done
) &
WATCHER_PID=$!
# --exit forces process.exit(failures) after the suite completes,
# closing the post-suite event-loop drain window where Windows +
# Node 24 hard-kills the process. Scoped to Windows so Linux/local
# runs still surface real handle leaks via natural drain.
set +e
pnpm test -- --exit
EXIT=$?
set -e
kill "$WATCHER_PID" 2>/dev/null || true
wait "$WATCHER_PID" 2>/dev/null || true
exit $EXIT
- name: Upload Node diagnostic reports on failure
if: ${{ failure() }}
uses: actions/upload-artifact@v7
@ -319,11 +356,48 @@ jobs:
NODE_OPTIONS: "--report-on-fatalerror --report-uncaught-exception --report-on-signal --report-compact --report-directory=${{ github.workspace }}/node-report"
run: |
mkdir -p "${{ github.workspace }}/node-report"
OUT="${{ github.workspace }}/node-report"
# Out-of-process OS-level watcher for the silent-ELIFECYCLE flake.
# In-process diagnostics (diagnostics.ts heartbeat + node-report
# snapshots) showed that during the death window the V8 main
# isolate is starved — heartbeat stops firing entirely, then the
# process is externally terminated, bypassing all JS handlers and
# Node's --report-on-fatalerror. To capture state during that
# starvation we need a process that doesn't depend on the dying
# process's event loop. A bash background loop polling Windows
# OS state every 500 ms gives us that:
# - netstat.log: localhost TCP socket states over time
# (TIME_WAIT/CLOSE_WAIT accumulation, handle exhaustion)
# - tasklist.log: node.exe process handle count, working set,
# CPU time — captured by the OS independent of V8.
# Both logs are appended to node-report/ which already gets
# uploaded as an artifact on failure.
(
while true; do
ts=$(date '+%H:%M:%S.%3N')
{
echo "=== $ts ==="
netstat -an 2>/dev/null | grep -E "TCP\s+(127\.0\.0\.1|\[::1\])" || true
} >> "$OUT/netstat.log"
{
echo "=== $ts ==="
tasklist /v /fi "imagename eq node.exe" /fo csv 2>/dev/null || true
} >> "$OUT/tasklist.log"
sleep 0.5
done
) &
WATCHER_PID=$!
# --exit forces process.exit(failures) after the suite completes,
# closing the post-suite event-loop drain window where Windows +
# Node 24 hard-kills the process. Scoped to Windows so Linux/local
# runs still surface real handle leaks via natural drain.
set +e
pnpm test -- --exit
EXIT=$?
set -e
kill "$WATCHER_PID" 2>/dev/null || true
wait "$WATCHER_PID" 2>/dev/null || true
exit $EXIT
- name: Upload Node diagnostic reports on failure
if: ${{ failure() }}
uses: actions/upload-artifact@v7