Three race conditions in multi_worker_connection_manager could leave
the Redis profile_connections:<id> counter permanently elevated with no
active streams, causing all VOD requests to 503 "All profiles at capacity".
Bug 1 — decrement_active_streams() return value was ignored
All three generator exit paths (normal, GeneratorExit, Exception) called
decrement_active_streams() and unconditionally set decremented = True
regardless of whether the lock was acquired. On lock contention the decrement
was silently skipped, active_streams remained > 0, the subsequent
has_active_streams() check returned True, and _decrement_profile_connections()
was never called. The counter was then stuck until manual DEL.
Fix: add decrement_active_streams_and_check() which performs the decrement and
the "are there remaining streams?" check atomically under a single lock,
eliminating the race window. All three exit paths and the finally block now
use this method and propagate its success/remaining return values.
Bug 2 — non-atomic GET-then-DECR in _decrement_profile_connections()
The previous implementation read the counter with GET then conditionally
called DECR. Two concurrent decrements could both pass the > 0 guard and
both fire, driving the counter to -1. A subsequent _check_and_reserve_profile_slot()
INCR would then produce 0 which passes the <= max_streams check, allowing
an extra stream to bypass the limit on the next request.
Fix: replace GET-then-DECR with a direct DECR (matching the INCR-first
pattern already used by _check_and_reserve_profile_slot) and clamp the
result to 0 if it goes negative.
Bug 3 — has_active_streams() read state without holding the lock
The separate has_active_streams() call after decrement_active_streams()
released its lock left a window where another concurrent stream could
increment active_streams back to 1, causing the profile decrement to be
skipped. This is resolved as a consequence of Bug 1's fix: the new
decrement_active_streams_and_check() method reads active_streams while
the lock is still held, eliminating the window entirely.
Adds tests covering all three scenarios in
apps/proxy/vod_proxy/tests/test_profile_connections.py.
Fixes#1121.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
