- Removed dead VOD URL routes: `VODPlaylistView` (playlist generation), `VODPositionView` (position tracking), and the class-based `VODStatsView` (replaced by the existing function-based `vod_stats` view).
- Removed dead `updateVODPosition()` API method from `frontend/src/api.js`, which called the now-removed position tracking endpoint.
Three race conditions in multi_worker_connection_manager could leave
the Redis profile_connections:<id> counter permanently elevated with no
active streams, causing all VOD requests to 503 "All profiles at capacity".
Bug 1 — decrement_active_streams() return value was ignored
All three generator exit paths (normal, GeneratorExit, Exception) called
decrement_active_streams() and unconditionally set decremented = True
regardless of whether the lock was acquired. On lock contention the decrement
was silently skipped, active_streams remained > 0, the subsequent
has_active_streams() check returned True, and _decrement_profile_connections()
was never called. The counter was then stuck until manual DEL.
Fix: add decrement_active_streams_and_check() which performs the decrement and
the "are there remaining streams?" check atomically under a single lock,
eliminating the race window. All three exit paths and the finally block now
use this method and propagate its success/remaining return values.
Bug 2 — non-atomic GET-then-DECR in _decrement_profile_connections()
The previous implementation read the counter with GET then conditionally
called DECR. Two concurrent decrements could both pass the > 0 guard and
both fire, driving the counter to -1. A subsequent _check_and_reserve_profile_slot()
INCR would then produce 0 which passes the <= max_streams check, allowing
an extra stream to bypass the limit on the next request.
Fix: replace GET-then-DECR with a direct DECR (matching the INCR-first
pattern already used by _check_and_reserve_profile_slot) and clamp the
result to 0 if it goes negative.
Bug 3 — has_active_streams() read state without holding the lock
The separate has_active_streams() call after decrement_active_streams()
released its lock left a window where another concurrent stream could
increment active_streams back to 1, causing the profile decrement to be
skipped. This is resolved as a consequence of Bug 1's fix: the new
decrement_active_streams_and_check() method reads active_streams while
the lock is still held, eliminating the window entirely.
Adds tests covering all three scenarios in
apps/proxy/vod_proxy/tests/test_profile_connections.py.
Fixes#1121.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix multiple VOD proxy connection counter leak paths that caused
profile_connections to remain permanently elevated after client
disconnects, leaving profiles locked at capacity.
Changes:
- Fix get_stream() race condition: save state under lock with fresh active_streams to prevent overwriting concurrent decrements during scrubbing/seeking (root cause of counter stuck after timeshift)
- Add early active_streams increment in session reuse path to prevent cleanup race between GeneratorExit and new request
- Add rollback on get_stream() returning None (range not satisfiable)
- Add profile_id fallback to closure variable when Redis state is deleted before generator cleanup runs
- Change increment/decrement return types and add warning logs for lock acquisition and state failures
- Clean up debug traceback logging from increment/decrement methods
Three fixes:
- TOCTOU: Replace GET-check-INCR with atomic INCR-first-then-check pattern in both connection managers to prevent concurrent requests exceeding max_streams
- Immediate DECR: Decrement profile counter directly in stream generator exit paths (normal completion, client disconnect, error) instead of deferring to daemon thread cleanup which may never execute
- Rollback: Decrement profile counter on create_connection() failure so the reserved slot is released
Fixes#962
- multi_worker_connection_manager.py: Add missing count > 0 guard to multi-worker remove_connection()
This method doesn't appear to be called anywhere but added the guard in case it is put back into service
- multi_worker_connection_manager.py: Error handler at line 1042: removed connection_manager=self from cleanup() call since the DECR was already done at line 1036. Prevents double-DECR.
- connection_manager.py: Added connection_key_exists check before calling remove_connection(). The fallback DECR from session data now only fires when the connection tracking key has already expired. Prevents double-DECR.
Add support for authentication when connecting to external Redis instances in modular deployment mode. This enables secure Redis deployments using either password-only authentication (Redis <6) or
username + password authentication (Redis 6+ ACL).
Changes:
- Add REDIS_PASSWORD and REDIS_USER environment variables
- Implement URL encoding for special characters in passwords
- Update all Redis connection points to support auth
- Add comprehensive documentation and examples to docker-compose.yml
- Maintains full backward compatibility (empty defaults = no auth)
All authentication mechanisms have been fully tested including pasword-only authentication, Redis 6+ ACL authentication with username + password, volume-mounted configuration files, and special character handling.
Existing deployments are not effected, authentication support is entirely opt-in using docker-compose.
Also, acknowledging the fact that the docker-compose file for modular deployments has been getting out of hand with auth support, the docker-compose.yml file was re-formatted for better visibility in configuration. This seemed like a better way to go than mandating a .env file.
Replace hardcoded localhost:6379 values throughout codebase with environment-based configuration. Add REDIS_PORT support and allow REDIS_URL override for external Redis services. Configure Celery broker/result backend to use Redis settings with environment variable overrides.
Closes#762
