This update addresses issues related to geventpool database connection leaks by consistently invoking `close_old_connections()` in `finally` blocks within the backup scheduler, plugin repository refresh, and core notification synchronization processes. These changes ensure that idle connections are properly released, preventing resource exhaustion. HLS proxy server has been disabled (hasn't been used). Additionally, the proxy server cleanup logic has been enhanced to safely handle the absence of proxy instances. Tests have been updated to validate these improvements and ensure stability in high-load scenarios.
Three interrelated bugs cause TS proxy streams to terminate prematurely
in multi-worker uWSGI deployments:
1. Double ProxyServer instantiation: ProxyConfig.ready() in apps/proxy/apps.py
calls TSProxyServer() directly, bypassing get_instance(). The subsequent
TSProxyConfig.ready() call to get_instance() creates a second instance.
Each instance starts its own cleanup thread, but only one holds channel
data — the orphaned cleanup thread cannot extend ownership.
2. Redis flushdb() on every client init: RedisClient.get_client() in
core/utils.py calls flushdb() whenever a new connection is created.
Celery autoscale workers spawning mid-stream nuke all Redis keys
including ownership, client records, and channel metadata.
3. No recovery from expired ownership: get_channel_owner() has a TOCTOU
bug (two separate GET calls in a lambda). extend_ownership() silently
fails when keys expire. Non-owner cleanup unconditionally kills streams
even when the worker holds the stream_manager.
Fixes:
- Use TSProxyServer.get_instance() in ProxyConfig.ready()
- Remove flushdb() from Redis client initialization
- Use sentinel pattern for gevent-safe singleton (threading.Lock does not
work with gevent greenlets)
- Single GET in get_channel_owner() to avoid TOCTOU race
- Re-acquire expired ownership keys in extend_ownership()
- Attempt re-acquisition before cleanup in non-owner path
Relates to #992, #980
