Commit graph

3 commits

Author SHA1 Message Date
PFalko
8bd38ad71c Fix stream ownership bugs causing streams to die after 30-200s
Three interrelated bugs cause TS proxy streams to terminate prematurely
in multi-worker uWSGI deployments:

1. Double ProxyServer instantiation: ProxyConfig.ready() in apps/proxy/apps.py
   calls TSProxyServer() directly, bypassing get_instance(). The subsequent
   TSProxyConfig.ready() call to get_instance() creates a second instance.
   Each instance starts its own cleanup thread, but only one holds channel
   data — the orphaned cleanup thread cannot extend ownership.

2. Redis flushdb() on every client init: RedisClient.get_client() in
   core/utils.py calls flushdb() whenever a new connection is created.
   Celery autoscale workers spawning mid-stream nuke all Redis keys
   including ownership, client records, and channel metadata.

3. No recovery from expired ownership: get_channel_owner() has a TOCTOU
   bug (two separate GET calls in a lambda). extend_ownership() silently
   fails when keys expire. Non-owner cleanup unconditionally kills streams
   even when the worker holds the stream_manager.

Fixes:
- Use TSProxyServer.get_instance() in ProxyConfig.ready()
- Remove flushdb() from Redis client initialization
- Use sentinel pattern for gevent-safe singleton (threading.Lock does not
  work with gevent greenlets)
- Single GET in get_channel_owner() to avoid TOCTOU race
- Re-acquire expired ownership keys in extend_ownership()
- Attempt re-acquisition before cleanup in non-owner path

Relates to #992, #980
2026-02-27 16:40:28 +01:00
dekzter
7351264e8a centralized and lazy-loaded redis client singleton, check for manage.py commands so we don't init proxyservers (redis connection), put manage commmands before starting uwsgi 2025-04-04 16:18:12 -04:00
SergeantPanda
85e41d5def Initial integration. 2025-03-03 18:40:30 -06:00