fix(proxy): enhance channel shutdown management and client reconnection handling
Some checks are pending
CI Pipeline / prepare (push) Waiting to run
CI Pipeline / docker (amd64, ubuntu-24.04) (push) Blocked by required conditions
CI Pipeline / docker (arm64, ubuntu-24.04-arm) (push) Blocked by required conditions
CI Pipeline / create-manifest (push) Blocked by required conditions
Build and Push Multi-Arch Docker Image / build-and-push (push) Waiting to run

- Improved the handling of channel shutdown delays to ensure they reset correctly upon client reconnections, preventing premature shutdowns.
- Implemented logic to cancel pending shutdowns when clients reconnect, ensuring proper resource management and preventing client disconnect issues across uWSGI workers.
- Enhanced logging for shutdown processes and client management to provide clearer insights during channel operations.
This commit is contained in:
SergeantPanda 2026-06-15 18:53:49 -05:00
parent 7989fa3673
commit 32442e0b64
7 changed files with 378 additions and 23 deletions

View file

@ -44,13 +44,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **Live proxy channels could remain running with no clients after disconnect.** `stop_channel` called `log_system_event('channel_stop')` synchronously before local cleanup and Redis key deletion; Connect integrations or DB event writes on that path could block teardown indefinitely while the cleanup thread kept refreshing metadata TTL for a stale `stream_buffers` entry. Teardown now signals shutdown (`buffer.stopping`), runs model `release_stream()` and Redis key deletion, releases ownership, stops ffmpeg/output managers and local buffers, and only then logs the stop event asynchronously. Metadata TTL refresh skips channels mid-shutdown and non-owned channels; a stuck-stop watchdog forces cleanup if `stop_channel` does not return within ~10s (or 2× `channel_shutdown_delay`, whichever is greater). Stream buffers ignore late `add_chunk()` writes once shutdown begins (`buffer.stopping` set at teardown start).
- **Client reconnect during channel teardown could wedge the channel across uWSGI workers.** The disconnect path called `proxy_server.stop_channel()` directly, so other workers were not notified and reconnecting clients could attach while ownership was released on the owner worker. The cleanup thread then re-acquired expired ownership and looped on orphaned clients while the upstream connection stayed open. All lifecycle stops now go through coordinated Redis teardown (`channel_stopping` flag, metadata `stopping` state, and `CHANNEL_STOP` pubsub). New stream requests are rejected with `503 Retry-After` during pending shutdown or active teardown; `initialize_channel()` and ownership re-acquisition refuse to proceed in those states; non-owner workers clean local resources only on pubsub; and orphaned-client sweeps force cleanup when teardown is already active. (Fixes #1342)
- **Client reconnect during channel teardown could wedge the channel across uWSGI workers.** The disconnect path called `proxy_server.stop_channel()` directly, so other workers were not notified and reconnecting clients could attach while ownership was released on the owner worker. The cleanup thread then re-acquired expired ownership and looped on orphaned clients while the upstream connection stayed open. All lifecycle stops now go through coordinated Redis teardown (`channel_stopping` flag, metadata `stopping` state, and `CHANNEL_STOP` pubsub). New stream requests are rejected with `503 Retry-After` only during active teardown (not during the post-disconnect shutdown delay grace period); `initialize_channel()` and ownership re-acquisition refuse to proceed in those states; non-owner workers clean local resources only on pubsub; and orphaned-client sweeps force cleanup when teardown is already active. (Fixes #1342)
- **Orphaned upstream threads kept writing Redis after teardown.** Partial cleanup could remove a channel from `stream_managers` while the `stream-{uuid}` OS thread kept running, so orphan metadata sweeps only deleted Redis keys and chunks/metadata were immediately recreated (bare `total_bytes` hash with TTL -1). Stream managers now stop upstream when ownership is lost, orphan cleanup stops local ffmpeg/stream threads even when registry entries are gone (`_live_stream_managers` tracks managers until the thread exits), and forced recovery stops processes before Redis deletion instead of dropping dict entries without calling `stop()`.
- **Rapid channel switching could leave bare `buffer:index` keys in Redis.** `buffer.stop()` flushed a final partial chunk via `INCR` during teardown, creating a persistent index key (TTL -1) after chunk keys expired; under load overlapping disconnect and cleanup-thread stops could race on `_stop_local_stream_activity`, blocking on ffmpeg stderr join before `_clean_redis_keys` ever ran. Teardown no longer writes to Redis from `buffer.stop()`, disconnect uses coordinated stop only (no duplicate upstream stop), model `release_stream()` runs before Redis keys are scanned/deleted (so `profile_connections` counters are not left stuck), ownership is released after Redis cleanup but before the blocking local ffmpeg stop, and a `finally` block guarantees Redis cleanup if local teardown raises.
- **Live proxy could leak geventpool DB checkouts outside HTTP requests.** With `django-db-geventpool`, `connection.close()` returns handles to the per-worker pool (`MAX_CONNS=8`); without it, greenlets and OS threads keep connections checked out until the pool blocks on `pool.get()`. Proxy paths that touch the ORM outside Django's request cycle now call `close_old_connections()` after work completes: `_clean_redis_keys()`, profile lookup in `_establish_transcode_connection()` before spawning ffmpeg, and fMP4 client disconnect cleanup when releasing streams (TS disconnect relies on `log_system_event()`).
- **System events could block callers on Connect and plugin handlers.** `log_system_event()` ran Connect subscriptions and plugin `"events"` hooks synchronously after the DB write, so slow integrations could stall live-proxy and streaming paths even when inserting the event row was fast. Connect/plugin dispatch now runs on a separate gevent when the hub is available (synchronously in Celery workers without a hub). `log_system_event()` and `dispatch_event_system()` also call `close_old_connections()` in `finally` blocks so integration work does not leave pool slots checked out on the caller or dispatch greenlet.
- **Plugins could leak geventpool DB checkouts after UI or Connect event runs.** Third-party plugins run inline on uWSGI greenlets (manual actions and Connect `"events"` hooks) with no guaranteed connection cleanup at the plugin boundary. `PluginManager.run_action()` and `stop_plugin()` now call `close_old_connections()` in a `finally` block so each action returns its pool slot whether the plugin succeeds or raises.
- **VOD proxy could leak geventpool DB checkouts during playback and stats updates.** `stream_vod()` ran ORM lookups for content and M3U profiles, then returned a long-lived `StreamingHttpResponse` without releasing the checkout, so each movie/episode stream could hold a pool slot for its full duration. Background VOD stats refresh (`build_vod_stats_data()`, triggered on start/stop and by the admin stats API) also queried movie, episode, and profile rows from daemon threads with no cleanup. `stream_vod()` now calls `close_old_connections()` before handing off to the streaming generator, and `build_vod_stats_data()` releases its checkout in a `finally` block.
- **Channel shutdown delay did not reset after a reconnect within the grace period.** `handle_client_disconnect()` used a fixed `gevent.sleep(shutdown_delay)` from the first last-client disconnect. If a client reconnected and disconnected again during the delay, an earlier disconnect handler could still stop the channel on the original timer instead of waiting the full delay from the latest disconnect. Shutdown now polls Redis (`last_client_disconnect` timestamp and client count) so concurrent disconnect handlers and multi-worker reconnects always honour the latest disconnect time.
- **Zombie ffmpeg could survive owner-lock expiry and orphan Redis sweeps.** When the 30s owner lock lapsed under single-worker load, disconnect handling treated the worker as non-owner so coordinated stop never ran, while ffmpeg kept writing and the orphan sweeper only deleted Redis keys (recreating them immediately). Disconnect now re-acquires ownership when local upstream is still active, stops locally when the last client leaves without a lock, orphan cleanup stops local ffmpeg before Redis deletion via `_has_local_upstream_activity`, re-init stops lingering upstream before starting a duplicate thread, and `is_channel_teardown_active` includes channels mid-`stop_channel` on this worker so rapid reconnect gets 503 during teardown.
- **Stale `channel_stream` Redis keys after a channel stopped could skip connection accounting on retune.** On `dev`, `get_stream()` reused any existing `channel_stream` / `stream_profile` assignment without reserving a new slot. If those keys were left behind after a stop, the next tune-in could reach the provider without incrementing Redis counters. `get_stream()` now releases stale assignments when proxy metadata shows the channel is inactive, then reserves fresh slots.
- **EPG auto-match reliability fixes.**

View file

@ -41,6 +41,7 @@ def _configure_ownership_pipeline(
def _mock_proxy_server(redis_client=None):
server = MagicMock()
server.redis_client = redis_client or MagicMock()
server._stopping_channels = set()
return server
@ -72,7 +73,47 @@ class ChannelTeardownAvailabilityTests(TestCase):
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertTrue(ChannelService.is_shutdown_pending(CHANNEL_ID))
self.assertTrue(ChannelService.is_channel_unavailable_for_new_clients(CHANNEL_ID))
self.assertFalse(ChannelService.is_channel_unavailable_for_new_clients(CHANNEL_ID))
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_cancel_pending_shutdown_clears_disconnect_key(self, mock_get_instance, mock_delay):
mock_delay.return_value = 30
redis = MagicMock()
redis.exists.side_effect = lambda key: "last_client_disconnect" in key
redis.get.return_value = None
redis.hget.return_value = ChannelState.ACTIVE.encode()
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertTrue(ChannelService.cancel_pending_shutdown(CHANNEL_ID))
redis.delete.assert_any_call(RedisKeys.last_client_disconnect(CHANNEL_ID))
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_cancel_pending_shutdown_skips_during_active_stop(self, mock_get_instance, mock_delay):
mock_delay.return_value = 30
redis = MagicMock()
redis.exists.return_value = True
server = _mock_proxy_server(redis)
server._stopping_channels = {CHANNEL_ID}
mock_get_instance.return_value = server
self.assertFalse(ChannelService.cancel_pending_shutdown(CHANNEL_ID))
redis.delete.assert_not_called()
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
def test_cancel_pending_shutdown_skips_real_teardown_without_grace(self, mock_get_instance, mock_delay):
mock_delay.return_value = 30
redis = MagicMock()
redis.exists.side_effect = lambda key: "stopping" in key
redis.get.return_value = None
redis.hget.return_value = ChannelState.STOPPING.encode()
mock_get_instance.return_value = _mock_proxy_server(redis)
self.assertFalse(ChannelService.cancel_pending_shutdown(CHANNEL_ID))
redis.hset.assert_not_called()
redis.delete.assert_not_called()
@patch("apps.proxy.live_proxy.services.channel_service.ConfigHelper.channel_shutdown_delay")
@patch("apps.proxy.live_proxy.services.channel_service.ProxyServer.get_instance")
@ -86,6 +127,28 @@ class ChannelTeardownAvailabilityTests(TestCase):
self.assertFalse(ChannelService.is_shutdown_pending(CHANNEL_ID))
class ClientManagerAddClientTests(TestCase):
@patch("apps.proxy.live_proxy.services.channel_service.ChannelService.cancel_pending_shutdown", return_value=False)
@patch("apps.proxy.live_proxy.client_manager.send_websocket_update")
def test_add_client_stores_ip_and_user_agent_in_redis(self, _mock_ws, _mock_cancel):
from apps.proxy.live_proxy.client_manager import ClientManager
redis = MagicMock()
cm = ClientManager(CHANNEL_ID, redis_client=redis, worker_id="worker-1")
cm.proxy_server = MagicMock()
result = cm.add_client(
"client-1",
"10.0.2.163",
user_agent="VLC/3.0.21",
)
self.assertEqual(result, 1)
mapping = redis.hset.call_args[1]["mapping"]
self.assertEqual(mapping["ip_address"], "10.0.2.163")
self.assertEqual(mapping["user_agent"], "VLC/3.0.21")
class LocalStreamActivityTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
@ -509,6 +572,104 @@ class HandleClientDisconnectUpstreamFirstTests(TestCase):
mock_coordinated.assert_called_once_with(CHANNEL_ID)
class ShutdownDelayWaitTests(TestCase):
def _make_server(self):
with patch("apps.proxy.live_proxy.server.RedisClient.get_client", return_value=MagicMock()):
with patch.object(ProxyServer, "_start_cleanup_thread"):
server = ProxyServer()
server.redis_client = MagicMock()
server.redis_client.scard.return_value = 0
return server
@patch("apps.proxy.live_proxy.server.gevent.sleep")
@patch("apps.proxy.live_proxy.server.time.time", return_value=1000.0)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_aborts_when_disconnect_key_deleted(self, _mock_delay, _mock_time, mock_sleep):
server = self._make_server()
server.redis_client.get.side_effect = [b"1000.0", None]
result = server._wait_for_shutdown_delay(CHANNEL_ID)
self.assertFalse(result)
self.assertGreaterEqual(mock_sleep.call_count, 1)
@patch("apps.proxy.live_proxy.server.gevent.sleep")
@patch("apps.proxy.live_proxy.server.time.time", return_value=1000.0)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_aborts_when_clients_reconnect(self, _mock_delay, _mock_time, _mock_sleep):
server = self._make_server()
server.redis_client.get.return_value = b"1000.0"
server.redis_client.scard.side_effect = [0, 1]
result = server._wait_for_shutdown_delay(CHANNEL_ID)
self.assertFalse(result)
server.redis_client.delete.assert_called_with(
RedisKeys.last_client_disconnect(CHANNEL_ID)
)
@patch("apps.proxy.live_proxy.server.gevent.sleep")
@patch("apps.proxy.live_proxy.server.time.time")
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_timer_resets_when_disconnect_timestamp_updated(
self, _mock_delay, mock_time, mock_sleep
):
server = self._make_server()
disconnect_key = RedisKeys.last_client_disconnect(CHANNEL_ID)
current_time = [1000.0]
disconnect_timestamp = [1000.0]
poll_count = [0]
mock_time.side_effect = lambda: current_time[0]
def get_side_effect(key):
poll_count[0] += 1
if poll_count[0] >= 3:
disconnect_timestamp[0] = 1020.0
if key == disconnect_key:
return str(disconnect_timestamp[0]).encode()
return None
server.redis_client.get.side_effect = get_side_effect
def advance_sleep(duration):
current_time[0] += duration
mock_sleep.side_effect = advance_sleep
result = server._wait_for_shutdown_delay(CHANNEL_ID)
self.assertTrue(result)
self.assertGreaterEqual(current_time[0], 1050.0)
@patch.object(ProxyServer, "_coordinated_stop_channel")
@patch.object(ProxyServer, "_wait_for_shutdown_delay", return_value=True)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_handle_client_disconnect_uses_polling_wait(
self, _mock_delay, mock_wait, mock_coordinated
):
server = HandleClientDisconnectUpstreamFirstTests()._make_server()
server.redis_client.get.return_value = b"1700000000.0"
server.handle_client_disconnect(CHANNEL_ID)
mock_wait.assert_called_once_with(CHANNEL_ID)
mock_coordinated.assert_called_once_with(CHANNEL_ID)
@patch.object(ProxyServer, "_coordinated_stop_channel")
@patch.object(ProxyServer, "_wait_for_shutdown_delay", return_value=False)
@patch("apps.proxy.live_proxy.server.ConfigHelper.channel_shutdown_delay", return_value=30)
def test_handle_client_disconnect_skips_stop_when_wait_aborted(
self, _mock_delay, _mock_wait, mock_coordinated
):
server = HandleClientDisconnectUpstreamFirstTests()._make_server()
server.redis_client.get.return_value = b"1700000000.0"
server.handle_client_disconnect(CHANNEL_ID)
mock_coordinated.assert_not_called()
class InitWaitAbortTests(TestCase):
def _make_generator(self):
from apps.proxy.live_proxy.output.ts.generator import StreamGenerator

View file

@ -256,6 +256,14 @@ class ClientManager:
# Store in Redis
if self.redis_client:
from .services.channel_service import ChannelService
if ChannelService.cancel_pending_shutdown(self.channel_id):
logger.info(
f"Cancelled pending shutdown for channel {self.channel_id} "
f"(client {client_id} reconnected)"
)
self.redis_client.hset(client_key, mapping=client_data)
self.redis_client.expire(client_key, self.client_ttl)
@ -302,7 +310,10 @@ class ClientManager:
return len(self.clients)
except Exception as e:
logger.error(f"Error adding client {client_id}: {e}")
logger.error(f"Error adding client {client_id}: {e}", exc_info=True)
with self.lock:
self.clients.discard(client_id)
self._registered_clients.discard(client_id)
return False
def remove_client(self, client_id):
@ -337,7 +348,8 @@ class ClientManager:
if remaining == 0:
logger.warning(f"Last client removed: {client_id} - channel may shut down soon")
disconnect_key = RedisKeys.last_client_disconnect(self.channel_id)
self.redis_client.setex(disconnect_key, 60, str(time.time()))
ttl = max(int(ConfigHelper.channel_shutdown_delay() * 2), 60)
self.redis_client.setex(disconnect_key, ttl, str(time.time()))
self._notify_owner_of_activity()

View file

@ -348,7 +348,11 @@ class FMP4StreamGenerator:
client_count = proxy_server.client_managers[
self.channel_id
].get_total_client_count()
if client_count <= 1 and proxy_server.am_i_owner(self.channel_id):
if (
client_count <= 1
and proxy_server.am_i_owner(self.channel_id)
and ConfigHelper.channel_shutdown_delay() <= 0
):
try:
try:
obj = Channel.objects.get(uuid=self.channel_id)

View file

@ -603,7 +603,8 @@ class StreamGenerator:
if self.channel_id in proxy_server.client_managers:
client_count = proxy_server.client_managers[self.channel_id].get_total_client_count()
# Pool slots are global; the last client on any worker must release.
if client_count <= 1:
# During shutdown_delay, keep the slot until coordinated stop runs.
if client_count <= 1 and ConfigHelper.channel_shutdown_delay() <= 0:
try:
try:
obj = Channel.objects.get(uuid=self.channel_id)

View file

@ -905,6 +905,81 @@ class ProxyServer:
logger.error(f"Error cleaning zombie channel {channel_id}: {e}", exc_info=True)
return False
@staticmethod
def _shutdown_disconnect_ttl():
delay = ConfigHelper.channel_shutdown_delay()
return max(int(delay * 2), 60)
def _wait_for_shutdown_delay(self, channel_id):
"""
Wait until shutdown_delay has elapsed since the Redis disconnect
timestamp. Returns False if clients reconnect or the timer is cancelled.
Uses Redis state so concurrent disconnect handlers and multi-worker
reconnects always honour the latest last-client disconnect time.
"""
if not self.redis_client:
return True
shutdown_delay = ConfigHelper.channel_shutdown_delay()
if shutdown_delay <= 0:
return True
disconnect_key = RedisKeys.last_client_disconnect(channel_id)
client_set_key = RedisKeys.clients(channel_id)
poll_interval = 1.0
logger.info(
f"Waiting up to {shutdown_delay}s before stopping channel {channel_id}..."
)
while True:
total = self.redis_client.scard(client_set_key) or 0
if total > 0:
logger.info(
f"New clients connected during shutdown delay for "
f"{channel_id} - aborting shutdown"
)
self.redis_client.delete(disconnect_key)
return False
disconnect_value = self.redis_client.get(disconnect_key)
if not disconnect_value:
logger.info(
f"Shutdown delay cancelled for {channel_id} - aborting shutdown"
)
return False
try:
if isinstance(disconnect_value, bytes):
disconnect_value = disconnect_value.decode()
disconnect_time = float(disconnect_value)
except (ValueError, TypeError):
logger.warning(
f"Invalid disconnect timestamp for {channel_id}, aborting wait"
)
return False
elapsed = time.time() - disconnect_time
if elapsed >= shutdown_delay:
total = self.redis_client.scard(client_set_key) or 0
if total > 0:
logger.info(
f"Clients connected at end of shutdown delay for "
f"{channel_id} - aborting shutdown"
)
self.redis_client.delete(disconnect_key)
return False
return True
elapsed_display = max(0.0, elapsed)
remaining = max(0.0, shutdown_delay - elapsed)
logger.debug(
f"Channel {channel_id[:8]} shutdown timer: "
f"{elapsed_display:.1f}s of {shutdown_delay}s elapsed "
f"({remaining:.1f}s remaining)"
)
gevent.sleep(min(poll_interval, remaining))
def handle_client_disconnect(self, channel_id):
"""
Handle client disconnect event - check if channel should shut down and
@ -995,18 +1070,20 @@ class ProxyServer:
disconnect_key = RedisKeys.last_client_disconnect(channel_id)
if shutdown_delay > 0:
self.redis_client.setex(disconnect_key, 60, str(time.time()))
logger.info(f"Waiting {shutdown_delay}s before stopping channel...")
gevent.sleep(shutdown_delay)
total = self.redis_client.scard(client_set_key) or 0
if total > 0:
logger.info(f"New clients connected during shutdown delay - aborting shutdown")
self.redis_client.delete(disconnect_key)
if not self.redis_client.get(disconnect_key):
self.redis_client.setex(
disconnect_key,
self._shutdown_disconnect_ttl(),
str(time.time()),
)
if not self._wait_for_shutdown_delay(channel_id):
return
if shutdown_delay <= 0:
self.redis_client.setex(disconnect_key, 60, str(time.time()))
else:
self.redis_client.setex(
disconnect_key,
self._shutdown_disconnect_ttl(),
str(time.time()),
)
# Coordinated stop runs local teardown + Redis cleanup once.
# Do not call _stop_upstream_before_redis_cleanup here — it races
@ -1794,7 +1871,11 @@ class ProxyServer:
if not disconnect_time:
# First time seeing zero clients, set timestamp
if self.redis_client:
self.redis_client.setex(disconnect_key, 60, str(current_time))
self.redis_client.setex(
disconnect_key,
self._shutdown_disconnect_ttl(),
str(current_time),
)
logger.warning(f"No clients detected for channel {channel_id}, starting shutdown timer")
elif current_time - disconnect_time > ConfigHelper.channel_shutdown_delay():
# We've had no clients for the shutdown delay period
@ -1808,7 +1889,9 @@ class ProxyServer:
else:
# There are clients or we're still connecting - clear any disconnect timestamp
if self.redis_client:
self.redis_client.delete(f"live:channel:{channel_id}:last_client_disconnect_time")
self.redis_client.delete(
RedisKeys.last_client_disconnect(channel_id)
)
else:
# === NON-OWNER CHANNEL HANDLING ===

View file

@ -93,10 +93,103 @@ class ChannelService:
@staticmethod
def is_channel_unavailable_for_new_clients(channel_id):
"""Reject new stream requests while teardown is active or shutdown is pending."""
return (
ChannelService.is_channel_teardown_active(channel_id)
or ChannelService.is_shutdown_pending(channel_id)
"""Reject new stream requests only while coordinated teardown is active."""
return ChannelService.is_channel_teardown_active(channel_id)
@staticmethod
def cancel_pending_shutdown(channel_id):
"""
Abort the post-disconnect grace timer when a client reconnects.
Clears the disconnect timestamp and any leaked stopping markers. When
upstream is still active but the last client released its profile slot
during the grace window, re-reserve the slot from Redis metadata.
Does not run during coordinated stop_channel() clearing teardown
markers mid-stop would leave clients attached to upstream that is
about to be torn down.
"""
from django.db import close_old_connections
proxy_server = ProxyServer.get_instance()
if not proxy_server.redis_client:
return False
if channel_id in proxy_server._stopping_channels:
return False
disconnect_key = RedisKeys.last_client_disconnect(channel_id)
had_pending = bool(proxy_server.redis_client.exists(disconnect_key))
in_grace = had_pending or ChannelService.is_shutdown_pending(channel_id)
if not in_grace:
return False
try:
proxy_server.redis_client.delete(disconnect_key)
metadata_key = RedisKeys.channel_metadata(channel_id)
state = proxy_server.redis_client.hget(
metadata_key, ChannelMetadataField.STATE
)
if state:
state_str = state.decode() if isinstance(state, bytes) else state
if state_str == ChannelState.STOPPING:
proxy_server.redis_client.hset(metadata_key, mapping={
ChannelMetadataField.STATE: ChannelState.ACTIVE,
ChannelMetadataField.STATE_CHANGED_AT: str(time.time()),
})
stop_key = RedisKeys.channel_stopping(channel_id)
if proxy_server.redis_client.exists(stop_key):
proxy_server.redis_client.delete(stop_key)
if ChannelService._channel_proxy_is_active(
proxy_server.redis_client, channel_id
):
from apps.channels.models import Channel
channel = Channel.objects.filter(uuid=channel_id).first()
if channel and not proxy_server.redis_client.get(
f"channel_stream:{channel.id}"
):
sid, pid, error, slot_reserved = channel.get_stream()
if error:
logger.warning(
f"Could not re-reserve stream for {channel_id} "
f"after shutdown cancel: {error}"
)
elif slot_reserved and sid and pid:
proxy_server.redis_client.hset(metadata_key, mapping={
ChannelMetadataField.STREAM_ID: str(sid),
ChannelMetadataField.M3U_PROFILE: str(pid),
})
logger.info(
f"Re-reserved profile slot for {channel_id} "
f"(stream={sid}, profile={pid})"
)
finally:
close_old_connections()
return True
@staticmethod
def _channel_proxy_is_active(redis_client, channel_id):
"""True when live proxy metadata shows this channel is still running."""
metadata_key = RedisKeys.channel_metadata(channel_id)
if not redis_client.exists(metadata_key):
return False
state = redis_client.hget(metadata_key, ChannelMetadataField.STATE)
if state is None:
return False
if isinstance(state, bytes):
state = state.decode()
return state in (
ChannelState.ACTIVE,
ChannelState.WAITING_FOR_CLIENTS,
ChannelState.BUFFERING,
ChannelState.INITIALIZING,
ChannelState.CONNECTING,
)
@staticmethod